SEO
Robots.txt and Sitemap Checklist Before Asking Why a Page Is Not Indexed
A practical checklist for checking robots.txt, sitemap URLs, canonical tags, redirects, and internal links before investigating indexing issues.
Search intent: the page is published, but search engines still do not show it
Indexing questions often start too late in the process. A page is live, the team can open it in a browser, and then somebody asks why it is not appearing in search. Before treating that as a ranking problem, I check whether crawlers can discover the page, fetch it, understand the canonical URL, and see a useful internal path to it.
For a small static site, the safest first pass is evidence-based: open robots.txt, open the sitemap, confirm the exact public URL, check redirects, and make sure the page has a descriptive internal link from a related page. Sambro pages can keep company context at https://sambro.space/, browser utilities at https://tools.sambro.space/en/tools, and practical content at https://blog.sambro.space/ without changing the site structure.
Check robots.txt before changing the article
Robots.txt is not only a technical file for engineers. It is part of the publishing evidence. A single disallow rule, stale staging rule, or typo in the sitemap location can make a perfectly good page harder to crawl. I check whether the file is reachable, whether it names the expected sitemap, and whether any broad rule blocks the path where the new content lives.
I also keep the exact URL that was checked. `blog.sambro.space` and `www.blog.sambro.space` would be different hosts if both existed, and HTTP-to-HTTPS redirects can hide the first response. If a copied URL contains parameters, I inspect the safe parts with https://tools.sambro.space/en/tools/url-encoder before using it as evidence in a report.
Confirm the sitemap URL and the canonical URL match the intended page
A sitemap helps discovery, but only when it points to the canonical public page. If the sitemap contains an old slug, a redirected URL, a mixed-language path, or a preview host, search engines may spend time on the wrong address. The page itself should agree by exposing a canonical URL that matches the final public route.
My checklist is simple: open the sitemap entry, follow the final response, compare the final URL with the page canonical, and check that the title and description describe the same search intent. If the meta description is being tightened, I use https://tools.sambro.space/en/tools/word-counter to keep the snippet focused on the task instead of adding generic adjectives.
Use internal links as discovery evidence
A page that only exists in a sitemap can still be useful, but a page with a relevant internal link is easier for both visitors and crawlers to understand. The link does not need to be site-wide. One contextual link from a related article, tool page, or category page can show where the page belongs and what next action it supports.
For Sambro content, I prefer descriptive anchors such as JSON Formatter checklist, Markdown Preview workflow, or browser tools index instead of vague text like here. When the destination is a tool, the link should point to the matching language route where possible. That small habit improves usefulness without touching brand positioning or routing.
A practical indexing evidence checklist
My final pass is fixed: exact URL, final redirected URL, robots.txt response, sitemap location, sitemap entry, canonical URL, title, meta description, internal link source, language path, and whether the page returns a normal 200 response. If one item fails, I fix the smallest safe issue first instead of rewriting the whole page.
For a daily SEO routine, this checklist keeps action practical: publish one useful page, verify build output, push only the intended file, and watch CI/CD status. The goal is not to guarantee instant indexing. The goal is to remove avoidable crawl and discovery friction so search engines and visitors can understand the page that already exists.