Technical SEO: a checklist to improve crawling
Technical SEO is about making sure search engines can reach your pages, understand them and store them in the index. If that base fails, the best content never appears. The good news is that most technical problems on a small or mid-sized site repeat themselves and yield to an ordered checklist.
Key takeaways
- Order matters: crawling first, then indexing, then performance.
- One robots.txt block or a forgotten noindex can remove whole sections.
- Every page needs a single valid URL, declared with a canonical.
- Search Console is the reference for what Google actually sees.
What technical SEO covers
| Area | Question it answers |
|---|---|
| Crawling | Can the crawler reach the pages and their resources? |
| Indexing | Does Google store the right page, without duplicates? |
| Rendering | Does Google see the content a visitor sees? |
| Performance | Does it load fast and stay stable? |
| Structured data | Does Google understand what the page is about? |
| International | Does each user get the version in their language? |
Swipe the table sideways to see all columns.
1. Crawling
- robots.txt: check it does not block useful sections or the CSS and JavaScript needed to render pages.
- XML sitemap: only URLs that return 200, are indexable and are the canonical version. Submit it in Search Console.
- Internal links: every important page should be linked from another. Orphans get discovered late or never.
- Status codes: fix internally linked 404s and any 5xx errors.
- Redirects: one hop only; avoid chains and loops.
- CDN and firewall: make sure they are not blocking legitimate crawlers by mistake.
2. Indexing
- noindex: useful on thank-you pages and internal search results; serious if it survives a redesign.
- Canonicals: each page declares its preferred URL, and parameter variants point at the main one.
- One version of the domain: https, with or without www, with 301s from the rest.
- Duplicate content: merge or differentiate near-identical pages.
- The Search Console pages report: read the exclusion reasons. "Crawled – currently not indexed" usually points at quality or duplication, not a technical fault.
3. Architecture and URLs
- Short, readable, stable URLs. If one changes, redirect the old one.
- Key pages a few clicks from the homepage.
- Breadcrumbs with their matching markup.
- Pagination with crawlable links, not buttons that depend on JavaScript.
4. Rendering and JavaScript
Google can process JavaScript, but it is an extra step. If the main content only appears after scripts run, use Search Console's URL inspection to see the HTML Google receives. The safest route is to have important text, links and metadata in the initial HTML.
5. Speed and Core Web Vitals
| Metric | Good threshold | Typical fixes |
|---|---|---|
| LCP | 2.5 s or less | Optimise the hero image, caching, faster server |
| INP | 200 ms or less | Less JavaScript, break up long tasks |
| CLS | 0.1 or less | Set width and height on images and ad slots |
Swipe the table sideways to see all columns.
Use PageSpeed Insights to see field data where it exists, and prioritise the templates with the most traffic.
6. Mobile and security
- Responsive layout: no horizontal scrolling, tap targets that work.
- The same content on mobile and desktop; Google indexes the mobile version first.
- HTTPS everywhere, with no resources loaded over http.
7. Structured data
Add the markup that matches each page type (Organization, LocalBusiness, Article, Product, BreadcrumbList, FAQPage) and validate it with the Rich Results Test. Markup must describe what the visitor sees; do not use it to add hidden claims.
8. Multilingual sites
- One URL per language, with reciprocal `hreflang` and `x-default`.
- The correct `lang` attribute on each page.
- No automatic redirects by IP without letting the visitor choose.
Tools
- Search Console: indexing, sitemaps, URL inspection, Core Web Vitals.
- PageSpeed Insights: performance.
- A crawler: broken links, redirects, tags.
- Rich Results Test: structured data.
To organise the whole job, pair this list with the step-by-step SEO audit.
Common mistakes
- Shipping a redesign with the staging noindex still in place.
- Blocking a page in robots.txt that also carries noindex: Google cannot read the tag.
- Sitemaps full of redirected or deleted URLs.
- Canonicals all pointing at the homepage.
- Changing URLs without redirects.
Frequently asked questions
What is the difference between crawling and indexing?
Crawling is the robot visiting the page; indexing is storing it so it can be shown. A page can be crawled and never indexed.
Do I need to code?
To find the problems, no. To fix some of them (server, templates, JavaScript) you may need technical help.
How often should I check?
A monthly look at Search Console, plus a full pass after any significant change to the site.
Does a sitemap guarantee indexing?
No. It helps URLs get discovered, but Google decides what to index based on quality and accessibility.
Does robots.txt remove a page from Google?
No. It blocks crawling, but the URL can still appear. To exclude it, use noindex and allow crawling.
Sources
About this article. Produced by the MetaEX team with AI assistance, from the official documentation listed under Sources. Examples marked as illustrative are not client data. Last updated on 11 November 2026.
Want a read on your digital operation?
Talk to us: we first understand your situation, then tell you what makes sense.
Book a meeting