You can publish the best page on the internet, but if search engines can't reach it, render it or index it, it won't be found. Technical SEO is the work that makes sure they can. This technical SEO guide for beginners explains how crawling, rendering and indexing work, what to check, how to run a simple audit and which problems to fix first. You don't need to be a developer, just to know where to look.
What is technical SEO, and why does it matter?
Technical SEO is the practice of making a website easy for search engines to discover, crawl, render and index, and making sure the right version of each page is the one they store. It covers things like robots.txt, XML sitemaps, status codes, redirects, canonical tags, site structure, page speed, mobile versions, HTTPS and structured data.
It sits alongside the other two parts of SEO. On-page SEO is about what's on each page, such as titles, headings and internal links, and content strategy is about what you publish and why. Technical SEO is the layer underneath. It doesn't make a page rank on its own, but problems here can stop good content being found, often across many pages at once, such as a template that adds noindex to every product page.
How search engines crawl, render and index pages
Google describes processing pages in three main phases: crawling, rendering and indexing. Ranking happens later, when someone searches.
- Crawling: a crawler such as Googlebot requests a URL it has discovered through links, sitemaps or other sources.
- Rendering: the page's code, including JavaScript, is run to see the content a visitor would see.
- Indexing: the search engine analyses the page, decides which URL is the canonical version, and may store it in its index.
- Ranking: when someone searches, indexed pages are evaluated and ordered by relevance and many other signals.
These are separate processes, and no step is guaranteed. A URL can be discovered but never crawled, crawled but not indexed, or indexed under a different canonical URL from the one you expected. Most technical SEO work is about removing obstacles at each stage.
Crawlability: helping search engines find your pages
How search engines discover URLs
Crawlers mainly find URLs by following links from pages they already know and from sitemaps you submit. A page with no internal links pointing to it, an orphan page, is hard to discover even if it's in a sitemap. Use normal HTML links with an href attribute; Google extracts URLs from the href of links.
robots.txt: what it does, and what it doesn't
The robots.txt file at the root of your domain (for example, example.com/robots.txt) tells crawlers which paths they may request. A simple file might look like this:
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
The most important thing to understand is what robots.txt doesn't do. Google says it's used mainly to avoid overloading your site with requests and "is not a mechanism for keeping a web page out of Google." A blocked URL can still be indexed if other pages link to it, just without a description. To keep a page out of search results, use noindex or password protection instead (Google's robots.txt guide).
Common mistakes include a leftover Disallow: / from a staging site, which blocks everything; blocking CSS or JavaScript that Google needs to render pages; and blocking a page marked noindex, which hides the rule from Google. Search Console's robots.txt report shows the files Google found, when they were last crawled and any errors.
XML sitemaps
An XML sitemap lists the URLs you want search engines to know about. Include canonical, indexable pages that return a 200 status. Leave out redirected URLs, error pages, noindex pages and duplicate variants. According to Google's sitemap documentation:
- A single sitemap is limited to 50MB uncompressed or 50,000 URLs; larger sites split them and use a sitemap index file.
- Google uses
<lastmod>if it's consistently and verifiably accurate, and ignores<priority>and<changefreq>. - Sitemap inclusion is a weak signal of which URL you prefer as canonical.
Submit it in Search Console's Sitemaps report, which shows any parsing errors, and reference it in robots.txt. Most CMSs generate sitemaps automatically; your job is to check they list the right URLs.
Crawl budget: when it matters
Crawl budget is the amount of crawling a search engine is willing and able to do on your site. Google's crawl budget guide is aimed at large sites (roughly 1 million or more unique pages that change weekly), medium or larger sites (10,000 or more pages that change daily), and sites with many URLs classified as "Discovered – currently not indexed." It says that if your pages are usually crawled the day they're published, you don't need it. For most small sites, internal links and an accurate sitemap matter far more.
The first hurdle A page that can't be crawled or indexed can't compete, however good it is. Fix discovery and indexing problems before anything else.
Indexability: getting the right pages indexed
Crawling is not indexing
Being crawled doesn't mean being indexed. Search Console's Page indexing report shows which URLs Google has indexed and why others aren't, with reasons such as "Crawled – currently not indexed," "Discovered – currently not indexed," "Not found (404)," "Server error (5xx)," "Page with redirect," and "Duplicate, Google chose different canonical than user." Many exclusions are intentional, so focus on important pages that should be indexed but aren't.
Nightwatch
Accurate daily rank tracking tool with beautiful visual reporting dashboards
Best for: Daily Rank Tracking
noindex and X-Robots-Tag
To keep a page out of Google's index, add a robots meta tag to its <head>:
<meta name="robots" content="noindex">
For files such as PDFs, where you can't add a meta tag, your server can send the same rule as an HTTP header: X-Robots-Tag: noindex. Google's noindex documentation stresses that the page must not be blocked by robots.txt and must be otherwise accessible, or the crawler will never see the rule. Use it for pages such as internal search results, and check it hasn't been left on pages you want found, a common problem after launches and theme changes.
Canonical tags and duplicate URLs
Duplicate URLs are normal. The same content can be reached with and without tracking parameters, through sorting and filtering options, or via HTTP and HTTPS versions. Google says some duplicate content on a site is normal and not a violation of its spam policies; it groups duplicates and chooses one canonical URL to show.
You influence that choice with signals. Google's canonicalisation guide describes redirects and rel="canonical" as strong signals and sitemap inclusion as a weak one. A canonical tag looks like this:
<link rel="canonical" href="https://www.example.com/shoes/trail-running/">
It's a signal, not a command: Google can choose a different canonical, and the Page indexing report will tell you when it has. Common mistakes include canonicals pointing to redirected, broken or noindex URLs; every page canonicalising to the homepage; conflicting signals, such as a sitemap listing one version and the canonical naming another; and using robots.txt for canonicalisation, which Google advises against.
Site architecture and URL structure
A clear site structure helps people and crawlers find important pages. Group content into logical sections, link from category pages to the pages within them, and keep important pages a few links from the homepage, not buried deep in pagination. Breadcrumbs and links between related pages help too, the same principle behind a content strategy that compounds over time.
Google's URL structure guidance recommends simple, descriptive URLs, hyphens rather than underscores between words, and standard encoding for parameters. Consistency matters more than any single format.
| Less clear | Clearer | Why |
|---|---|---|
| /index.php?id=4471&cat=7 | /shoes/trail-running/ | Readable and describes the content |
| /blog/2026/08/01/post_final_v2 | /blog/technical-seo-basics/ | Hyphens, no dates or version labels to go stale |
| /Products/Shoes and /products/shoes | /products/shoes/ only | One consistent version avoids duplicates |
Pagination: Google's pagination guidance says to give each page in a sequence its own URL, link the pages sequentially, and not use the first page as the canonical for the whole sequence. Faceted navigation: filters for size, colour or price can create huge numbers of URL combinations. Google's faceted navigation guidance describes the overcrawling this causes, and options such as disallowing filter URLs you don't need crawled or using URL fragments for filters.
HTTP status codes and redirects
Every URL returns a status code that tells crawlers what happened. Google explains how it handles each in its HTTP status code documentation:
| Code | What it means for Google | When to use it |
|---|---|---|
| 200 OK | The content can be considered for indexing | Any live page you want found |
| 301 (or 308) | Permanent redirect; a strong signal that the target should be shown in search | A page has moved for good |
| 302 (or 307) | Temporary redirect; a weak signal, so the original URL is more likely to stay in search | A short-term move, such as a temporary promotion |
| 404 Not Found | Not indexed; previously indexed URLs are removed | A page is gone and has no replacement |
| 410 Gone | Treated the same as other 4xx errors | Content removed deliberately; either 404 or 410 is fine |
| 5xx server errors | Google slows crawling; URLs that keep returning errors are eventually dropped | Never deliberately; fix the server problem |
Google's crawlers follow up to 10 redirect hops, but each hop adds delay and a point of failure. A redirect chain (A to B to C) should become A to C, a redirect loop (A to B to A) must be fixed because the page never loads, and internal links should point at final URLs.
When a page goes, redirect it with a 301 if there's a genuinely equivalent replacement, such as a newer version of the same product. If there isn't, a 404 or 410 is the honest answer. Google's site move guide warns that redirecting many old URLs to one irrelevant page, such as the homepage, can confuse users and might be treated as a soft 404. Make sure "not found" pages return a real 404 status, not a 200.
JavaScript SEO and rendering
Many sites build content with JavaScript, which adds a rendering step. Google's JavaScript SEO basics explain that Googlebot queues pages for rendering after crawling. Problems arise when important content, links or meta tags only exist after scripts run, when links use click handlers instead of href attributes, or when error pages return a 200 status.
- Server-side or static rendering sends ready-made HTML, so content is visible without running JavaScript. Google says pre-rendering is still a good idea because it makes sites faster and not all bots can run JavaScript.
- Client-side rendering builds the page in the browser. Google can render it, but it adds a step that can fail.
- Dynamic rendering, serving bots a different pre-rendered version, is described by Google as a workaround, not a recommended solution; it recommends server-side rendering, static rendering or hydration instead.
How to check: in Search Console, run the URL Inspection tool on an important page, click "Test live URL", then "View tested page" to see the rendered HTML and a screenshot. If your main content or links are missing there, Google probably can't see them either.
Page speed and Core Web Vitals
Core Web Vitals are Google's metrics for loading, responsiveness and visual stability. The current thresholds for a good experience, measured at the 75th percentile of page loads, are set out on web.dev:
| Metric | Measures | Good |
|---|---|---|
| Largest Contentful Paint (LCP) | How quickly the main content loads | 2.5 seconds or less |
| Interaction to Next Paint (INP) | How quickly the page responds to interactions | 200 milliseconds or less |
| Cumulative Layout Shift (CLS) | How much the layout moves unexpectedly | 0.1 or less |
INP replaced First Input Delay (FID) as a Core Web Vital in March 2024, so guides that still mention FID are out of date. Google says Core Web Vitals are used by its ranking systems as part of page experience, but good scores don't guarantee top rankings, and it will still show the most relevant content even when the page experience is weaker.
How to use PageSpeed Insights: enter a URL at PageSpeed Insights and check mobile results first. The top section shows field data from real Chrome users when enough exists; the lower section is lab data from Lighthouse, a simulation that's useful for diagnosis. Work through the diagnostics for your slowest templates, such as large images, render-blocking scripts or images without set dimensions, and use Search Console's Core Web Vitals report to see which groups of pages need attention across the site.
Mobile-first indexing and HTTPS
Google uses the mobile version of a site's content, crawled with its smartphone crawler, for indexing and ranking. If your mobile pages lack content, links, structured data or metadata that desktop pages have, Google may not see it. Responsive design, which serves the same HTML on the same URL to every device, avoids most of these problems, the same user-first thinking behind good landing page design. Google retired its Mobile-Friendly Test and Search Console's Mobile Usability report on December 1, 2023, so check pages on real phones, in Chrome DevTools' device mode and with Lighthouse instead.
HTTPS protects visitors' data, and Google announced in 2014 that it uses HTTPS as a very lightweight ranking signal. Search Console's HTTPS report shows whether your pages are served securely. Watch for mixed content, where an HTTPS page loads images or scripts over HTTP, which most browsers now block. Make sure every version of your domain (HTTP, HTTPS, www and non-www) redirects in a single hop to one preferred version. When moving to HTTPS or a new domain, Google's site move guide notes that 301 and other permanent redirects don't cause a loss in PageRank. Common migration mistakes include forgetting to update canonicals and sitemaps, and leaving staging-site noindex rules or robots.txt blocks in place.
Structured data basics
Structured data, often called schema markup, describes a page's content in the Schema.org vocabulary, so search engines can understand details such as a product's price or an article's author. Google supports JSON-LD, Microdata and RDFa, and recommends whichever is easiest to maintain, in most cases JSON-LD. A minimal example for an article:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Technical SEO basics",
"datePublished": "2026-08-01",
"author": { "@type": "Organization", "name": "YourBrand" }
}
</script>
Common types include Organization, Article, BreadcrumbList, Product and LocalBusiness; for shops, product markup is especially useful, as covered in product page optimisation. Only mark up content that's visible on the page.
Validate markup with Google's Rich Results Test, which checks eligibility for Google's features, and the Schema Markup Validator for general Schema.org syntax. Valid markup isn't a promise: Google says it doesn't guarantee that structured data will appear in results, even when the Rich Results Test shows no errors. Features also change. Google stopped showing FAQ rich results in Search from May 7, 2026, according to its documentation changelog.
Technical SEO and AI search
Google says there are no additional requirements to appear in AI Overviews or AI Mode: a page must be indexed and eligible to be shown in Google Search with a snippet. Its guide to generative AI features adds that structured data isn't required for generative AI search, that there's no special schema for it, and that Google Search doesn't use llms.txt files.
So a page blocked by robots.txt, marked noindex or broken in rendering can't appear in Google's AI features any more than in its regular results. Other AI services run their own crawlers with their own names and documentation, so check each provider's documentation before allowing or blocking them in robots.txt. Google-Extended, the token Google offers for limiting use of content in some of its other AI systems, doesn't affect inclusion in Google Search. None of this guarantees AI citations: technical SEO makes content accessible, and quality and relevance do the rest. For more, see how AI Overviews are reshaping SEO, generative engine optimisation, designing websites for AI search and the zero-click reality.
How to do a technical SEO audit: a beginner workflow
- Set up Google Search Console. Verify your site, ideally as a domain property so it covers HTTP, HTTPS, www and non-www versions.
- Check robots.txt and your sitemap. Open your robots.txt file in a browser (your domain followed by
/robots.txt) and look for unexpectedDisallowrules; then check the robots.txt report. Open your sitemap, spot-check a few URLs, and confirm in the Sitemaps report that it was read successfully. - Review the Page indexing report. Look for important pages listed as not indexed, and note the reason given for each.
- Inspect your key URLs. Run the URL Inspection tool on your homepage and a sample of important templates, such as a category, product, article and service page. Check whether each is indexable, the user-declared canonical against the Google-selected canonical, and whether the rendered page contains your main content.
- Crawl the site. Use an SEO crawler to list status codes, redirect chains, broken internal links,
noindextags, canonicals and orphan pages. - Test performance. Run your main templates through PageSpeed Insights and review the Core Web Vitals report for groups of slow pages.
- Check mobile and HTTPS. Review key pages on a phone, check the HTTPS report, and confirm that HTTP and non-preferred versions redirect in one hop.
- Validate structured data on your main templates with the Rich Results Test.
- Prioritise and log fixes. Record each issue, the pages affected, the fix and the date, then recheck; Search Console lets you validate fixes for some issues.
How to prioritise what you find
Rank issues by impact and urgency, not by how many a tool reports. First fix anything that stops important pages being crawled or indexed: site-wide robots.txt blocks, stray noindex tags, server errors, broken migrations. Next, fix what wastes signals or confuses search engines: wrong canonicals, redirect chains, broken internal links, orphaned key pages. Then work on performance, mobile and structured data, starting with the templates that drive the most traffic or revenue. A template issue affecting thousands of pages beats a warning on one minor page.
Technical SEO checklist
Use this checklist as a working document for your first audit. Priorities are general recommendations; adjust them to your site.
| What to check | Tool or report | Common problem | Recommended action | Priority |
|---|---|---|---|---|
| robots.txt rules | Your /robots.txt file; robots.txt report | Site or key sections blocked; CSS or JS blocked | Remove the unintended Disallow rules | High |
| Accidental noindex | URL Inspection; crawler | noindex left on live templates | Remove from pages that should rank | High |
| Indexing status | Page indexing report | Important pages not indexed | Fix the reason shown, then request indexing | High |
| Server errors | Page indexing report; Crawl Stats report | 5xx errors on key pages | Fix hosting or application errors with your developer | High |
| HTTPS and redirects | HTTPS report; crawler | HTTP pages, mixed content, multi-hop redirects | Serve HTTPS everywhere; one-hop redirects to one version | High |
| XML sitemap | Sitemaps report | Lists redirects, 404s or noindex URLs | Include only canonical, indexable 200 URLs | Medium |
| Canonical tags | URL Inspection; crawler | Pointing to wrong, redirected or broken URLs | Point each to the preferred, indexable URL | Medium |
| Broken links and chains | Crawler | Internal links to 404s or through several redirects | Update links to final URLs; fix or remove dead links | Medium |
| Rendering | URL Inspection (live test) | Content or links missing from rendered HTML | Server-side or static rendering; use real links | Medium |
| Core Web Vitals | PageSpeed Insights; Core Web Vitals report | Slow LCP, poor INP, layout shifts | Optimise images, scripts and layout on key templates | Medium |
| Mobile experience | Real devices; Lighthouse | Missing content or hard-to-use pages on mobile | Match mobile and desktop content; fix usability issues | Medium |
| Site structure | Crawler; site navigation | Orphan or deeply buried important pages | Add internal links from relevant pages | Medium |
| Structured data | Rich Results Test; Schema Markup Validator | Errors, or markup that doesn't match the page | Fix errors; mark up only visible content | Low to medium |
For deeper audits, migrations or JavaScript-heavy sites, it's worth involving a developer or an experienced web team, because many fixes happen in templates, server settings or code.
The short version
Technical SEO makes sure search engines can discover, crawl, render and index the right version of your pages. Use robots.txt for crawling, noindex for keeping pages out of search, and canonical tags as signals, not guarantees. Keep sitemaps clean, return honest status codes, avoid redirect chains and don't hide key content behind fragile JavaScript. Aim for good Core Web Vitals, consistent mobile content and HTTPS everywhere. Start with Search Console and fix what blocks indexing first.
Want a technically sound site that search engines can find?
We audit, fix and maintain the technical foundations of your website so your content can be crawled, indexed and found.
Explore Search Engine Optimization →Frequently asked questions
What is technical SEO?
Technical SEO is the work of making a website easy for search engines to discover, crawl, render and index, and making sure the right version of each page is stored. It covers robots.txt, sitemaps, status codes, redirects, canonical tags, site structure, JavaScript rendering, page speed, mobile versions, HTTPS and structured data.
What is the difference between technical SEO and on-page SEO?
On-page SEO covers the content and elements of individual pages, such as titles, headings and copy. Technical SEO covers whether search engines can access, process and index those pages at all, often through site-wide settings and templates. You need both.
Is technical SEO hard for beginners?
The fundamentals are learnable without being a developer: technical SEO for beginners is mostly about knowing what to check in Search Console, PageSpeed Insights and a site crawler. Fixes such as server errors, rendering problems or migrations usually need a developer, but you can find and prioritise them yourself.
How do I do a basic technical SEO audit?
Set up Google Search Console, check robots.txt and your sitemap, review the Page indexing report and inspect key URLs. Crawl the site for broken links, redirect chains, stray noindex tags and wrong canonicals, test main templates in PageSpeed Insights, check mobile pages and HTTPS, and validate structured data. Fix what blocks important pages from being indexed first.
Does robots.txt stop a page from appearing in Google?
No. Google says robots.txt is not a mechanism for keeping a page out of Google; a blocked URL can still appear in results without a description. To keep a page out, use noindex or password protection, and don't block that page in robots.txt, or Google can't see the noindex rule.
Does structured data guarantee rich results or visibility in AI answers?
No. Google says it does not guarantee that structured data will appear in search results, even when markup is valid, and its guidance for generative AI features says structured data isn't required and there is no special schema for them. Use it to describe visible content accurately.
What tools do I need for technical SEO?
Start with Google Search Console, which is free and includes the Page indexing, Sitemaps, robots.txt, Core Web Vitals and HTTPS reports and the URL Inspection tool. Add PageSpeed Insights, an SEO crawler for site-wide checks, and the Rich Results Test or Schema Markup Validator.