Home / SEO / Technical SEO Fundamental...

SEO

Technical SEO Fundamentals: A Complete Beginner's Guide

August 01, 2026 · 17 min read
A search engine crawler moving smoothly through a well-structured website to crawl, index, and understand its pages

You can publish the best page on the internet, but if search engines can't reach it, render it or index it, it won't be found. Technical SEO is the work that makes sure they can. This technical SEO guide for beginners explains how crawling, rendering and indexing work, what to check, how to run a simple audit and which problems to fix first. You don't need to be a developer, just to know where to look.

What is technical SEO, and why does it matter?

Technical SEO is the practice of making a website easy for search engines to discover, crawl, render and index, and making sure the right version of each page is the one they store. It covers things like robots.txt, XML sitemaps, status codes, redirects, canonical tags, site structure, page speed, mobile versions, HTTPS and structured data.

It sits alongside the other two parts of SEO. On-page SEO is about what's on each page, such as titles, headings and internal links, and content strategy is about what you publish and why. Technical SEO is the layer underneath. It doesn't make a page rank on its own, but problems here can stop good content being found, often across many pages at once, such as a template that adds noindex to every product page.

How search engines crawl, render and index pages

Google describes processing pages in three main phases: crawling, rendering and indexing. Ranking happens later, when someone searches.

  • Crawling: a crawler such as Googlebot requests a URL it has discovered through links, sitemaps or other sources.
  • Rendering: the page's code, including JavaScript, is run to see the content a visitor would see.
  • Indexing: the search engine analyses the page, decides which URL is the canonical version, and may store it in its index.
  • Ranking: when someone searches, indexed pages are evaluated and ordered by relevance and many other signals.

These are separate processes, and no step is guaranteed. A URL can be discovered but never crawled, crawled but not indexed, or indexed under a different canonical URL from the one you expected. Most technical SEO work is about removing obstacles at each stage.

Crawlability: helping search engines find your pages

How search engines discover URLs

Crawlers mainly find URLs by following links from pages they already know and from sitemaps you submit. A page with no internal links pointing to it, an orphan page, is hard to discover even if it's in a sitemap. Use normal HTML links with an href attribute; Google extracts URLs from the href of links.

robots.txt: what it does, and what it doesn't

The robots.txt file at the root of your domain (for example, example.com/robots.txt) tells crawlers which paths they may request. A simple file might look like this:

User-agent: *
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

The most important thing to understand is what robots.txt doesn't do. Google says it's used mainly to avoid overloading your site with requests and "is not a mechanism for keeping a web page out of Google." A blocked URL can still be indexed if other pages link to it, just without a description. To keep a page out of search results, use noindex or password protection instead (Google's robots.txt guide).

Common mistakes include a leftover Disallow: / from a staging site, which blocks everything; blocking CSS or JavaScript that Google needs to render pages; and blocking a page marked noindex, which hides the rule from Google. Search Console's robots.txt report shows the files Google found, when they were last crawled and any errors.

XML sitemaps

An XML sitemap lists the URLs you want search engines to know about. Include canonical, indexable pages that return a 200 status. Leave out redirected URLs, error pages, noindex pages and duplicate variants. According to Google's sitemap documentation:

  • A single sitemap is limited to 50MB uncompressed or 50,000 URLs; larger sites split them and use a sitemap index file.
  • Google uses <lastmod> if it's consistently and verifiably accurate, and ignores <priority> and <changefreq>.
  • Sitemap inclusion is a weak signal of which URL you prefer as canonical.

Submit it in Search Console's Sitemaps report, which shows any parsing errors, and reference it in robots.txt. Most CMSs generate sitemaps automatically; your job is to check they list the right URLs.

Crawl budget: when it matters

Crawl budget is the amount of crawling a search engine is willing and able to do on your site. Google's crawl budget guide is aimed at large sites (roughly 1 million or more unique pages that change weekly), medium or larger sites (10,000 or more pages that change daily), and sites with many URLs classified as "Discovered – currently not indexed." It says that if your pages are usually crawled the day they're published, you don't need it. For most small sites, internal links and an accurate sitemap matter far more.

The first hurdle A page that can't be crawled or indexed can't compete, however good it is. Fix discovery and indexing problems before anything else.

Indexability: getting the right pages indexed

Crawling is not indexing

Being crawled doesn't mean being indexed. Search Console's Page indexing report shows which URLs Google has indexed and why others aren't, with reasons such as "Crawled – currently not indexed," "Discovered – currently not indexed," "Not found (404)," "Server error (5xx)," "Page with redirect," and "Duplicate, Google chose different canonical than user." Many exclusions are intentional, so focus on important pages that should be indexed but aren't.

Featured Recommendation AD · AFFILIATE
Nightwatch logo
4.5 / 5.0

Nightwatch

Accurate daily rank tracking tool with beautiful visual reporting dashboards

Best for: Daily Rank Tracking

noindex and X-Robots-Tag

To keep a page out of Google's index, add a robots meta tag to its <head>:

<meta name="robots" content="noindex">

For files such as PDFs, where you can't add a meta tag, your server can send the same rule as an HTTP header: X-Robots-Tag: noindex. Google's noindex documentation stresses that the page must not be blocked by robots.txt and must be otherwise accessible, or the crawler will never see the rule. Use it for pages such as internal search results, and check it hasn't been left on pages you want found, a common problem after launches and theme changes.

Canonical tags and duplicate URLs

Duplicate URLs are normal. The same content can be reached with and without tracking parameters, through sorting and filtering options, or via HTTP and HTTPS versions. Google says some duplicate content on a site is normal and not a violation of its spam policies; it groups duplicates and chooses one canonical URL to show.

You influence that choice with signals. Google's canonicalisation guide describes redirects and rel="canonical" as strong signals and sitemap inclusion as a weak one. A canonical tag looks like this:

<link rel="canonical" href="https://www.example.com/shoes/trail-running/">

It's a signal, not a command: Google can choose a different canonical, and the Page indexing report will tell you when it has. Common mistakes include canonicals pointing to redirected, broken or noindex URLs; every page canonicalising to the homepage; conflicting signals, such as a sitemap listing one version and the canonical naming another; and using robots.txt for canonicalisation, which Google advises against.

Site architecture and URL structure

A clear site structure helps people and crawlers find important pages. Group content into logical sections, link from category pages to the pages within them, and keep important pages a few links from the homepage, not buried deep in pagination. Breadcrumbs and links between related pages help too, the same principle behind a content strategy that compounds over time.

Google's URL structure guidance recommends simple, descriptive URLs, hyphens rather than underscores between words, and standard encoding for parameters. Consistency matters more than any single format.

Examples of clearer and less clear URL structures (illustrative)
Less clear Clearer Why
/index.php?id=4471&cat=7/shoes/trail-running/Readable and describes the content
/blog/2026/08/01/post_final_v2/blog/technical-seo-basics/Hyphens, no dates or version labels to go stale
/Products/Shoes and /products/shoes/products/shoes/ onlyOne consistent version avoids duplicates

Pagination: Google's pagination guidance says to give each page in a sequence its own URL, link the pages sequentially, and not use the first page as the canonical for the whole sequence. Faceted navigation: filters for size, colour or price can create huge numbers of URL combinations. Google's faceted navigation guidance describes the overcrawling this causes, and options such as disallowing filter URLs you don't need crawled or using URL fragments for filters.

HTTP status codes and redirects

Every URL returns a status code that tells crawlers what happened. Google explains how it handles each in its HTTP status code documentation:

Common HTTP status codes and their SEO implications, based on Google's documentation
Code What it means for Google When to use it
200 OKThe content can be considered for indexingAny live page you want found
301 (or 308)Permanent redirect; a strong signal that the target should be shown in searchA page has moved for good
302 (or 307)Temporary redirect; a weak signal, so the original URL is more likely to stay in searchA short-term move, such as a temporary promotion
404 Not FoundNot indexed; previously indexed URLs are removedA page is gone and has no replacement
410 GoneTreated the same as other 4xx errorsContent removed deliberately; either 404 or 410 is fine
5xx server errorsGoogle slows crawling; URLs that keep returning errors are eventually droppedNever deliberately; fix the server problem

Google's crawlers follow up to 10 redirect hops, but each hop adds delay and a point of failure. A redirect chain (A to B to C) should become A to C, a redirect loop (A to B to A) must be fixed because the page never loads, and internal links should point at final URLs.

When a page goes, redirect it with a 301 if there's a genuinely equivalent replacement, such as a newer version of the same product. If there isn't, a 404 or 410 is the honest answer. Google's site move guide warns that redirecting many old URLs to one irrelevant page, such as the homepage, can confuse users and might be treated as a soft 404. Make sure "not found" pages return a real 404 status, not a 200.

JavaScript SEO and rendering

Many sites build content with JavaScript, which adds a rendering step. Google's JavaScript SEO basics explain that Googlebot queues pages for rendering after crawling. Problems arise when important content, links or meta tags only exist after scripts run, when links use click handlers instead of href attributes, or when error pages return a 200 status.

  • Server-side or static rendering sends ready-made HTML, so content is visible without running JavaScript. Google says pre-rendering is still a good idea because it makes sites faster and not all bots can run JavaScript.
  • Client-side rendering builds the page in the browser. Google can render it, but it adds a step that can fail.
  • Dynamic rendering, serving bots a different pre-rendered version, is described by Google as a workaround, not a recommended solution; it recommends server-side rendering, static rendering or hydration instead.

How to check: in Search Console, run the URL Inspection tool on an important page, click "Test live URL", then "View tested page" to see the rendered HTML and a screenshot. If your main content or links are missing there, Google probably can't see them either.

Page speed and Core Web Vitals

Core Web Vitals are Google's metrics for loading, responsiveness and visual stability. The current thresholds for a good experience, measured at the 75th percentile of page loads, are set out on web.dev:

Current Core Web Vitals and "good" thresholds
Metric Measures Good
Largest Contentful Paint (LCP)How quickly the main content loads2.5 seconds or less
Interaction to Next Paint (INP)How quickly the page responds to interactions200 milliseconds or less
Cumulative Layout Shift (CLS)How much the layout moves unexpectedly0.1 or less

INP replaced First Input Delay (FID) as a Core Web Vital in March 2024, so guides that still mention FID are out of date. Google says Core Web Vitals are used by its ranking systems as part of page experience, but good scores don't guarantee top rankings, and it will still show the most relevant content even when the page experience is weaker.

How to use PageSpeed Insights: enter a URL at PageSpeed Insights and check mobile results first. The top section shows field data from real Chrome users when enough exists; the lower section is lab data from Lighthouse, a simulation that's useful for diagnosis. Work through the diagnostics for your slowest templates, such as large images, render-blocking scripts or images without set dimensions, and use Search Console's Core Web Vitals report to see which groups of pages need attention across the site.

Mobile-first indexing and HTTPS

Google uses the mobile version of a site's content, crawled with its smartphone crawler, for indexing and ranking. If your mobile pages lack content, links, structured data or metadata that desktop pages have, Google may not see it. Responsive design, which serves the same HTML on the same URL to every device, avoids most of these problems, the same user-first thinking behind good landing page design. Google retired its Mobile-Friendly Test and Search Console's Mobile Usability report on December 1, 2023, so check pages on real phones, in Chrome DevTools' device mode and with Lighthouse instead.

HTTPS protects visitors' data, and Google announced in 2014 that it uses HTTPS as a very lightweight ranking signal. Search Console's HTTPS report shows whether your pages are served securely. Watch for mixed content, where an HTTPS page loads images or scripts over HTTP, which most browsers now block. Make sure every version of your domain (HTTP, HTTPS, www and non-www) redirects in a single hop to one preferred version. When moving to HTTPS or a new domain, Google's site move guide notes that 301 and other permanent redirects don't cause a loss in PageRank. Common migration mistakes include forgetting to update canonicals and sitemaps, and leaving staging-site noindex rules or robots.txt blocks in place.

Structured data basics

Structured data, often called schema markup, describes a page's content in the Schema.org vocabulary, so search engines can understand details such as a product's price or an article's author. Google supports JSON-LD, Microdata and RDFa, and recommends whichever is easiest to maintain, in most cases JSON-LD. A minimal example for an article:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Technical SEO basics",
  "datePublished": "2026-08-01",
  "author": { "@type": "Organization", "name": "YourBrand" }
}
</script>

Common types include Organization, Article, BreadcrumbList, Product and LocalBusiness; for shops, product markup is especially useful, as covered in product page optimisation. Only mark up content that's visible on the page.

Validate markup with Google's Rich Results Test, which checks eligibility for Google's features, and the Schema Markup Validator for general Schema.org syntax. Valid markup isn't a promise: Google says it doesn't guarantee that structured data will appear in results, even when the Rich Results Test shows no errors. Features also change. Google stopped showing FAQ rich results in Search from May 7, 2026, according to its documentation changelog.

Technical SEO and AI search

Google says there are no additional requirements to appear in AI Overviews or AI Mode: a page must be indexed and eligible to be shown in Google Search with a snippet. Its guide to generative AI features adds that structured data isn't required for generative AI search, that there's no special schema for it, and that Google Search doesn't use llms.txt files.

So a page blocked by robots.txt, marked noindex or broken in rendering can't appear in Google's AI features any more than in its regular results. Other AI services run their own crawlers with their own names and documentation, so check each provider's documentation before allowing or blocking them in robots.txt. Google-Extended, the token Google offers for limiting use of content in some of its other AI systems, doesn't affect inclusion in Google Search. None of this guarantees AI citations: technical SEO makes content accessible, and quality and relevance do the rest. For more, see how AI Overviews are reshaping SEO, generative engine optimisation, designing websites for AI search and the zero-click reality.

How to do a technical SEO audit: a beginner workflow

  1. Set up Google Search Console. Verify your site, ideally as a domain property so it covers HTTP, HTTPS, www and non-www versions.
  2. Check robots.txt and your sitemap. Open your robots.txt file in a browser (your domain followed by /robots.txt) and look for unexpected Disallow rules; then check the robots.txt report. Open your sitemap, spot-check a few URLs, and confirm in the Sitemaps report that it was read successfully.
  3. Review the Page indexing report. Look for important pages listed as not indexed, and note the reason given for each.
  4. Inspect your key URLs. Run the URL Inspection tool on your homepage and a sample of important templates, such as a category, product, article and service page. Check whether each is indexable, the user-declared canonical against the Google-selected canonical, and whether the rendered page contains your main content.
  5. Crawl the site. Use an SEO crawler to list status codes, redirect chains, broken internal links, noindex tags, canonicals and orphan pages.
  6. Test performance. Run your main templates through PageSpeed Insights and review the Core Web Vitals report for groups of slow pages.
  7. Check mobile and HTTPS. Review key pages on a phone, check the HTTPS report, and confirm that HTTP and non-preferred versions redirect in one hop.
  8. Validate structured data on your main templates with the Rich Results Test.
  9. Prioritise and log fixes. Record each issue, the pages affected, the fix and the date, then recheck; Search Console lets you validate fixes for some issues.

How to prioritise what you find

Rank issues by impact and urgency, not by how many a tool reports. First fix anything that stops important pages being crawled or indexed: site-wide robots.txt blocks, stray noindex tags, server errors, broken migrations. Next, fix what wastes signals or confuses search engines: wrong canonicals, redirect chains, broken internal links, orphaned key pages. Then work on performance, mobile and structured data, starting with the templates that drive the most traffic or revenue. A template issue affecting thousands of pages beats a warning on one minor page.

Technical SEO checklist

Use this checklist as a working document for your first audit. Priorities are general recommendations; adjust them to your site.

A beginner's technical SEO checklist, with tools, common problems, actions and suggested priority
What to check Tool or report Common problem Recommended action Priority
robots.txt rulesYour /robots.txt file; robots.txt reportSite or key sections blocked; CSS or JS blockedRemove the unintended Disallow rulesHigh
Accidental noindexURL Inspection; crawlernoindex left on live templatesRemove from pages that should rankHigh
Indexing statusPage indexing reportImportant pages not indexedFix the reason shown, then request indexingHigh
Server errorsPage indexing report; Crawl Stats report5xx errors on key pagesFix hosting or application errors with your developerHigh
HTTPS and redirectsHTTPS report; crawlerHTTP pages, mixed content, multi-hop redirectsServe HTTPS everywhere; one-hop redirects to one versionHigh
XML sitemapSitemaps reportLists redirects, 404s or noindex URLsInclude only canonical, indexable 200 URLsMedium
Canonical tagsURL Inspection; crawlerPointing to wrong, redirected or broken URLsPoint each to the preferred, indexable URLMedium
Broken links and chainsCrawlerInternal links to 404s or through several redirectsUpdate links to final URLs; fix or remove dead linksMedium
RenderingURL Inspection (live test)Content or links missing from rendered HTMLServer-side or static rendering; use real linksMedium
Core Web VitalsPageSpeed Insights; Core Web Vitals reportSlow LCP, poor INP, layout shiftsOptimise images, scripts and layout on key templatesMedium
Mobile experienceReal devices; LighthouseMissing content or hard-to-use pages on mobileMatch mobile and desktop content; fix usability issuesMedium
Site structureCrawler; site navigationOrphan or deeply buried important pagesAdd internal links from relevant pagesMedium
Structured dataRich Results Test; Schema Markup ValidatorErrors, or markup that doesn't match the pageFix errors; mark up only visible contentLow to medium

For deeper audits, migrations or JavaScript-heavy sites, it's worth involving a developer or an experienced web team, because many fixes happen in templates, server settings or code.

The short version

Technical SEO makes sure search engines can discover, crawl, render and index the right version of your pages. Use robots.txt for crawling, noindex for keeping pages out of search, and canonical tags as signals, not guarantees. Keep sitemaps clean, return honest status codes, avoid redirect chains and don't hide key content behind fragile JavaScript. Aim for good Core Web Vitals, consistent mobile content and HTTPS everywhere. Start with Search Console and fix what blocks indexing first.

Want a technically sound site that search engines can find?

We audit, fix and maintain the technical foundations of your website so your content can be crawled, indexed and found.

Explore Search Engine Optimization →

Frequently asked questions

What is technical SEO?

Technical SEO is the work of making a website easy for search engines to discover, crawl, render and index, and making sure the right version of each page is stored. It covers robots.txt, sitemaps, status codes, redirects, canonical tags, site structure, JavaScript rendering, page speed, mobile versions, HTTPS and structured data.

What is the difference between technical SEO and on-page SEO?

On-page SEO covers the content and elements of individual pages, such as titles, headings and copy. Technical SEO covers whether search engines can access, process and index those pages at all, often through site-wide settings and templates. You need both.

Is technical SEO hard for beginners?

The fundamentals are learnable without being a developer: technical SEO for beginners is mostly about knowing what to check in Search Console, PageSpeed Insights and a site crawler. Fixes such as server errors, rendering problems or migrations usually need a developer, but you can find and prioritise them yourself.

How do I do a basic technical SEO audit?

Set up Google Search Console, check robots.txt and your sitemap, review the Page indexing report and inspect key URLs. Crawl the site for broken links, redirect chains, stray noindex tags and wrong canonicals, test main templates in PageSpeed Insights, check mobile pages and HTTPS, and validate structured data. Fix what blocks important pages from being indexed first.

Does robots.txt stop a page from appearing in Google?

No. Google says robots.txt is not a mechanism for keeping a page out of Google; a blocked URL can still appear in results without a description. To keep a page out, use noindex or password protection, and don't block that page in robots.txt, or Google can't see the noindex rule.

Does structured data guarantee rich results or visibility in AI answers?

No. Google says it does not guarantee that structured data will appear in search results, even when markup is valid, and its guidance for generative AI features says structured data isn't required and there is no special schema for them. Use it to describe visible content accurately.

What tools do I need for technical SEO?

Start with Google Search Console, which is free and includes the Page indexing, Sitemaps, robots.txt, Core Web Vitals and HTTPS reports and the URL Inspection tool. Add PageSpeed Insights, an SEO crawler for site-wide checks, and the Rich Results Test or Schema Markup Validator.

THE LAB REPORT

Tactics that move metrics — every Tuesday.

Be an early subscriber. No spam, unsubscribe anytime.