Whole-site crawl

The crawl reads every page it can find on a site and scores each one the way the homepage scan does, then rolls the results up into whole-site averages, an issue list and a per-page table.

Plan: Starter (25 pages) and Pro / Enterprise. On Pro and Enterprise the web's Audit every page (the site audit, start_site_audit) covers up to 300 pages; the app's crawl (the deep-SEO crawl, /api/deep-seo/jobs) covers up to 5,000. Free has no crawl; the endpoints answer 402.

Where

• Web: SEO in the rail, open the domain, Whole-site audit, Audit every page; or a domain's SEO & search tab, Crawl site, with the results under Pages.
• iOS: SEO & accessibility, Pages, or Crawl every page on the Website module. The button reads "Crawling 41/152" with a progress bar. The app runs the deep-SEO crawl, not the site audit.
• API/MCP: POST /api/site-audit/start, GET /api/site-audit/status, GET /api/site-audit/results; tools start_site_audit, get_site_audit_status, get_site_audit_results. The older deep-SEO job endpoints (/api/deep-seo/jobs, fetch_sitemap, start_deep_seo_job) crawl a list of pages you choose, up to 5,000 on Pro and Enterprise.

How pages are found

1. The sitemap (and sitemap index) is read.
2. The site's own links are followed from the homepage and every discovered page.
3. The two lists are merged. When pages came from links rather than the sitemap, the crawl says so.

Duplicates are collapsed: /about and /about/ are one page, and one page linked with fifty different query strings counts as at most three pages (enough to keep real pagination such as ?page=2). On a real 300-page site that once doubled the crawl, and real pages fell off the end of the cap.

The cap

A Starter crawl stops at 25 pages and the report says "25 of 152 pages checked", which is the number that tells you how much of the site is unmeasured. On Pro and Enterprise the site audit (web Audit every page, start_site_audit) stops at 300 pages, the most one run can fetch; the deep-SEO crawl the app uses goes up to 5,000 pages. When your page list is trimmed the response says so.

What each page gets

Title and description with lengths, heading outline, word count, images and alt text, links in and out, canonical, indexability, structured data, and the same accessibility and mobile checks as the homepage. Each page has its own score and a defects list. Every defect carries the same fix as the homepage scan: what is on the page now, what to change, a draft where the page gives one, and where in your site builder to do it. A page over 256KB of HTML is read up to that point rather than skipped, and a heading or structured-data verdict is not given for a page that was cut off.

Templated pages

When the deep-SEO crawl finishes it compares the pages with each other, which no per-page check can do. Pages whose main text is mostly the same (an estimated 80% or more of their five-word runs in common, once each page's own name from its URL is set aside) are reported in groups of three or more as Templated Pages, on every page in the group. That is the city page copied for every town with the name swapped, which Google's spam policies call scaled content or doorway pages. Product pages and pages that are noindex or point their canonical elsewhere are left out, and pages with too little text are not compared. Pages rewritten for each town, with their own sentences, are not flagged; the check would rather miss a pattern than accuse a site of one. What to do about a group is in AI-written content and location pages.

Competing pages, duplicate titles

The same finish compares what each page is for. Two pages with one subject, one intent (a guide, a comparison, a service page) and one place, two pages aimed at the same tracked phrase, or two pages with one title are competing for one search: Google shows one page of a site for a search it treats as answered, so the two split the links and the relevance and neither ranks as well as one would. The page to change gets a Competing Pages finding naming the page to keep, why, and the steps (redirect, merge, or give it a different intent or a narrower subject). Where rank checks have been recorded and Google has ranked two different pages of the site for one phrase, that pair is reported at high severity whether or not the crawl's reading caught it. A title shared by three or more pages is a Duplicate Title on each of them (usually a layout default), a description shared by two or more a Duplicate Meta Description. The reasoning behind each pair is on the search intent report; the stored finding is what the fix list, the whole-site result, the API and the AI plan read. Pages that are noindex or point their canonical elsewhere are left out.

The whole-site audit (Audit every page) makes the same comparison and files competing_pages and, for a guide that never links to the service page on its subject, missing_funnel_link.

Running in the background

A crawl is asynchronous. Start it, poll the status, then read the results. In the app the page checking runs at a lower priority than anything you touch. Cancelling keeps every page already scored.

Rollups

The SEO & search tab shows whole-site averages once a crawl has run, the issues grouped by rule with the affected pages and their word counts, and the search intent report, which is computed from the newest crawl.

Common questions

The crawl found pages my sitemap does not list. Link discovery found them. Either add them to the sitemap or, if they should not be indexed, mark them noindex.

Some pages show as failed. A page that returned an error or timed out is counted as failed, not as a zero score.

How often should I crawl? After a content change, or monthly. Pro and Enterprise get an automatic homepage SEO scan every day and Starter every week; see Reports.