SEO Crawler

What an SEO crawler should tell you

A crawl is only useful if it ends in a decision. Knowing that 40 pages lack a meta description is a task list. Knowing which of those 40 pages already get impressions, which rank on page two, and which have no internal links pointing at them is a plan. Vee3.ai stores every fetched page with its status, canonical, robots directive, title, H1, meta description, H2 and H3 headings, JSON-LD types and main-content text, then joins that record to Search Console, GA4, provider rankings and backlinks for the same URL.

In one real workspace (The Marketing Agency, September 2026), a single scan read and judged 714 pages and found 32 that belonged on the roadmap but were not on it yet.

Crawlability issues, checked from one stored crawl

The Dashboard's Technical checklist reads the stored crawl evidence, so opening it never starts a new crawl. Each check reports issues, warnings or a pass, how many pages are affected, a priority, and example URLs you can open:

• Sitemap discovery, crawl coverage and indexability (status, noindex and off-site canonicals)
• HTTP responses and redirects, and canonical consistency across crawled URLs
• Missing or duplicate title tags, missing H1s, missing or duplicate meta descriptions
• Structured data on the page types that need it, and supporting H2/H3 structure
• Broken internal links, orphan pages and HTTPS pages linking to internal HTTP URLs
• Thin pages with under 300 characters of main content, and backlink coverage once backlinks are measured

Orphans and link counts come from the same crawl. For link-level detail, editorial versus automated links and pages that should link to each other, see the internal linking tool.

Evidence you can read, per page

Every keyword assignment arrives with a plain-English reason and a confidence score. Strong Search Console evidence scores 0.94, a top-10 provider ranking 0.88, validated demand 0.78. Anything under 0.68 is flagged for review instead of quietly guessed. Pages that are not search targets, such as legal, recruiting, blog index and archive pages, stay visible but are labeled with the reason, so they never pollute your plan.

Site crawling that is polite to fragile hosts

Requests run six at a time, 150 ms apart. If a server starts refusing with 403, 429 or 5xx responses, pacing drops to 700 ms with 20-second pauses, and the scan stops at a checkpoint rather than hammering the host. The crawler learns whether direct HTML, the WordPress REST API or Shopify JSON reads your site best, retries slow pages in two follow-up rounds, and checkpoints every step, so a scan of thousands of pages survives restarts.

From crawl to roadmap, without re-keying anything

When the scan finishes, qualifying pages go straight onto the SEO roadmap with their keyword, type and metrics. Because every page now owns one primary keyword, the keyword cannibalization checker can catch two pages chasing the same search before you write a third. The same crawl feeds the SEO dashboard and the topical authority map, so you crawl once and read the result everywhere.

While it works, a live insight stream narrates what the evidence already shows: how many URLs are real search targets, which competitor overlaps you most, which page has impressions but no clicks. By the time the table fills, you already know where to look.