GuideGet found

02Registration checklist

How to get found by AI answer engines

Five steps — with honest recrawl timelines. A well-optimised page that is blocked by robots.txt, not in a sitemap, or unverified in Search Console will not appear in AI answers regardless of how good the content is.

Last updated

Go to checklistAI crawler reference →

Priority

Crawl access comes first. Content quality second.

An AI engine cannot cite a page it cannot reach. Before working on content optimisation, verify that crawlers are allowed, the site is indexed, and registration is complete. This checklist covers the infrastructure layer; Lokrix AI measures the content layer.

01The five steps

Registration, crawl access, and sitemap — in order.

  1. 01

    Google Search Console

    Google AI OverviewsGoogle AI ModeGemini

    Google Search Console (GSC) is the authoritative registration point for every Google AI surface. Verifying ownership tells Google you are the site owner, enables sitemap submission, and exposes crawl status including the "last crawled" date you need to track index refresh.

    Verify
    Add your property at search.google.com/search-console and verify via DNS TXT, HTML file, or Google Analytics tag.
    Submit sitemap
    Navigate to Sitemaps in GSC and submit your sitemap.xml URL.
    Monitor
    Use the URL Inspection tool to see the last crawl date and request indexing after major content updates.
    Check coverage
    Pages with crawl errors, noindex tags, or blocked by robots.txt will not be indexed — and will not appear in AI answers.
  2. 02

    Bing Webmaster Tools — including AI Performance

    ChatGPTMicrosoft Copilot

    Bing Webmaster Tools is the registration point for ChatGPT (which retrieves via Bing) and Copilot. Since February 2026 it includes an "AI Performance" report that surfaces the grounding queries Copilot is using against your site — a near-official signal for Copilot visibility.

    Verify
    Register at bing.com/webmasters and verify via XML file or auto-import from Google Search Console.
    Submit sitemap
    Submit your sitemap.xml under the Sitemaps section. Bing crawls sitemaps more aggressively for recently changed pages.
    Check AI Performance
    Find the AI Performance report in Webmaster Tools. It shows which queries Copilot is grounding against your pages — a near-official signal for validating Copilot visibility. (Copilot is a grounding signal, not a separately scored Lokrix AI engine.)
    Monitor last crawled
    Use URL Inspection in Bing Webmaster Tools to see the last Bing crawl date per page.
  3. 03

    Allow AI crawlers in robots.txt

    All engines

    If an AI crawler is blocked in your robots.txt, the corresponding answer engine cannot see your content and your presence probability for that engine will be near zero — regardless of how good your content is. The default should be to allow all crawlers and block only the ones you have a specific reason to block.

    Audit
    Check your robots.txt at yourdomain.com/robots.txt. Look for Disallow rules that apply to *, User-agent: GPTBot, User-agent: ClaudeBot, or any other AI crawler.
    Allow explicitly
    If you want to allow all AI crawlers, ensure no Disallow: / rule applies to them. Optionally, add explicit Allow: / rules per user-agent for clarity.
    Google-Extended
    Google-Extended is Google's opt-out user-agent for AI training data — separate from Googlebot. Blocking Google-Extended does not block Google AI Overviews or AI Mode; it only opts your content out of AI training datasets.
    Verify
    Use Google's robots.txt Tester in GSC or a third-party tester to confirm your target user-agents are not blocked.

    Blocking a crawler does not remove you from an AI engine's index if you were previously indexed. It prevents future crawls and updates from reaching the engine.

  4. 04

    XML sitemap

    All engines

    A sitemap tells crawlers which URLs exist, their last modified date, and their priority. Without a sitemap, crawlers discover your pages only through links — which means new or recently updated pages may take significantly longer to be found and indexed.

    Create
    Generate a sitemap.xml listing all canonical, public URLs. Include lastmod dates — crawlers use them to prioritise re-crawls of changed pages.
    Submit
    Submit to both Google Search Console and Bing Webmaster Tools. Re-submit or ping whenever significant content changes are made.
    Keep fresh
    Set up automated sitemap regeneration when pages are added, updated, or removed. A stale sitemap with incorrect lastmod dates can slow re-crawl.
    Reference in robots.txt
    Add Sitemap: https://yourdomain.com/sitemap.xml to your robots.txt so any crawler can find it without requiring submission.
  5. 05

    llms.txt — optional, no measured ranking effect

    Varies

    An llms.txt file at your site root can guide AI agents to key pages in a machine-readable format. It is optional. There is no published evidence that llms.txt changes AI citation rates in any current answer engine. Creating one causes no harm and costs little time, but it is not a ranking lever and Lokrix AI does not report it as one.

    Format
    Place a plain-text file at /llms.txt. List your most important pages, their purpose, and any context an AI agent should know about your brand.
    Optional
    Create it if you have bandwidth. Do not prioritise it over the four steps above — those have measurable, documented effects on AI visibility.

    Source: No measured effect of llms.txt on citation rates in published research as of mid-2026. Lokrix AI tracks this and will update guidance if evidence changes.

02Recrawl timing

How long until a content change reaches AI answers.

Recrawl timing is not under Lokrix AI's control and no SLA exists. The ranges below are observed industry patterns for sites in normal standing. A brand-new domain or a site with crawl errors will be slower. A high-authority, frequently updated domain will be faster.

Rule of thumb: plan for 1–4 weeks from publishing a content change to seeing it reflected in AI answers. Verify using the "last crawled" date in Google Search Console URL Inspection and Bing Webmaster Tools URL Inspection.

Crawl (Bing)
Hours–days (high-authority sites) to weeks (low-authority, rarely updated)
Bing prioritises sites with high crawl frequency signals: consistent updates, many inbound links, and fast server response. A new page on a high-authority domain can appear in Bing within hours; a new domain may take weeks.
Crawl (Google)
Hours–days (high-authority) to weeks (low-authority)
Googlebot crawl cadence depends on PageRank, crawl budget, server health, and how recently the page changed. After submitting a URL via GSC URL Inspection, a fresh crawl typically follows within days, but is not guaranteed.
Bing index → ChatGPT / Copilot
Additional days to weeks after crawl
ChatGPT's retrieval via Bing, and Copilot's grounding, depend on Bing's web index — not the real-time crawl queue. Indexed content may take additional time to propagate to the search index that backs these AI surfaces.
Google index → AI Overviews / AI Mode / Gemini
Additional days to weeks after crawl
Google AI Overviews, AI Mode, and Gemini ground on Google's search index. A page must be crawled, indexed, and ranked before it can appear in AI answers. New content or recent updates typically take 1–4 weeks to reach AI surfaces, even if search results update faster.
Perplexity (live search)
Near-real-time for newly crawled pages
Perplexity runs live search queries at the time of the user's question, pulling from Google and Bing search results in real time. Pages that appear in Bing or Google top results for a query will typically be considered by Perplexity immediately — but they must first be indexed by those search engines.

How to verify last-crawled dates

Google Search Console
Open URL Inspection for any page → "Last crawl" date is shown in the Coverage section. Click "Request indexing" after major content updates to request a fresh crawl (not a guarantee, but usually processed within days for established sites).
Bing Webmaster Tools
Use URL Inspection to see the last Bingbot crawl date. Submit individual URLs via "URL Submission" to request a fresh crawl (limited to a daily quota). Check the AI Performance report to see if Copilot is grounding against your pages.

03Common blockers

What prevents AI engines from seeing your content.

Each of these is a hard block — content quality cannot compensate for them.

Crawler blocked in robots.txt
A Disallow rule for a crawler's user-agent prevents it from fetching any page on that path. The engine sees nothing on the blocked path — regardless of content.
noindex meta tag or X-Robots-Tag
A page with <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex HTTP header will not be indexed by Google or Bing, and therefore will not appear in AI answers.
Not verified in Search Console / Webmaster Tools
Lack of verification doesn't prevent crawling, but it removes your ability to submit sitemaps, monitor crawl health, request indexing, and see the AI Performance report in Bing.
No sitemap (for new or updated pages)
Without a sitemap, crawlers discover pages only through links. New pages, recently updated pages, or pages with few inbound links will take significantly longer to be found.
Crawl errors (5xx, 4xx)
Server errors on key pages cause crawlers to give up and retry later, reducing crawl frequency. Sustained errors can lead to pages being dropped from the index.
Page behind login or paywalled
AI crawlers do not log in. Any content requiring authentication or that returns a 401/403 for the crawler user-agent is invisible to AI answer engines.

Once crawlers can find you, measure your visibility.

Lokrix AI runs your prompt universe across every engine and tells you exactly where you appear — and where you don't — with a calibrated 95% CI.

Run your free scanGEO readiness setup →

Start free · no credit card