Make sure AI engines can crawl you

# Allow every AI + search crawler to reach the whole site.
User-agent: *
Allow: /

# Search + AI-retrieval crawlers (do NOT disallow these)
User-agent: Googlebot        # Google Search → AI Overviews / AI Mode / Gemini
User-agent: Bingbot          # Bing → ChatGPT retrieval + Copilot grounding
User-agent: GPTBot           # OpenAI crawler
User-agent: OAI-SearchBot    # ChatGPT search
User-agent: PerplexityBot    # Perplexity
User-agent: ClaudeBot        # Anthropic (Claude) crawler
User-agent: Google-Extended  # Google's AI-training opt-in/out UA (see note)
Allow: /

# Point crawlers at your sitemap
Sitemap: https://example.com/sitemap.xml

Audit

Open yourdomain.com/robots.txt. Look for any Disallow rule that applies to *, Googlebot, Bingbot, GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot on the paths you want cited.

Never blanket-block

A bare "Disallow: /" under "User-agent: *" blocks every crawler from the entire site. This is the single most common cause of near-zero AI visibility.

Google-Extended

Google-Extended is a separate opt-out UA for AI-training data — blocking it does NOT remove you from AI Overviews, AI Mode or Gemini (those use Googlebot). Only block it if you intend to opt out of training.

Add your sitemap

Include a "Sitemap:" line so any crawler can discover it without submission.

Verify

Use the robots.txt tester in Google Search Console (and Bing) to confirm the target user-agents are allowed on your core paths.

Note: blocking a crawler does not delete you from an engine's existing index — it only prevents future crawls and updates from reaching it.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-06-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-06-15</lastmod>
  </url>
</urlset>

Create

Generate a sitemap.xml of all canonical, indexable URLs with accurate <lastmod> dates. Most CMSs and frameworks can emit one automatically.

Include core pages

Make sure your homepage and the pages you want surfaced (product, pricing, docs, key articles) are listed. The readiness check flags a landing page that is missing from the sitemap.

Submit

Submit the sitemap URL in both Google Search Console and Bing Webmaster Tools (see below). Re-submit after major content changes.

Keep fresh

Regenerate the sitemap when pages are added, changed, or removed. Stale <lastmod> dates can slow re-crawl of updated pages.

Reference in robots.txt

Add the "Sitemap:" line to robots.txt so crawlers can find it even without submission.

# llms.txt — optional pointer file for AI agents (no measured ranking effect)
# https://example.com/llms.txt

# Example Co — cloud invoicing for freelancers
> One-line summary an AI agent can quote.

## Core pages
- [Product](https://example.com/product): what it does
- [Pricing](https://example.com/pricing): plans and prices
- [Docs](https://example.com/docs): API and how-to guides

Source: no measured effect of llms.txt on citation rates in published research as of mid-2026. The readiness check reports its presence as informational only — never a failure.

1 · Add property

Go to search.google.com/search-console and add your site as a Domain property (covers all subdomains + protocols) or a URL-prefix property.

2 · Verify

Verify ownership via a DNS TXT record (Domain property), an HTML file upload, an HTML meta tag, or a linked Google Analytics / Tag Manager account.

3 · Submit sitemap

Open Sitemaps in the left nav and submit https://yourdomain.com/sitemap.xml. GSC reports how many URLs it discovered.

4 · Inspect a URL

Use the URL Inspection tool to see the last crawl date, index status, and any coverage errors (noindex, blocked by robots.txt, 4xx/5xx).

5 · Request indexing

After major content changes, use "Request indexing" in URL Inspection. It is a request, not a guarantee, but is usually processed within days for established sites.

Pages with crawl errors, a noindex tag, or blocked by robots.txt will not be indexed — and therefore will not appear in Google's AI answers.

1 · Register

Go to bing.com/webmasters and add your site.

2 · Verify / import

Verify via XML file, meta tag, or DNS — or import your already-verified sites directly from Google Search Console in one click.

3 · Submit sitemap

Submit your sitemap.xml under Sitemaps. Bing crawls sitemaps aggressively for recently changed pages.

4 · AI Performance

Open the AI Performance report to see which queries Copilot is grounding against your pages — a near-official signal for validating Copilot visibility. (Copilot is a grounding signal, not a separately scored Lokrix AI engine.)

5 · URL submission

Use URL Submission to push individual new/updated URLs for faster crawling (subject to a daily quota), and URL Inspection to see the last Bingbot crawl date.