Why Do ChatGPT, Perplexity, Claude, and Gemini Ignore Most Business Websites?
When a commercial buyer asks ChatGPT, Perplexity, Claude, or Google Gemini a high-intent question—such as "Who are the best commercial finance brokers in the UK with transparent arrangement fees?" or "Which B2B AI search platform automates buyer guides?"—the underlying large language model executes a multi-step Retrieval-Augmented Generation (RAG) workflow.
Instead of matching a two-word keyword string, the AI engine decomposes the buyer's prompt into 4 to 8 sub-queries (known as query fan-out): pricing transparency, regulatory credentials, regional coverage, side-by-side competitor comparisons, and verified customer reviews. Most corporate websites fail this retrieval test because they are built as visual brochure sites: their pricing is hidden behind contact forms, their HTML is bloated with client-side JavaScript bundles, and they lack structured Organization, Service, and FAQPage JSON-LD graphs.
At RankBee (rankb.ee), our live 16-point PACT audit across thousands of UK and global domains shows that 78% of SMEs inadvertently block AI search crawlers via Cloudflare WAF rules or fail to provide a machine-readable /llms.txt specification.
How Does the 16-Point PACT Framework Measure AI Search Readiness?
RankBee evaluates every domain using the PACT Technical AI-Readiness Methodology, weighted across three core citation pillars that mirror how OpenAI, Perplexity, Anthropic, and Google ground their responses:
- Pillar 1: Infrastructure (30% Weight — 6 Checks): Verifies live HTTP 200 reachability and explicit
robots.txtpermissions forGPTBot,OAI-SearchBot,PerplexityBot,ClaudeBot, andGoogle-Extended. Checks for a valid/llms.txtroot index,/llms-full.txtdeep RAG corpus, XML sitemap discovery, and reconciledOrganization+ServiceJSON-LD schema. - Pillar 2: Narrative Structure (35% Weight — 5 Checks): Audits whether your pages open with a 50-word direct answer summary, align
<h1>and question-based<h2>headings to real buyer prompts, expose semantic HTML<table>pricing/comparison matrices, and embedFAQPageJSON-LD question graphs. - Pillar 3: Entity Authority & Corroboration (35% Weight — 5 Checks): Validates
sameAsentity disambiguation links (Companies House, Wikidata, LinkedIn, Trustpilot), regulatory verification anchors (FCA, FENSA, ISO, CQC), third-party review schema, and presence in neutral category comparison guides.
Step-by-Step: How to Fix Your Website for AI Search Engines in Under 15 Minutes
Moving from 0% AI Share of Voice to becoming the primary recommended provider in your category requires executing four concrete technical and editorial upgrades:
- Unblock AI Search Crawlers in robots.txt and WAF: Explicitly declare
Allow: /forGPTBot,OAI-SearchBot,PerplexityBot,ClaudeBot, andGoogle-Extended. Ensure Cloudflare Bot Fight Mode does not serve managed JavaScript challenges to verified AI retrieval user-agents. - Publish /llms.txt and /llms-full.txt at Your Root Domain: Create a clean Markdown index at
https://yourdomain.com/llms.txtsummarizing your legal entity, core services, transparent pricing tiers, ideal customer profile, and canonical URLs. Pair it with/llms-full.txtcontaining your complete service methodology and FAQs. - Inject Reconciled Organization, Service & FAQPage JSON-LD: Embed a single unified
@graphin your site header linking yourOrganizationnode (withsameAsregistry URIs and official registration numbers) to explicitServiceoffers andFAQPageQ&A pairs. - Automate Daily Answer-First Buyer Guide Publishing: Connect RankBee CMS Autopilot to your WordPress, Webflow, Shopify, Wix, Ghost, or custom webhook site to publish 15 to 90 structured buyer comparison and pricing guides every month.