Our payment API docs show up in ChatGPT answers about integration best practices, but we're blocked by our CDN's bot rules. Should we allowlist AI crawlers, or does our robots.txt matter more?

Your CDN's bot rules take precedence over robots.txt. If your CDN is blocking AI crawlers at the edge, robots.txt directives won't help—the crawler never reaches your origin to read them. ChatGPT's citations likely come from cached or pre-block snapshots. You need to act on the CDN layer first.

Start by auditing your WAF and CDN settings this week. Check your firewall rules for blocks on known AI crawler IPs and user agents (OpenAI, Anthropic, Google's AI crawlers). Most CDNs let you create allowlist rules by user agent or IP range. If your fintech compliance team requires restrictions, create a narrow exception for your public-facing docs and API reference pages—don't open your entire origin.

Once you've allowlisted the crawlers, update your robots.txt to explicitly allow them on paths you want cited (e.g., /docs, /api-reference), and disallow sensitive paths like /admin or /billing. Then add an llms.txt file at your root with a pointer to your preferred crawl policy.

Test by running a manual crawl simulation: use curl with the OpenAI user agent to fetch a doc page from your CDN. If you get a 200 response, the crawler can reach it. Wait 2–3 weeks, then search ChatGPT for a question your docs answer—you should see fresher citations or new ones. If citations disappear, your CDN rule was too broad; tighten it.