DeepSeek is citing our content in answers about managed IT services, but our robots.txt blocks AI crawlers. Should we allow DeepSeek through, or is it already reading us somehow?

DeepSeek likely crawled your site before you added the block, or it's accessing cached versions from search engines and third-party sources. Robots.txt alone doesn't prevent all AI access—it's a request, not a firewall. If you want DeepSeek to crawl fresh content going forward, you need to explicitly allow it in robots.txt.

Start by checking your current robots.txt file. If it contains a blanket "User-agent: *" with "Disallow: /", that's blocking everything including DeepSeek. To allow DeepSeek specifically, add these lines:

User-agent: DeepSeek Disallow:

If you're using a CDN or WAF (Cloudflare, AWS Shield, etc.), verify that bot rules aren't filtering DeepSeek's requests. Check your access logs for requests from DeepSeek's IP ranges—if you see none but citations exist, the crawler likely used older snapshots.

If your site is JavaScript-heavy, that's a separate blocker. DeepSeek and most LLM crawlers don't wait for JS rendering the way Google does. Ensure your core content (service descriptions, case studies, pricing) is in plain HTML, not behind a JS paywall.

Test this week by allowing DeepSeek in robots.txt, then monitor your logs for crawl activity over the next 5–7 days. Cross-check by asking DeepSeek directly about your services—if it's pulling fresh citations instead of repeating old claims, the blocks are working.