AI crawler robots.txt generator

The first step of GEO (getting AI to cite you) is not writing content—it’sFirst confirm you’re not blocking AI crawlersMany CDNs, WAFs, and frameworks block AI crawlers outright in their default templates—so even excellent content goes unindexed. This tool lists 40+ major AI and search crawlers for granular per-crawler control, and automatically blocks backend and console paths.

Free · No login required Local Generation · Content Not Uploaded Override · 40+ AI and search crawlers

1Site Information

Site URL

Default private paths (recommended to block all) /admin Backend /console Console /admin.html /api/ Internal API

Additional paths to block (one per line, optional)

2Crawler list (check = allow)

All crawlers are allowed by default. Uncheck to block any.

Outputrobots.txt

0 lines

—

Place in the site root directory https://your-domain/robots.txt. After editing, verify it once with curl: curl --noproxy '*' https://your-domain/robots.txt

Self-checkHave you blocked yourself?

We’ve fallen into this pit before

1 · The default template blocks all AI crawlers.Many CDN/WAF and website framework robots.txt templates deny AI crawlers by default. As a result, content remains publicly accessible while GEO delivers zero effect—because no crawlers ever visit.

2 · GPTBot and OAI-SearchBot are two distinct user agents.Only allowing GPTBot still blocks OpenAI's search crawler. Likewise, ClaudeBot / Claude-User / Claude-SearchBot Each is independent.

3 · robots.txt is merely a "gentlemen's agreement."It won’t stop rogue crawlers. To truly protect your backend, use authentication (basic auth / login) instead of relying on robots.txt.

4 · Modifying robots.txt alone is insufficient.For purely frontend-rendered (SPA) sites, crawlers fetch empty shells. We encountered this when adding backlinks to a Vue site: you mustSimultaneouslyModifying either the JS array (for humans) or the 387 pre-rendered snapshot pages (for crawlers) alone renders the effort futile.