# Crawler rules live HERE now, not in Cloudflare.
#
# Cloudflare's managed robots.txt (Security Settings -> Bot traffic -> "block
# training in robots.txt"; API: bot_management.is_robots_txt_managed) used to
# PREPEND a block to this file that disallowed nine AI crawlers -- Amazonbot,
# Applebot-Extended, Bytespider, CCBot, ClaudeBot,
# CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot and
# meta-externalagent -- and declared `search=yes, ai-train=no, use=reference`.
#
# That setting was turned off deliberately (2026-09-10). Blocking those crawlers
# made us uncitable in ChatGPT, Gemini and Perplexity answers, which is where
# finance buyers increasingly research vendors. Our competitors are answerable
# there and we were not. Absence from those answers costs more than the training
# data is worth.
#
# The signals below keep the training reservation while allowing retrieval:
#   search=yes     index us, excerpt us, link to us
#   ai-input=yes   ground real-time AI answers in our content -- this is the point
#   ai-train=no    do not train or fine-tune on it
#
# Flipping ai-train to yes should be a deliberate decision, not a default.
#
# If managed robots.txt is ever switched back on in Cloudflare, its group gets
# prepended and everything below is duplicated. Turn one off or the other.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://meetalder.com/sitemap.xml