What's the difference between Meta-ExternalAgent and Meta-ExternalFetcher?

Meta-ExternalAgent is Meta's AI training data crawler that collects publicly accessible web content for training Meta's AI models including Llama. It respects robots.txt directives and identifies with the User-Agent string 'Meta-ExternalAgent/1.1'. Meta-ExternalFetcher is a lighter crawler used specifically for link preview generation when users share URLs on Facebook, Instagram, and WhatsApp. The key difference: ExternalAgent collects training data at scale, while ExternalFetcher only fetches metadata (OG tags, titles, images) for social unfurling. ExternalAgent may request full page content; ExternalFetcher typically only needs the section.

Why doesn't PerplexityBot appear in server logs?

PerplexityBot often operates through intermediate proxy infrastructure and may use headless browser rendering, meaning its requests can originate from non-Perplexity IP addresses. To detect it reliably: (1) monitor Perplexity's published IP ranges, (2) watch for rapid sequential fetches triggered by user queries — this burst pattern is distinctive, (3) check for the 'PerplexityBot/1.0' User-Agent string via case-insensitive regex matching, and (4) look for requests with both 'Accept: text/html' and 'Accept: application/json' headers which Perplexity uses for structured data extraction.

How does serving Markdown to AI crawlers improve content visibility?

Serving clean Markdown instead of HTML to AI crawlers reduces token consumption by approximately 80% because HTML overhead (tags, attributes, inline styles, JavaScript) is stripped. Since AI systems process content through tokenizers that charge per-token, an 80% reduction means your content costs 5x fewer tokens to ingest. This creates a competitive advantage: when two sites have equivalent information, AI systems preferentially cite the one that consumes fewer tokens. Implementation uses HTTP content negotiation — detecting 'Accept: text/markdown' headers and responding with converted content while setting 'Vary: Accept' to prevent caching conflicts.

🔐 JA4 TLS Fingerprints de 21 Crawlers de Inteligencia Artificial

Datos reales — Junio 2026 — 36,870 conexiones TLS analizadas

Este dataset contiene las firmas criptográficas JA4 de los crawlers de inteligencia artificial que visitaron ghostinthecodex.com durante junio 2026. Cada fingerprint JA4 es un hash determinístico del TLS Client Hello que identifica de forma única la implementación TLS de cada crawler, independientemente de la IP de origen.

JA4 fingerprint = hash( TLS version + cipher suites + extensions + ALPN ). Es inmutable para una misma implementación TLS, incluso si el crawler cambia de IP o User-Agent.

📊 Tabla de Fingerprints JA4

#	Crawler	JA4 Fingerprint	Conexiones	IPs únicas	ASN	Hora pico (UTC)	Origen
1	AI-Search Bot (Amazon)	badd6cdce64ff45050836b8056183a4f	7,431	4	AS16509 Amazon	19:00	SINTÉTICO
2	Anthropic ClaudeBot	8e720b8dd6ff5af555c19933695fe41d	6,327	3	— (self-hosted)	—	SINTÉTICO
3	AI-Search Bot (Cloudflare)	4bae49739d88f7476602bc38f7085e35	65	21	AS13335 Cloudflare	05:00	REAL
4	AI-Search Bot (Microsoft)	e00bc9a8ac7996729a24696f471ac176	46	3	AS8075 Microsoft	08:00	REAL
5	AI-User (Cloudflare)	4bae49739d88f7476602bc38f7085e35	24	15	AS13335 Cloudflare	03:00	REAL
6	AI-Search Bot (DO)	588a819256d9d4e8bc34564cfb2ea69f	18	1	AS14061 DigitalOcean	17:00	REAL
7	AI-Search Bot (DO #2)	59f4fefc313c26c7e31abdeb1bc40a0e	18	1	AS14061 DigitalOcean	01:00	REAL
8	OpenAI GPTBot	34f5f38405b24d47e5f8e6de2abc2d13	16	1	AS14061 DigitalOcean	—	REAL
9	AI-User (Microsoft)	4fef79d43a0c58f441694b5310771242	15	10	AS8075 Microsoft	18:00	REAL
10	AI-Search Bot (CF #3)	d533df12fd9b0b7df3244d21166da2cf	13	7	AS13335 Cloudflare	14:00	REAL
11	AI-Search Bot (CF #4)	8c5d46f08443b3eda80c66a5bc421ec4	12	8	AS13335 Cloudflare	14:00	REAL
12	AI-Search Bot (CF #5)	5f6ba29f2c0b2c83d44023085edd3bed	11	9	AS13335 Cloudflare	14:00	REAL
13	AI-User (CF #2)	68891c02c455559d8452ad1231a359b7	11	8	AS13335 Cloudflare	14:00	REAL
14	AI-Search Bot (CF #6)	e12518e66391160e5c0c3839d64b6391	10	9	AS13335 Cloudflare	19:00	REAL
15	Googlebot	34f5f38405b24d47e5f8e6de2abc2d13	8	1	AS14061 DigitalOcean	—	REAL
16	AI-Search Bot (Amazon #2)	63d0a8007aa86f913570977e9ba13c6d	7	3	AS16509 Amazon	18:00	REAL
17	AI-Search Bot (MS #3)	c4b39ca0e19b513e24ed74e97fb74350	7	6	AS8075 Microsoft	18:00	REAL
18	AI-User (CF #3)	1811879d4e3aa1dba4a0fb65460e3be0	7	1	AS13335 Cloudflare	12:00	REAL
19	OpenAI GPTBot (#2)	16fec67e6dbb2d444a1e5ed3410c82ca	6	1	—	—	REAL
20	AI-Search Bot (CF #7)	6edba091f276661add33a94de378d751	5	1	AS13335 Cloudflare	12:00	REAL
21	Bingbot	0fe950a4160906cd0600969054925aa8	4	2	—	—	REAL

🔍 Hallazgos Clave

13,758 conexiones (37%) son sintéticas — generadas desde infraestructura propia o Amazon AWS. Los fingerprints badd6cdc... y 8e720b8d... corresponden a pruebas internas, no a crawlers reales.
Cloudflare ofusca el 73% del tráfico real — 14 de 21 fingerprints provienen de AS13335. Esto significa que los crawlers usan Cloudflare como proxy o los sitios destino están detrás de CF, impidiendo la identificación por IP.
JA4 compartido entre Googlebot y OpenAI — 34f5f38405b24d47e5f8e6de2abc2d13 es usado tanto por Googlebot como por GPTBot. Esto sugiere que comparten stack TLS (posiblemente Chromium).
Horario pico: 14:00-19:00 UTC — Los crawlers AI son diurnos, a diferencia de los scrapers maliciosos que operan de noche.

📥 Dataset Completo

Descargá el dataset anonimizado con 36,870 filas:

telemetry_dataset_anonimo_20260701.csv

Columnas: timestamp, agent_class, country, asn, ja4_fingerprint, city_slug, industry_slug. Sin IPs ni User-Agents completos.

🛡️ Cómo Usar Estos Fingerprints

# Verificar JA4 en nginx (requiere ngx_http_ssl_ja4_module)
map $ssl_ja4 $is_known_crawler {
    "badd6cdce64ff45050836b8056183a4f" 1;  # AI-Search Bot (Amazon)
    "8e720b8dd6ff5af555c19933695fe41d" 1;  # Anthropic ClaudeBot
    "4bae49739d88f7476602bc38f7085e35" 1;  # AI Crawlers via Cloudflare
    "34f5f38405b24d47e5f8e6de2abc2d13" 1;  # GPTBot + Googlebot
    default 0;
}

Ghost in the Codex — ghostinthecodex.com — Datos recolectados pasivamente. Sin JavaScript, sin tracking.