webtelemetry.dev is a live survey of the web as it launches. It watches public certificate logs for new domains, asks each domain’s registry when it was registered, and visits the new sites once to record what they are made of.

The result is a set of live statistics about the web’s first days: the platforms and frameworks new sites are built with, where they are hosted, who runs their DNS and mail, and whether they let AI crawlers read them.

Why I built it

I mostly built it to learn how to process large amounts of data: a stream that never stops, tools like Kafka, and a pipeline that has to keep up with millions of records a day. It is also becoming a big database I can use for my next projects. New websites were a good subject for that: the data is public, there is always more of it, and there are interesting questions to answer, like what people build new sites with today and whether they let AI crawlers in.

Built with Claude

Claude helped me build most of it. Working in Claude Code, it wrote most of the code with me and did the full optimisation work, from summary tables and Kafka tuning to the crawl scheduling and the deploy pipeline.

What it measures

  • The technology stack of each site, from about 7,500 open fingerprints plus rules of my own.
  • Hosting network and country, DNS and mail providers, SPF, DMARC, IPv6 and the certificate issuer.
  • Which of fifteen AI crawlers a site’s robots.txt turns away, from the whole site or from some pages only.
  • Whether it serves an llms.txt, and whether a platform or SEO plugin wrote it.
  • How many of each week’s launches are still working one, four, eight and twelve weeks later.

Open by default

Every site has a public profile, every chart carries a dated citation, and the whole dataset is published as a CSV every day under CC BY 4.0, with a snapshot kept each week. From 5 October, a weekly launch audit checks 1,000 recent launches against their registry again and publishes the result.

The full story of how it works, from certificate logs to Kafka to the first findings, is in the build story.