← About Metawatch

About our crawler

If you have seen Metawatch/2.0 in your logs, this page is for you. Metawatch is a research crawler run by the IPTC, the global standards body for news media. Once a month it visits a few hundred news publishers, reads the embedded metadata of a small sample of the photographs they publish, and reports how much survives. It does not collect content for AI training or any other reuse.

At a glance

What it fetches

For each publisher, once a month:

• robots.txt.
• An RSS feed or news sitemap to find recent articles. If we don't already know yours, we look for one linked from your homepage or in robots.txt, then try a few common paths such as /feed and /sitemap.xml, stopping at the first that works.
• Up to 20 articles published in the last 30 days.
• One image per article: the lead photograph the publisher declares in its JSON-LD or og:image markup, up to 20 MB.
• A few site-wide policy files, so we can report publishers' AI and text-and-data-mining preferences: /.well-known/tdmrep.json, /ai.txt and /.well-known/trust.txt.

It does not run JavaScript, follow links beyond the sampled articles, submit forms, log in, or fetch anything behind a paywall.

What we keep

From each article: its URL, headline, publication date, language, keywords and its structured-data (NewsArticle) markup. Not the article text. From each image: its URL, dimensions, file size, which metadata fields are present and their values, and any C2PA manifest summary. The image file itself is deleted as soon as its metadata has been read. Everything we keep is published as open data on the Dataset page, so you can see exactly what we recorded about your site.

How it behaves

• One request at a time per site, at least one second apart plus a random delay, or longer if your robots.txt sets a Crawl-delay. If your crawl delay is long, we take fewer articles rather than staying longer: each site is capped at five minutes.
• robots.txt is honoured for our token (Metawatch) and for User-agent: *. Article URLs it disallows are dropped. Where we use a feed a publisher has listed for us, we fetch that feed before robots.txt, because some firewalls block an address outright after it reads robots.txt; the article URLs from the feed are still checked against your rules. If robots.txt can't be read at all, we treat the site as allowing us.
• No disguise. We always send the same User-Agent and From headers, never a browser's. We don't rotate addresses to get round blocks, solve CAPTCHAs, or try to defeat bot-management rules. When a site blocks us, we record it as blocked and stop.

Verifying that a request is really ours

The crawler runs on GitHub-hosted runners, whose IP addresses change from run to run, so we can't publish an IP list. Instead each request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). Each signed request carries:

• Signature-Agent: "https://iptc.org"
• Signature-Input covering @authority and signature-agent, with keyid="rtvhLrHEsgX7dk-Bj5Fb9Zzi2CjxEccPinja47gY6yk", alg="ed25519" and tag="web-bot-auth", valid for 60 seconds
• Signature: the Ed25519 signature

Our public key is published at https://iptc.org/.well-known/http-message-signatures-directory. A request that claims to be Metawatch but doesn't verify against that key isn't from us.

Blocking or limiting us

To stop Metawatch crawling your site, add this to your robots.txt:

User-agent: Metawatch
Disallow: /

To slow it down instead:

User-agent: Metawatch
Crawl-delay: 10

Changes take effect at the next monthly crawl. You can also ask us to remove your site, or correct its entry, at office@iptc.org. We honour removal requests without question. A blocked site still appears on Metawatch with the status robots_disallow, since honouring that choice is part of what we measure.

Seen something wrong?

If the crawler misbehaves (too many requests, ignoring your robots.txt, anything else) please tell us via this short form, with the time, your domain and, if you have them, the request headers. We read every response.