hrn-bot

hrn-bot is the crawler of Human Rights News, a non-commercial, community-ranked news aggregator for human-rights reporting. It reads the public RSS feeds of human-rights organisations and news outlets, and links readers to the original articles.

Identification

User agent: Mozilla/5.0 (compatible; hrn-bot/1.1; +https://human-rights.news/bot)

Every request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). The public key directory is https://api.human-rights.news/.well-known/http-message-signatures-directory, named by the Signature-Agent header on each request. A request claiming to be hrn-bot without a valid signature is not from us.

What it fetches

• Feeds: each configured RSS/Atom feed, about once a minute.
• Article pages: once per new story, to read the page's own og: metadata (title, description, image, site name) from the <head>. Failed fetches are retried a few times with growing back-off; pages that are gone are never retried.
• One image per story: the image the feed or page itself names, reduced to a single thumbnail of at most 400 pixels.
• robots.txt, cached per host.

Politeness

• Obeys robots.txt, including Crawl-delay. It matches the product token hrn-bot, then *.
• At most one request every two seconds to any host, or slower where the host asks for it.
• Honours 429 and Retry-After, and backs off on 403.
• Never fetches private or internal network addresses.

What is shown, and what is not done

Readers see the headline, the publisher's name, the publisher's own short description and a thumbnail, always linking to the original article. Full article text is not stored or republished. Content is not used to train AI models, and no summaries are generated.

Opting out, or asking for removal

To stop hrn-bot visiting a site, add to its robots.txt:

User-agent: hrn-bot
Disallow: /

To have stored thumbnails or stories removed, contact the operator via the imprint.