BidsterBot is the identified crawler operated by Bidster. It fetches pages that publicly list government procurement opportunities, reads them, and records every request it makes.
User agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 BidsterBot/1.0 (+https://bot.bidster.ai/bot; ops@bidster.ai)
Contact: ops@bidster.ai
Signing key id: NIOH8mePckZCz4yaxRjnwKTqa03VQoG151tK-gLi6NI
Every request carries an HTTP Message Signature (Web Bot Auth). The public key is published at /.well-known/http-message-signatures-directory.
Requests come from exactly these addresses. The signed list is at /bot/ips.json.
Every address above has a reverse DNS name ending in
.bot.bidster.ai, and that name resolves forward
to the same address. The plain list is at
/bot/ips.txt.
BidsterBot issues GET and HEAD requests only. It sends no cookies, fills no forms, clicks nothing, and uses no proxies. A From header carries the contact address. After deciding a page is permitted it may render that page in headless Chromium, so the page's own scripts and stylesheets are also requested. Images, media, fonts, and analytics hosts are not.
At most one request every 2 seconds per host, and slower when robots.txt sets Crawl-delay or a response carries Retry-After. It backs off exponentially on 429 or 503, and stops fetching a host after repeated 403 or 429.
The Signature-Agent, Signature-Input, and Signature headers carry an Ed25519 HTTP Message Signature whose key is in the directory. Or check the source address against the signed list and its reverse DNS.
BidsterBot issues GET and HEAD requests only. It collects the text of publicly posted procurement notices and the links between them.
Publish a robots.txt that disallows BidsterBot, or email ops@bidster.ai. A disallow or an opt-out email stops fetching for that host.
User-agent: BidsterBot Disallow: /