About our crawler
BidBeforeBot/1.0 is the crawler BidBefore uses to read public council committee pages.
What it reads
Public committee pages on UK council websites (committee lists, meeting lists, meeting pages and lists of forthcoming decisions), and published notices on Contracts Finder and Find a Tender.
How it behaves
- On weekdays it only fetches between 01:00 and 05:00 UK time. At weekends it may fetch at any time.
- It makes one request at a time per site, with a pause between requests.
- It doesn't read
robots.txt: many council sites serve their supplier's default file, which blocks all bots from papers published for the public. - If your site refuses it, for example with a 403, it doesn't try to get round that.
- Where possible it only downloads pages that have changed since its last visit.
- It doesn't keep copies of the pages it reads.
Stopping it
Email info@bidbefore.co.uk and we'll stop reading your site.

