About our crawler
If you've seen BidBeforeBot/1.0 in your logs, this page is for you.
What it reads
Public committee pages on London councils' own sites: committee lists, meeting lists and meeting pages. It also reads published notices from Contracts Finder and Find a Tender.
How it behaves
- It only fetches between 01:00 and 05:00 UK time.
- It makes one request at a time per site, and honours
robots.txt, includingCrawl-delay. - It sends conditional requests where it can, so an unchanged page costs almost nothing.
- It doesn't keep copies of the pages it reads.
Stopping it
Disallow BidBeforeBot in your robots.txt and it will stop on its next visit.
This page is a placeholder. It needs real text before the site is public, and that text needs the business details behind it: who operates BidBefore, and where to write to.