ChadaQBot

ChadaQBot

ChadaQBot reads the public pages ChadaQ customers add to their assistant. Here is how it behaves and how to control it.

(01) What it does

ChadaQBot fetches public web pages that a ChadaQ customer has added to their assistant as a source, so the assistant can answer that customer’s visitors from them. It only visits sites a customer points it at, and while the customer keeps weekly re-crawl on, it reads the site again about once a week.

(02) How it identifies itself

Every request carries this user agent:

ChadaQBot/1.0 (+https://chadaq.com/bot)

(03) How it behaves

  • It reads the robots.txt of the site it starts from and obeys the rules there for ChadaQBot, for its earlier name DocwrenBot, and for *. It understands plain path rules; wildcards such as * or $ inside a path aren’t supported.
  • It makes at most two requests at a time to one site.
  • It follows links only within the site it started from, but pages listed in that site’s sitemap and redirects can lead to another site, whose own robots.txt it doesn’t read.
  • It fetches only public http and https addresses on the usual ports, follows at most five redirects and skips pages larger than 5 MB.
  • It doesn’t fill in forms or run scripts.

(04) Blocking it

To keep ChadaQBot off your whole site, add this to your robots.txt:

User-agent: ChadaQBot
Disallow: /

A site that blocks it isn’t crawled, and the customer who added it sees that robots.txt blocked the crawl.

(05) Contact

Questions about ChadaQBot: [email protected].