Public website crawler
About YorvosBot
YorvosBot helps find publicly available information about companies when a customer asks Yorvos to research them.
What it reads
The crawler reads a site's home page and up to three related contact, about or wholesale pages. It may collect public business contact details and short descriptions.
How it visits
YorvosBot reads robots.txt before visiting a page, follows its access rules and spaces requests to the same domain. It stops when a site requires verification or refuses access.
Control access
Site owners can use robots.txt to allow or disallow YorvosBot. The crawler identifies itself with YorvosBot in its User-Agent header.