Free audit

CiteLeadAudit: the CiteLead audit crawler

CiteLeadAudit fetches a website only when a person asks CiteLead for a free audit of it. It reads the site the way AI crawlers do and reports what they can reach, read and cite.

How to recognise it

It sends this User-Agent on every request it makes as itself:

CiteLeadAudit/1.0 (+https://getcitelead.com/bot)

Its robots.txt product token is CiteLeadAudit.

What it fetches

  • Only public pages. It sends no cookies or passwords, never submits forms and does not run JavaScript.
  • For one audit: your robots.txt first, then your homepage, your llms.txt, up to 6 sitemap files and up to 10 pages in all (the homepage included), with at most 4 requests at a time. It stays on your site: your domain and its subdomains.
  • To compare what AI crawlers get with what people get, it also requests your homepage once with a web browser's User-Agent and once each with the OAI-SearchBot and PerplexityBot User-Agents. Those requests come from our servers, not from the companies that run those crawlers.
  • If your server refuses CiteLeadAudit but serves browsers, the audit reads the same pages as a browser, so the report can show what your visitors get.

How often

A report is kept for 24 hours. A request for the same site in that time is answered from the report, without fetching your site again. A report can be re-run at most once an hour. Report pages are not indexed by search engines.

How to block it

Add this to your robots.txt:

User-agent: CiteLeadAudit
Disallow: /

The audit reads robots.txt before anything else. With this rule it fetches nothing more from your site, and the person who asked is told your site does not allow our crawler. A narrower rule, such as Disallow: /private/, keeps it out of those paths only.

If your robots.txt has no CiteLeadAudit group, the audit follows your User-agent: * group, as the robots.txt standard (RFC 9309) says. If that group disallows your homepage, the report is built from robots.txt alone.

A firewall rule against the CiteLeadAudit User-Agent alone does not stop the audit (see above). robots.txt does.

Contact

Questions about CiteLeadAudit, or something we should know? Contact StartPro Corp., the company behind CiteLead.