Skip to content
RightMessage

Our scanner

What our scanner fetches, and how to stop it.

Someone typed your domain into the RightMessage Personalization Audit to see who your site sells to. Here's exactly what we read when they do.

How it identifies itself

Every request carries this user agent:

RightMessageScanner/1.0 (+https://rightmessage.com/personalize/bot)

What we fetch

  • Your robots.txt, first. If it disallows us, we stop.
  • Your sitemaps, including sitemap indexes. We read up to 5,000 URLs and sort them by what the address says, without fetching those pages.
  • Your homepage, your pricing or sign-up page, the page written for each buyer we find, and one sample from each family of landing pages. That's 10 pages at most per scan.

We only fetch pages on the domain that was entered and the hosts under it, such as www. We use plain GET requests with no cookies, never fill in forms and never log in.

Limits

  • 10 seconds and 2 MB per page. After a timeout, a server error, a 429 or no response, we try the page once more.
  • About one to two minutes for the whole scan, and never more than 150 seconds.
  • Reports are kept for 7 days. Until then, a repeat visit opens the same report without fetching your site again.
  • Each domain gets at most one fresh scan a day.

When a page needs JavaScript

If a page's HTML has almost no readable text, we load that one page in a headless browser (Cloudflare Browser Rendering) with images, fonts and media blocked. If that also fails, or we hit a bot check, we stop and tell the person your site blocked us. We don't try to get past challenges.

Screenshots of your pages

For the report's before and after, we open your homepage once more in a headless browser (Cloudflare Browser Rendering) at 1280 by 800, the way a visitor's browser would: it loads your images, fonts, styles and scripts, but no video, starts with no cookies and sends nothing but GET requests. In our copy of the page we hide pop-ups, cookie banners and chat widgets, swap in each buyer's lines and take screenshots. Your live site never changes. If the page shows a bot check, we take no screenshots.

When someone asks for the PDF, we open your homepage, each buyer's page and your pricing or sign-up page the same way, and take screenshots further down each page.

Robots.txt rules we follow

We obey Disallow rules written for RightMessageScanner and for *. To block us everywhere:

User-agent: RightMessageScanner
Disallow: /

To let us in when a broader rule blocks bots:

User-agent: RightMessageScanner
Allow: /

Want a report taken down?

Reports are shared by link and hidden from search engines unless the site owner opts in. If you'd rather your site's report didn't exist at all, email us at hello@rightmessage.com.