Leadlantern docs
 v1.0.0
Get help
● Daily work · Website check

The website check
what it reads, and how politely.

When you press Check the website, Leadlantern opens that business's public website the way a polite visitor would: robots.txt first, then the home page and at most four more pages. It writes down facts (is there a phone layout, a contact form, online booking) and scores the lead from them. It runs only when a person asks, one site at a time.

robots.txt respectedUser agent LeadlanternAuditMax. 5 pageshttps only

01Run a check

  1. Open a lead that has a website.
  2. Press Check the website (later: Check again). It takes a few seconds, longer with PageSpeed.
  3. Read the result: two lists, Missing and Found, the date and how many pages were read. The lead's score and reasons update at the same time.
Website check panel with the Missing and Found lists and the report buttons

02What it does, step by step

  1. robots.txt. It asks for /robots.txt first. If the rules forbid the home page, it stops and reads nothing else.
  2. The home page, always over https, even when the business lists an http:// address.
  3. Up to four more pages of the same site that look like contact, booking, about or imprint pages, one after another, each only if robots.txt allows it.
  4. Google PageSpeed, only when you set a PageSpeed key: one request for the home page's speed on phones (up to 60 seconds).

Each request waits at most 10 seconds and reads at most 2 MB. Only public https addresses on the standard port are asked; addresses inside private networks are refused, and every redirect is checked the same way.

03robots.txt behaviour

  • Rules are read as the robots.txt standard (RFC 9309) describes. A group for LeadlanternAudit wins over the * group; the longest matching rule decides; a tie goes to Allow.
  • No robots.txt (for example 404): everything is allowed.
  • A robots.txt the server cannot deliver (error 5xx or 429): the check stops and records that the site did not answer.
  • A robots.txt too large to read: the check stops as blocked, rather than guess.
  • Blocked: the lead shows "The site asks robots not to read it, so nothing was fetched." Nothing is scored from a site that was not read.

04The user agent

Every request of the check introduces itself as:

LeadlanternAudit/1.0 (+https://your-domain/r/about)

The link points to a public page on your own server that explains the check to the owner of the site (next section). Without a public URL the user agent is LeadlanternAudit/1.0 only, so set the public URL before you check sites.

05The /r/about page

https://your-domain/r/about is a public page, in your brand, titled How the website check works. It tells a site owner that the program is called LeadlanternAudit, that it reads robots.txt first, opens the home page and at most four more pages, does not log in, fill in forms or run scripts, keeps only general addresses such as info@, runs only when a person asks and does not come back on a schedule. It shows how to block it:

User-agent: LeadlanternAudit
Disallow: /

A Disallow: / for all programs (User-agent: *) is followed as well. When the Brand support e-mail is set, the page invites questions and requests not to be contacted at that address.

Public page How the website check works, in the agency's brand

06What it writes down

FactRead from
Secure connection (https)Whether the site answers over https at all.
Layout for phonesThe viewport tag of the home page.
Page title, description for search results, main headingThe home page's title, meta description and h1.
Business details for search enginesStructured data (JSON-LD types such as LocalBusiness).
Online booking25 booking tools, e.g. Calendly, Booksy, Fresha, Treatwell, OpenTable, Doctolib.
Visitor statistics14 tools, e.g. Google Analytics, Plausible, Matomo.
Advertising pixels8 pixels, e.g. Meta Pixel, Google Ads, LinkedIn Insight Tag.
Chat15 chat tools and links, e.g. Intercom, Tawk.to, Crisp, WhatsApp links.
Contact form, phone number, social profilesForms (including common form plugins and embeds), tel: links and numbers, links to social networks.
Copyright year, built withThe newest year in a copyright line; the site builder or CMS (WordPress, Wix, Squarespace and others).
SpeedHow long the home page took to answer; with a PageSpeed key, Google's score on phones (0–100).
E-mail addressesOnly general addresses are kept (info, contact, hello, office, booking, bookings, sales, support, mail, kontakt, reception, enquiries, team, studio, admin @…), the business's own domain first. Other addresses are counted and dropped. If the lead has no e-mail yet, the first kept one is saved on it.

07What it cannot see

  • It does not run JavaScript. A site that draws its content with scripts can show false "missing" findings; open the site yourself before you mention a finding.
  • A social media page listed as the website (Facebook, Instagram, Linktree and others) is treated as "no website of its own" and not read.
  • A site without working https is not read: it is recorded as "no working secure connection", which is itself a finding.