DNSniffer Crawler
We operate web crawlers to support our domain intelligence platform. Transparent identification, robots.txt compliance, and adaptive backoff - by default.
DNSnifferBot
$ user-agent
$ robots.txt
- User-agent: DNSnifferBot
- Obeys robots.txt: yes
- Obeys crawl-delay: yes
$ purpose
Powers the DNSniffer database for domain intelligence, SSL certificate monitoring, and subdomain discovery. Crawls websites to gather domain information, certificate data, and link relationships.
Certificate Transparency
DNSniffer is an official participant in the Certificate Transparency protocol, helping ensure secure and transparent SSL certificate monitoring.
▸ DNSnifferBot indexes fresh, accurate information about websites and how they link to each other.
Domain Intelligence
Discover subdomains, SSL certificate issues, and domain relationships across your infrastructure.
Security Monitoring
Monitor SSL certificates, detect expired certificates, and identify security vulnerabilities.
Free Tools
Create a free DNSniffer account to access domain analytics, backlink data, and security monitoring.
DNSnifferAudit
Powers our security audit tools - SSL issues, domain misconfigurations, and security vulnerabilities.
Obeying Robots.txt
Both bots strictly respect robots.txt - disallow rules, allow rules, and crawl-delay directives. Only verified site owners can allow DNSnifferAudit to disobey robots.txt on their site.
Crawl-delay is strictly followed when requesting HTML pages. It cannot be followed when rendering JavaScript, as modern web pages often require multiple resources to be loaded simultaneously.
Caching assets
During crawls, we cache frequently requested files (images, CSS, JS) to minimize repeated fetches, reducing bandwidth consumption and server load.
Load management
If we encounter non-200 status codes, especially 4xx or 5xx errors, we automatically reduce our crawling speed for that site. This ensures minimal stress on sites experiencing outages or high load.
Transparent practices
Hosting providers, CDNs, and CMS platforms can verify our traffic via our publicly available user-agent strings. We're committed to transparency about our crawling activities.
▸ We provide clear, user-friendly options to control our bots:
Via Robots.txt
To change the frequency at which DNSnifferBot visits your site, specify a minimum delay:
Crawl-Delay: [value]
To stop DNSnifferBot from visiting your site or a section, use Disallow:
Disallow: /path-to-disallow/
Returning non-200 status codes
You may temporarily reduce DNSnifferBot crawling speed by returning 4xx or 5xx HTTP status codes for the duration of an outage or maintenance window. Our bot will detect these errors and back off automatically.
Adjusting speed in Security Audit
DNSnifferAudit limits crawling to 30 URLs per minute by default. Site owners can crawl their own sites at higher speeds after verifying ownership in the Security Audit tool.
Contacting us
If you have any concerns about how frequently we crawl, get in touch at [email protected].
# inspired by ahrefs/robot
Questions about our crawlers?
We're committed to transparent and responsible crawling practices.
contact crawler team ↵