DomainMetaDataBot
You most likely landed here because you saw DomainMetaDataBot in your web server logs
and want to know what it is. This page explains who we are, what we fetch, and how to control or block us.
What is DomainMetaDataBot?
DomainMetaDataBot is the web crawler operated by DomainMetaData. We visit public web pages to collect metadata about internet domains — technologies in use, language, topic, and a few other signals. The result feeds a publicly available database of internet domains. Learn more on How it works.
How to verify it is really us
Our crawler identifies itself with the following HTTP User-Agent header:
DomainMetaData/1.0 (+https://domainmetadata.com/bot)Requests originate from our own servers. Spoofing the User-Agent is trivial, so please also check the source IP address against the list below before blaming us for traffic you see:
57.131.13.3357.128.223.715.235.187.136148.113.44.188
All crawler traffic originates from one of these addresses. Anything else using our User-Agent is not us.
How to block or limit us
We respect robots.txt. To stop DomainMetaDataBot from crawling your site, add the
following to the robots.txt at the root of your domain:
User-agent: DomainMetaDataBot
Disallow: /
We re-check robots.txt on every crawl and apply changes the next time we visit.
You can also allow specific paths only, or set a Crawl-delay value — we honor both.
Note: blocking us via robots.txt only stops future crawls. Metadata we already
collected stays in our database until the next crawl. If you want your domain removed
immediately, contact us using the details below.
What we fetch — and what we do not
- We fetch the public HTML of your homepage and a small number of linked pages.
- We do not execute JavaScript, submit forms, or follow links behind login walls.
- We do not send
POSTrequests or attempt any write operations. - We crawl each domain infrequently — typically a handful of requests per visit, a few visits per year.
- We do not store copies of your content. We extract structured metadata only.
Legal basis and data protection
We only process publicly accessible information. Our legal basis under GDPR is legitimate interest (Art. 6(1)(f) GDPR) in maintaining a public database of internet domains. No personal data is collected from the crawled pages. See our Privacy Policy for details.
Questions, complaints, or removal requests
Write to our contact form or email abuse@domainmetadata.com. We respond within one business day and honor removal requests immediately.