How it works

We use multiple ways to discover new domains:

• We use crawlers to scan web pages and analyze the links we find.
• We monitor public zone files for changes. Zone files are lists of domains that are used by DNS servers to resolve hostnames to IP addresses.
• We use machine learning algorithms to predict new domain names based on patterns and trends. After that, we check if those domains have been registered or not.

For each domain that we find, we collect DNS records and other metadata such as creation date. In addition to that, we also parse the content of the website and extract any relevant information such as website title, description, keywords, and the topics of the website.

Naturally, we respect the robots.txt file and do not crawl any pages that are disallowed.