Why do different DNS Abuse measurement projects result in different numbers?
Measuring DNS Abuse—however you define it—is hard and complicated, but why do reasonable minds reach different conclusions on the numbers?
We hear this question frequently since the DNS Abuse Institute ("Institute") developed our measurement initiative: DNSAI Compass ("Compass"), a collaboration with KOR Labs.
This blog is a condensed overview from our full report "Why do different DNS Abuse measurement projects result in different numbers" and is meant to create a greater awareness of how DNS Abuse is measured and help the community to understand and interrogate data presented to them. It also highlights the importance of having transparent methodology.
How is DNS Abuse measured?
A measurement project should be able to provide you with details on how they reach their final numbers and explain decisions that were made along the way: which source lists were used, how was the data cleaned, high level details of the analysis, and guidance on how the data has be presented to help you understand and interpret the information provided.
Projects to measure DNS Abuse typically follow a process of aggregating, cleaning, analyzing, and presenting data from multiple source lists.
Each step presents several decisions that can influence the outcome of the project. How these decisions are made will be influenced by the purpose and focus of the research as well as any priorities that have been identified. Finally, care must be taken in how to interpret data.
We've compiled a report which dives into more detail on each of these steps and also explains how they apply to our measurement project: Compass.
The Institute explored this topic in detail with the ccNSO DNS Abuse Standing Committee (DASC) at ICANN78. Slides are available here. This blog post is a summary of my contributions to that session.
For more information on the Compass methodology, see our published methodology and our PDF reports.
Compass provides free Dashboards for registries and registrars to understand the prevalence and persistence of phishing and malware in the domains they manage. If you'd like to access your own data, or meet with the Institute in person at ICANN79 or virtually, please email: support@DNSAbuseInstitute.org.