This is a non-IMPACT record, meaning that access to the data is not controlled by IMPACT. For access, see the directions below.

Disclaimer:
This Resource is offered and provided outside of the IMPACT mediation framework. IMPACT and the IMPACT Coordination Council/Blackfire Technology, Inc. expressly disclaim all conditions, representations and warranties including but not limited to Resource availability, quality, accuracy, non-infringement, and non-interference. All Resource information and access is controlled by entities and under terms that are external to the IMPACT legal framework.

Summary

DS-0948
Building a DGA Classifier: Part 1, Data Preparation
External Dataset
External Data Source
Data Driven Security
Unknown
11/14/2007
55 (lowest rank is 55)

Category & Restrictions

Other
cyber attack, botnet, malware, simulated attacks, cyber crime, network data
Unrestricted
Unknown

Description


Dataset used for creating a DGA (Domain Generating Algorithm) classifier, a deterministic generator of random (hard to guess) domains, of which the bot maintainer only has to register one (or a handful) to enable the malware to phone home. If the domain or IP is taken down, a new name can be used by the botnet maintainer with a new IP address.

The purpose of building aDGAclassifier isn't specifically for takedowns of botnets, but to discover and detect the use on our network or services. If we can you have a list of domains resolved and accessed at your organization, it is possible now to see which of those are potentially generated and used bymalware.

The dataset consists of three sources (as decribed in the Data-Driven Security blog):

Alexa: For samples of legitimate domains, an obvious choice is to go to the Alexa list of top web sites. But it's not ready for our use as is. If you grab thetop 1 Million Alexa domainsand parse it, you'll find just over 11 thousand are full URLs and not just domains, and there are thousands of domains with subdomains that don't help us (we are only classifying on domains here). So after I remove the URLs, de-duplicated the domains and clean it up, I end up with the Alexa top965,843.

"Real World" Data fromOpenDNS: After reading the post from Frank Denis at OpenDNS titled"Why Using Real World Data Matters For Building Effective Security Models", I grabbed their10,000 Top Domainsand their10,000 Random samples. If we compare that to the top Alexa domains, 6,901 of the top ten thousand are in the alexa data and 893 of the random domains are in the Alexa data. I will clean that up as I make the final training dataset.

DGAdo: The Click Security version wasn't very clear in where they got their bad domains so I decided to collect my own and this was rather fun. Because I work with some interesting characters (who know interesting characters), I was able to collect several data sets from recent botnets: "Cryptolocker", two seperate "Game-Over Zues" algorithms, and an anonymous collection of malicious (and algorithmically generated) domains. In the end, I was able to collect 73,598 algorithmically generateddomains.
;

Additional Details

N/A
false
false
cybercrime, spamming, alexa internet, bot, building a dga classifier: part 1, data preparation, blog, external data source, history of the internet, domain name system, website, malware, inferlink corporation, opendns, machine learning, cyberwarfare, data preparation, training test and validation sets, ip address, cyberattack, cryptanalysis, subdomain, cryptolocker, botnet, exploit, 948