DS-0948

Building a DGA Classifier: Part 1, Data Preparation

External Dataset

External Data Source

Data Driven Security

Unknown

11/14/2007

56 (lowest rank is 56)

Other

cyber attack, botnet, malware, simulated attacks, cyber crime, network data

Unrestricted

Unknown

Data Use Terms

Short Description
Dataset used for creating a DGA (Domain Generating Algorithm) classifier, a deterministic generator of random (hard to guess) domains, of which the bot maintainer only has to register one (or a handful) to enable the malware to phone home. If the domain or IP is taken down, a new name can be used by the botnet maintainer with a new IP address.

Long Description
The purpose of building aDGAclassifier isn't specifically for takedowns of botnets, but to discover and detect the use on our network or services. If we can you have a list of domains resolved and accessed at your organization, it is possible now to see which of those are potentially generated and used bymalware.

The dataset consists of three sources (as decribed in the Data-Driven Security blog):

Alexa: For samples of legitimate domains, an obvious choice is to go to the Alexa list of top web sites. But it's not ready for our use as is. If you grab thetop 1 Million Alexa domainsand parse it, you'll find just over 11 thousand are full URLs and not just domains, and there are thousands of domains with subdomains that don't help us (we are only classifying on domains here). So after I remove the URLs, de-duplicated the domains and clean it up, I end up with the Alexa top965,843.

"Real World" Data fromOpenDNS: After reading the post from Frank Denis at OpenDNS titled"Why Using Real World Data Matters For Building Effective Security Models", I grabbed their10,000 Top Domainsand their10,000 Random samples. If we compare that to the top Alexa domains, 6,901 of the top ten thousand are in the alexa data and 893 of the random domains are in the Alexa data. I will clean that up as I make the final training dataset.

DGAdo: The Click Security version wasn't very clear in where they got their bad domains so I decided to collect my own and this was rather fun. Because I work with some interesting characters (who know interesting characters), I was able to collect several data sets from recent botnets: "Cryptolocker", two seperate "Game-Over Zues" algorithms, and an anonymous collection of malicious (and algorithmically generated) domains. In the end, I was able to collect 73,598 algorithmically generateddomains.
;

External URL
http://datadrivensecurity.info/blog/posts/2014/Sep/dga-part1/

N/A

false

false

dga, classifier, building, building a dga classifier: part 1, data preparation, preparation, 948, corporation, source, external data source, external, inferlink, inferlink corporation, domains, maintainer, domain, dataset, random, register, phone, creating, deterministic, malware, algorithm, handful, hard, generating, generator, bot, botnet, enable, guess, alexa, top, security, collect, list, urls, 000, domainsand, generated, botnets, clean, characters, samples, real, algorithmically, their10, decided, zues, detect, bymalware, subdomains, 598, sources, game, dont, grabbed, cryptolocker, final, 843, accessed, training, 901, consists, seperate, dgado, matters, parse, reading, youll, bad, ready, decribed, malicious, effective, generateddomains, duplicated, network, remove, purpose, ten, models, obvious, organization, takedowns, services, sets, top965, resolved, web, driven, 893, legitimate, denis, other, compare, algorithms, isnt, anonymous, choice, thousands, discover, fromopendns, thetop, post, opendns, blog, adgaclassifier, sites, click, titledwhy, de, version, fun, classifying, grab, frank