Data Analytics And Collections Lead
Denver Co
As the Data Analytics and Collections Lead for Dark Owl, I did all of the Machine Learning, Natural Language Processing, Pattern Detection, Search Optimizations, Data Indexing, Data Retrieval and Graph Modeling. Additionally, I developed the collectors for I2P, IRC and FTP while supervising/leading the development of collection for ZeroNet and Authenticated Crawl.With Respect to ML, NLP and Advanced Analytics, below are a few examples of the technology that developed for Dark Owl.1.) "MapTheDark": A daily summary of the dark domains that are OnLine, the size of the domains, the "topics" that the domains deal with, the languages contained and other "derived conclusions" about the domains.2.) Compute Document Similarity via the Jaccard Similarity Coefficient using Shingle-MinHash-LSH (Locality Sensitive Hashing). The document similarity computations are used to ensure that a user does not see lots of similar documents returned in their query results.3.) Compute "Hackishness" via Naive Bayes. Hackishness is the OWL Vision determination of how likely a document is to contain information that could be used for Nefarious Activity.4.) Determine the languages contained in a document using Ngram character combinations matched against Language Profiles. 5.) Determine where the Engine should crawl/scrape via a MapReduce based Scheduling Algorithm, using an adaptive, dynamic, realtime feedback loop.6.) Developed a Neo4j Graph model of the Dark Webs based on data collected and analyzed over the course of 4 years