• Designed and developed a next generation event processing service capable of scaling to trillions of events per day. This services is currently being enhanced to serve as a generic frontend to Kakfa.• Re-architected the Next Big Sound build system using Gradle, and Nexus.• Re-architected the Next Big Sound deploy system using Jenkins to manage build and code deployment with Mesos/Marathon/Chronos as the execution environment.• Introduced Luigi to manage data pipeline state and improve reliability.• Designed a tool for configuration driven ETL from Hive, via HiveServer2, and writing the output to a variety of locations including S3, MySQL, Redis, HDFS, etc.• Refactored legacy Java infrastructure code with a focus on simplicity and reproducibility.

- Hadoop and HBase cluster configuration- Avro data structure design for core datasets- Wrote Java/MapReduce data ingestion pipeline and data lifecycle management jobs- Wrote Oozie workflows to orchestrate the flow of data from ingestion through analysis

Backend software engineer working with: Python, Django, PostgreSQL, and MongoDB.

Built data management and analysis solutions with Python, MySQL, and Solr.

Managed and automated a 30+ node Hadoop cluster used for processing netflow data from the worlds largest IP network. Wrote map/reduce jobs in Ruby and Perl to enrich and analyze netflow data.Designed and deployed a prototype Hadoop cluster for processing machine generated data such as system and application logs. Wrote Python map/reduce jobs to convert syslog messages into enriched JSON documents for storage and analysis. Led a team of engineers who designed and built the first cloud computing platform deployed within Level 3. Designed and built much of the cloud platforms automation and infrastructure management components.Wrote tools and ETL jobs in Python (and some C, as needed) largely for the automation of a custom CMDB and private cloud computing platform. Led a team of senior architects and engineers in designing and building an internal private cloud computing solution for Level 3 IT.