Automation and Observability @ EdgeCast CDNVerizon Digital Media Services -> Yahoo -> Oath -> EdgioEdgecast (AS15133): 137Tbps capacity, with 175+ pops, 6 continents, 7k+ interconnects.Edgecast CDN Infrastructure Automation Team- Designed and deployed workflow automation system for 20K+ servers globally, reducing operational workload and downtime. Python, Stackstorm and Orquesta.- Developed a specialized policy engine to safely limit automation requests. Python, Flask, Postgres, Redis.- Created various workflows based on current manual procedures saving many hours of labor per day. Gracefully removes and safely returns servers to production.- Increased operational efficiency by automating manual procedures allowing existing employees to support the growing CDN.- Automated hardware failure detection and remediation, enhancing system uptime and reducing human intervention during failures.- Implemented queued requests for off-peak execution. Integrates with canary style rollout.- Facilitated the server provisioning and patching workflows to rapidly/safely update all servers.- Deployed system to restricted PCI compliant network.- Provided interface via ChatOps, REST API and command line interface for 24/7 NOC and internal users.- Worked with monitoring tools such as collectd, TSDB, Grafana, and Cortex to ensure comprehensive metric collection and visualization.Developed automation system as engineer and Team Lead, manage task priorities to successfully meet goals a quarter in advance, participate in interviews."They are very much further ahead of us in automation."- Bob Lyons, CEO of Limelight Networks, on the benefits of acquiring Edgecast.

Assisted design and development of queueing system for a large render farm.- Created flexible GUI tool to efficiently manage and interact with the queue.- Created service that runs on and controls every render node.- Co-developed API used by new python developers and experts.- Competed for an Academy Scientific and Technical Award.- Current lead developer on project.- Python, PyQt, Java, Oracle, Tomcat, Subversion, Git.Rapidly design and deploy business critical solutions: interoffice voice chat, i/o pipelines, remote application deployment, windows renderfarm.Backup developer for many applications. Maintain orphaned projects after each headcount reduction, including improving reliability and performance.Contribute features, plugins and support to in-house media player called Itview.Application and pipeline optimization to allow renderfarm expansion.Transitioned facility to new customized Linux distribution.- First transition was based on Foresight. (conary)- Second transition was based on Fedora 13. (rpm)- Currently working on CentOS 7. (rpm)- Assisted with planning, application migration, building, deployment.- In-depth application and pipeline debugging.- Developed new tools and methodologies to maintain 1000s of computers.Troubleshooting wide range of issues, software to hardware, under pressure. 24/7 on call support for render farm and operating system as well as for debugging other show stopper issues. Facilities in California, Vancouver, New Mexico, Bristol and India.Credited on many feature films. http://www.imdb.com/name/nm2814797/Team lead, communicate with all levels of production/management, conduct interviews.

Credited: Beowulf* Scripting tools with Python to automate common tasks and improve workflow.* Troubleshooting a wide variety of technical problems.* Render queue management on 7500+ core data center.* Communication with Digital, Feature Production, Editorial, I/O, Systems and Software departments.* File/image processing and manipulation.* Disk resource management. Data archiving and restoration.