Software Engineer Iii
Pasadena, Ca, Us
Senior member of an engineering team tasked with building tools to allow DevOps to manage over 5000 bare metal systems, hypervisors, and virtual nodes across 5 global datacenters.Helped implement service and host level monitoring using a mix of Mondemand, Graphite/Grafana, and Icinga.Used a stack of Modern Perl (Moose, Catalyst, DBIx, etc.), MySQL, Javascript, and Python to stitch together various APIs for configuring bare metal nodes, switches, and load balancers via loosely coupled RESTful services.Our tooling allows internal customers to quickly provision a node, deploy it with assigned responsibilities, monitor its health, emit service level metrics, and visualize these various steps. I led early integration efforts between our infrastructure's metadata system and SaltStack and Git.I developed and maintained web front-ends for various visual tools using jQuery and Perl's Dancer framework.I helped implement our continuous integration/deployment with GoCD after having supported the scripting of our release process via Jenkins and Bamboo.I scripted the reactive generation of Nagios/Icinga configuration based on changes made to our nodes duties.Other points of interest: - Performed RPM packaging for RedHat based distributions - Developed mirroring and patch management solutions on behalf of security engineers - Implemented a trial integration with a PingFederate SSO solution - Developed minor tooling managing Elasticsearch snapshots - Experience using Jira and various Atlassian tools. - Learned very basic Erlang while maintaining our proprietary monitoring system