Christopher Hicks Email & Phone Number

San Francisco, Ca, Us

* prioritizing observability projects and getting buy-in from our management* coordinating international contributors for observability projects* worked with key vendors to prioritize features we wanted* launched service catalog* documented processes and coordinated standardization of schemas

San Francisco, Ca, Us

Managing Kubernetes clusters in AWS with terraform, ansible, rancher, datadog, and kustomize.Projects include:* building coredns monitoring with datadog.* configuring VPC peering in AWS with terraform* updating datadog terraform provider* completed migration out of shared namespaces in kubernetes* onboarded our services onto the Datadog Service Catalog* evaluated and rejected AWS Budgets for improving our cost management

Newport News, Va, Us

FINI has been my consulting firm for most of my working life. We have completed projects for Case Corporation, Wheat First Securities, the National Weather Service, and many others. I developed and provided extensive training in UNIX, Linux, Perl, and Java for US News and World Report, Capital One, Circuit City, all branches of the military, the Federal Reserve, and Newport News Shipbuilding.

Mountain View, California, Us

Projects:* Tagged AWS resources to follow corporate standards using terraform. Wrote tag compliance scanner in python.* Moved S3 access logs to subdirectories using python and boto3.* Built jenkins infrastructure in ECS using terraform.* Built TeamCity infrastructure in ECS using terraform and CloudFormation.* Built Grafana and Prometheus infrastructure using terraform.* Built docker images for Jenkins and TeamCity agents in multiple versions of Ubuntu LTS using bash..* Migrated subcommands to next generation of internal command line tools.Technologies:* Python, bash, github* Terraform, CloudFormation* AWS IAM, ECS, RDS, EC2, S3, R53, Secrets Manager

San Francisco, California, Us

* Wrote tools for managing Elasticsearch clusters in OpsWorks. Wrinkles automated away include handling incompatible AMI versions and multiple OpsWorks stacks. Documented using asciinema to build animated GIFs. The GIFs quickly showed how commands looked that took 1-20 hours to run in real time.* Evolved process for Elasticsearch capacity management. After conferring with all engineering teams additional metrics were sent to DataDog to create a full view of performance issues. Even early iterations of this led to a major reduction in customer-visible downtime.* Wrote Go and bash for managing weekly compactions on Cassandra and ScyllaDB clusters. Included Slack notifications.* Tested migrating ScyllaDB clusters to require authentication. Found errors in vendor docs and filed github issue..* Security was my initial focus area:+ Built process for quarterly internal vulnerability scans and wrote reports with remediation suggestions+ Evaluated AWS Security Hub, Inspector, Trusted Advisor and GuardDuty+ Mitigated a wide variety of Trusted Advisor suggestions+ Scanned AWS IAM for customers and staff with old keys.+ Rotated out 50 ES nodes that had gotten public IPs.+ Reviewed security incidents and adjusted settings to improve signal-to-noise ratio (SNR).+ Monthly Security Review meetings for compliance.+ Helped evolve the process for customer offboarding.* Used boto3 to track inventory of services running in AWS ECS. Evaluated netbox and DeployHub for UX to the inventory.* Developed process for reviewing rollbar usage and ran meetings.* Wrote daemon to pull metrics from redis instances in OpsWorks and Elasticache and send them to DataDog and Prometheus.* Mentored DevOps engineers on our India team.* Wrote a dozen incident reports and ran those post mortems.* Ran monthly monyog review to avoid swamping RDS/mysql.

San Francisco, Ca, Us

* added packaging to buildkite CI pipelines for internal product and librdkafka using fpm and bintray* discovered method for deploying pipenv-based projects to production for air-gapped deployments. I then added it to buildkite, and wrote a script for verifying that all modules were available inside of the pipenv.* rewrote production deployment bash scripts to use packages and modern shell constructs* created new packaging format including manifests, component verification and signature verification. These kitchen-sink packages were stored in AWS S3 using an IAM key and role that I created.* wrote and delivered class on Debian packaging for software engineers* implemented pre-commit hooks for multiple python linters to reduce round-trips to buildkite for minor tweaks. This had been wasting hours per week for some members of the team after engineering management mandated moving to a monorepo..* assisted with onsite product installations for customers in the national security industry

San Francisco, Ca, Us

* automated proxmox VM creation using Perl because this was burning up so much time for the team* I trained our outsourced oncall vendor in how to deal with our PagerDuty L1. Pythian took my notes and wrote a surprisingly good runbook which has continued to evolve. Yet Pythian could not execute on the actual oncall. I didn't think they were ready for that from what I saw with shadowing, but management was using Pythian for other things and insisted we put them in for live L1 pages. After a few months management could see that I was essentially covering for them to ensure no major outages and we removed Pythian from this responsibility. I'm sorry this didn't work out with this vendor, but I don't understand why we didn't try with another vendor. With the runbook and our experience from the first time it might have gone easier.* deployed servers and wrote cookbooks in chef* managed DNS using FreeIPA led to the creation of a Perl wrapper available at https://github.com/chicks-net/p5-net-ipa-lite that was used in our VM creation process* wrote build scripts in jenkins for customized ISO generation and Debian package builds with fpm* wrote chef cookbook for deploying datadog to selected parts of our infrastructure to ease cash flow* maintained OMD/check_mk and PagerDuty infrastructure as new checks and hosts were added as well as when group refactoring was needed to match new organizational structures* assisted with Chef upgrade where we decided Chef Automate wasn't ready for mortals to use yet* wrote script to scrape JSON http endpoints and send any metrics to datadog. https://github.com/Telmate/json2dog* built dashboards and alerts in datadog for Kurento Media Server-based video visitation system* assisted with Kubernetes deployment to help it fit with the legacy infrastructure* managed vendor relationships with CloudBees, PagerDuty, DataDog, and OpsGenie

San Francisco, California, Us

- built and produced a weekly report of SLA's for realtime dispatch including running averages of service latency and error rate. Also included were host pings, counts and accumulated time for curated production incidents and PagerDuty pages.- created a debian package for haproxy to run on Debian Jessie servers in Google Compute Engine to build international latency reduction infrastructure- reviewed history of curated production incidents to encourage systemic fixes to recurring production issues- deployed Aurora and Mesos using Terraform and puppet on servers in our data centers and cloud providers- improved FAI-based RAID configuration by adding logging

San Francisco, Ca, Us

Focused on Perfect Audience's RTB platform. * Managing servers in AWS with ansible, opsworks, and chef. * Automated adding servers in OpsWorks into EC2-based Elastic Load Balancers (ELBs).* Planning systems in the Amazon Cloud with EC2, ELB, S3, EBS, Elastic Beanstalk, OpsWorks, IAM, CloudWatch, RDS, DynamoDB, VPC, Redshift, SNS, and SQS* Configuring monitoring with Nagios, graphite, and Scout. * Making architecture diagrams with Graphviz.* Prioritizing operations projects to maximize strategic benefits* Supporting junior operations personnel and developers with ad-hoc education in TCP/IP networking, bash, and the POSIX/Linux process model

San Francisco, Ca, Us

- server deployments in domestic and international data centers - maintaining vendor relationships with Equinix, Silicon Mechanics, Chatsworth, Anixter, and ServerTech- wrote a Perl script for mapping physical drives to their Linux designations for LSI MegaRaid cards https://github.com/chicks-net/megamap- wrote a Perl script for culling outlet power statistics from SeverTech Sentry PDU's which I later extended with a python equivalent. https://github.com/chicks-net/sentry_cdu_dumper- wrote a script for summarizing AWS usage for accounting and created a puppet manifest to deploy it- upgraded to haproxy 1.5 for SSL termination including updating puppet templates to conform to current config standard- coordinated migration of mysql database within AWS including getting developer buy-in to implement read-write splitting

New York, New York, Us

* rewriting icinga config generator* NetApp administration: adding shelves, configuring mirrors, applying space to aggregates, adding space to volumes

Pasadena, Ca, Us

- responsible for application and systems administration for 5000 servers on three continents in five data centers.- automating Nagios and Munin configuration. Reduced unactionable alerts and eliminated manual interactive work for pushing changes.- customer creation including Akamai configuration, DNS, and internal systems. The customer creation process for the 3.0 product was entirely my design and creation. Automation has reduced errors and improved completion times.- pager rotation including configuring PagerDuty to accept alerts from new subsystems and scheduling a multi-tier pager rotation.- talent acquisition coordination. I was solely responsible for the SRE pipeline before OpenX hired a full-time recruiter. I was the initial filter for SRE through phone interviews and in person interviews conducted at a recruiter's office in speed dating fashion.- training new SRE's to use gomez, akamai, jira, nagios, and our internal metrics and alerting systems

Sunnyvale, Ca, Us

Production Engineers at Yahoo! are systems administrators in operations. I managed 100,000 RHEL boxes with a very custom Perl-based management system to power Yahoo!'s search platform. Our server:admin ratio was 8000:1. I handled DNS updates for domains that were key to search operations. My final role was as Release To Production Coordinator (RTPC) which entailed ensuring that approved updates from thousands of developers made it to the appropriate tens of thousands of servers. I was the first RTPC to provide an organized transition to the team that replaced me. I did Perl training for Yahoo! administrators and developers as well as coordinating a training program that included a variety of developers and operations personnel. I also assisted with talent acquisition by creating training events at colleges and conferences as well as interviewing and improving prescreening questions.

I helped expand Widomaker from a one man shop into the dominant ISP in the Williamsburg, VA area. We transitioned from UnixWare to Linux and POTS-based 28.8kbps dialup to PRI-based 56k dialup. I was involved in all aspects of operations including web development, consulting, and phone support.

Oldsmar, Florida, Us

I started as a C programmer. Soon after I began handling systems administration of a variety of Sun Sparcstations used for software development. I handled integration work with other software vendors including onsite support and debugging. Eventually I became the General Manager managing programmers and systems administrators in a 24-5 data processing environment.