Dev Ops Data Engineer
● Developing data pipeline project for Fraud Department who saves Charter $100 million per yearProvided engineering support to Mobile 2.0, a multi-year migration initiative to migrate legacy data stores to an on-prem Hadoop cluster ○ Meets with business stakeholders to gather business requirements and report tasks’ statuses ○ Reworks existing queries to use different sources and targets - adjusts logic accordingly ○ Tests ETL processes and data extensively by validating data mapping and population of tables● Deliver data to support the fraud detection analytics team ○ Analyzes new data sets to construct preliminary SQL queries for team needs ○ Creates Hive tables for data sets in Fraud team’s database ○ Configures existing shell scripts and PySpark files to load data into HDFS ○ Develops and coordinates deployment of new sqoop jobs with the scheduling team● Supports existing data and jobs ○ Investigates data/job load issues to find root causes by following SQL/shell script logic ○ Corrects objects, using Hive and Impala, to better suit the needs of clients. ○ Uses PuTTY and WinSCP to modify files and scripts on remote servers and HDFS ○ Record causes of issues so similar problems can be identified and resolved swiftly in future