Senior Data Engineer
Current- Handled and worked on an ETL pipeline to get their drug test exam analytics and generated visualizations for stakeholders
- Migrated Legacy Data from MS SQL Servers to HDFS Developed REST API for internal users to indicate coverage
- Sourced Data from subsidiaries that used different methods to feeds us raw data, such as: Terradata, API’s, SFTP Servers and Landing Pages
- Transitioned Data into AWS Redshift and S3 Buckets Wrote a script to bring data and apply transformations in PySpark code, then automated the process using Airflow DAGS
- Created a Tableau Workbook with different sheets of reports for Stakeholders including public and private data by establishing a connection to a reporting table created by us in Redshift.