Data Engineering Analyst
Current- Working on migrating the client's Hadoop big data infrastructure to Cloud.
- Creating 800+ ELT pipelines in Python on Apache Airflow.
- Consuming data from various sources such as FTP/SFTP, SAP, Google BQ, Salesforce, Veeva Vault, etc. and ingesting it in Snowflake tables.
- Performing transformations using SQL and creating DBT models.
- Creating Snowflake UDF/UDTF, DBT macros for minor data manipulations.
- Performing complex transformations and manipulations using Spark on EMR.