Data Engineer
Current- Responsible for developing, monitoring, and maintaining new and legacy data pipelines, develop new integrations and data exports.
- Optimization of pipeline and platform queries, reducing pipeline query processing time from 50 minutes to 40 seconds.
- Using daily PostgreSQL, Clickhouse, Python, Airflow, SQL, SFTP, Bash, along with AWS services such as Lambda, Redshift, RDS, S3, EC2, SNS, SQS, IAM, AWS CLI.
- Developed a data integration process for the Customer Data Platform (CDP) integrating more than 20 external partners with data and 2,000 + files ingested daily, becoming a service widely used by customers.
- Successfully migrated data pipelines from AWS Datapipeline service to Airflow.