Data Engineer
Current• Responsible for developing, monitoring, and maintaining new and legacy data pipelines, develop new integrations and data exports.• Optimization of pipeline and platform queries, reducing pipeline query processing time from 50 minutes to 40 seconds.• Using daily PostgreSQL, Clickhouse, Python, Airflow, SQL, SFTP, Bash, along with AWS services such as Lambda, Redshift, RDS, S3, EC2, SNS, SQS, IAM, AWS CLI.• Developed a data integration process for the Customer Data Platform (CDP) integrating more than 20 external partners with data and 2,000 + files ingested daily, becoming a service widely used by customers.• Successfully migrated data pipelines from AWS Datapipeline service to Airflow.