Data Engineer
Current- Designed and implemented 8+ ETL pipelines, sourcing data from multiple origins to staging areas, improving data availability for analytical reporting and BI dashboards with the help of Python and SQL languages.- Diagnosed and resolved pipeline issues, increasing scalability and optimize performance by 30%, ensuring 99% data pipeline reliability.- Integrated incremental loading techniques, reducing resource consumption up to 80% and improving data synchronization speed by 30%.- Developed and fine-tuned ETL processes to enhance overall efficiency and performance.- Implemented an ETL logging system to track pipeline status, improving bug resolving time by reducing it to 50% less then previous time.- Executed a seamless data migration from PostgreSQL to MS SQL Server, improving access management and system integration.- Successfully Implemented Apache Airflow on standalone ETL pipelines for better scheduling and monitoring.