Data Engineer
Current• Designed and implemented scalable data pipelines using Python and SQL, improving data processing efficiency by 25%.• Developed and maintained ETL processes for data warehousing solutions using tools like Apache NiFi and Informatica.• Implemented data validation and monitoring solutions, reducing data discrepancies and ensuring data integrity.• Migrated on-premises databases to cloud platforms such as AWS and Azure, enhancing data accessibility and scalability while reducing operational costs by 15%.• Extensively utilized AWS services including EC2 for virtual server management, S3 for scalable storage solutions, and Redshift for data warehousing and analytics, ensuring efficient and scalable data engineering processes.• Collaborated with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.• Conducted performance tuning of SQL queries and database systems, achieving a 30% reduction in query execution time.• Created Python-based scripts for efficient data ingestion into AWS RDS and S3, ensuring seamless data flow and integrity.• Enhanced data integrity and data access by using SQL queries on different platforms like Snowflake, redshift, PostgreSQL, and MySQL.• Played a pivotal role in the development and maintenance of CI/CD pipelines using Jenkins, enhancing code deployment efficiency and ensuring seamless integration with GitHub for version control.• Innovated real-time data ingestion solutions by integrating Kafka with Spark Structured Streaming, enabling the capture and processing of streaming data for immediate insights.• Ensured data governance and compliance by implementing metadata management practices in Databricks, documenting data lineage, quality rules, and audit trails.• Enhanced data-driven decision-making by applying machine learning techniques with TensorFlow, resulting in a 20% improvement in predictive analytics accuracy.