Data Engineer Ii
Current- Designed and implemented robust data pipelines using Apache Hudi, Apache Iceberg, and Delta Lake to ensure data consistency, reliability, and scalability.- Orchestrated ETL jobs with AWS Glue to efficiently transform and load data from raw to gold layers.- Leveraged Redshift for advanced analytics and data warehousing, optimizing query performance and storage costs.- Built and managed Microsoft Fabric cloud environments, creating data lakehouses to centralize data assets.- Collaborated with data scientists and ML engineers to integrate ML models into the data pipeline, enabling real-time insights and predictions.- Implemented data quality checks and validation mechanisms to maintain data accuracy and integrity.- Worked effectively with cross-functional teams, including data analysts, data scientists, and business stakeholders, to understand requirements and deliver solutions.