Machine Learning Engineer
CurrentDesigned and implemented data warehousing solutions to store and manage large volumes of structured and unstructured data efficiently.Built orchestration pipelines using Apache Airflow to automate data workflows, ensuring data consistency and reliability across systems.Configured Apache Airflow DAGs (Directed Acyclic Graphs) to schedule and monitor data pipeline tasks, including data extraction, transformation, and loading (ETL).Integrated MS SQL Server as the primary database backend for storing and querying structured data, ensuring data integrity and security. Implemented ETL processes within Apache Airflow workflows to extract data from various sources, transform it using SQL transformations, and load it into MS SQL Server databases.Utilized Cloud storage with Amazon S3 as a cost-effective solution for storing and managing large datasets, ensuring scalability and durability. Incorporated Apache Airflow operators for interacting with S3 buckets, including uploading, and downloading data files, as well as performing data processing tasks directly on S3 objects.Designed and implemented dimensional modeling techniques within the data warehouse to optimize query performance and facilitate efficient data analysis. Collaborated with data scientists and analysts to define data pipeline requirements and optimize workflows for machine learning model training and inference.Implemented data governance and security measures within Apache Airflow, MS SQL Server, and S3 to protect sensitive data and ensure compliance with regulatory standards.Conducted performance tuning and optimization of Apache Airflow DAGs, MS SQL Server databases, and S3 storage configurations to improve pipeline efficiency and reduce latency.Worked closely with stakeholders to gather requirements, define data architecture, and implement solutions that meet business objectives.