Data Engineer
Current• Automated data extraction using SSIS, reducing manual effort by 30%. Implemented transformations to standardize, clean, and enrich data and load into the centralized data warehouse. • Streamlined data enrichment with Elasticsearch on AWS EMR, resulting in a 25% improvement in retrieval speed for enriched datasets. Leveraged Elasticsearch's indexing capabilities to achieve a 30% reduction in storage overhead, ensuring fast and efficient access to comprehensive data indices. • Automated and scheduled complex data pipelines in Apache Airflow, ensuring seamless transformation and resolving issues with Airflow tools, logs, and corrective actions • Build ackend application using Python libraries FastAPI, SQLAlchemy, NumPy,Pandas resulting in a 25% reduction in response time. Deployed the application on virtual machines using Azure DevOps. • Designed and optimized robust Data Pipelines utilizing Spark, Hudi, EMR cloud services, and Kubernetes containers