As a Senior Data Engineer, I specialize in overseeing and optimizing data movement pipelines to ensure seamless flow and integration of information. With a focus on real-time data integration, I design and implement strategies to capture, process, and distribute data promptly, enabling informed decision-making. My expertise extends to optimizing data storage and query performance, leveraging advanced techniques and technologies to enhance efficiency and scalability. Additionally, I excel in implementing robust data governance frameworks to maintain data integrity, quality, and compliance standards. I prioritize data security measures, implementing robust protocols and strategies to safeguard sensitive information and ensure regulatory compliance. With a proven track record of driving innovation and efficiency in data management processes, I am committed to delivering high-quality solutions that empower organizations to harness the full potential of their data assets.

Possessing extensive expertise, I specialize in designing robust databases and data warehouses, proficiently creating SQL objects. My experience includes building scalable Data Warehouses on Azure using Azure Databricks and Data Factory. I have successfully migrated data from on-premises SQL Server to Azure Cloud SQL Server and collaborated with production support for CDH cluster and data ingestion issues. In the Azure environment, I developed Custom Hadoop Applications, optimized Hive query performance, and transformed queries into Spark transformations. Additionally, I designed and implemented distributed data processing pipelines in the Hadoop Ecosystem using Apache Spark, Hive, Python, and Airflow DAGs, addressing resource management issues and ensuring SLA adherence.

As a Senior Data Engineer, I led the construction of data pipelines on AWS EMR, Databricks, and Palantir Foundry, optimizing Hive, Spark applications, and NoSQL databases. Utilizing Docker, Kubernetes, and AWS services, I enhanced data transfer efficiency between AWS S3, NoSQL databases, and Snowflake. Proficient in SQL, Unix, Python, and NoSQL scripts, I conducted ETL processes, data preprocessing, and feature engineering for predictive analytics. Ensuring compliance with healthcare data privacy regulations, I collaborated on seamless integration of EHR systems and NoSQL databases, providing real-time patient information. My role also involved enhancing team productivity through Agile principles and developing data visualization modules within analytics platforms.

• Created consumption views on top of metrics to reduce the running time for complex queries.• Involved in Functional Testing, Integration testing, Regression Testing, Smoke testing, and performance Testing.• Tested Hadoop Map Reduce developed in Python, Pig, and Hive.• As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to HDFS.• Created performance dashboards in Tableau/ Excel / PowerPoint for the key stakeholders.• Implemented Defect Tracking process using the JIRA tool by assigning bugs to the Development Team• Evaluated the traffic and performance of Daily deals PLA ads and compared those items with non-daily deal items to see the possibility of increasing ROI. suggested improvements and modify existing BI components (Reports, Stored Procedures)

• Responsible for requirements gathering and analyzing the data sources. • Loaded (ETL) data from various sources into various targets using Informatica Power Center (Repository Manager, Designer, and Workflow Manager) • Document the complete process flow to describe program development, logic, testing, implementation, application integration, and coding. • Recommended structural changes and enhancements to systems and databases. • Conducted Design reviews and technical reviews with other project stakeholders. • Was a part of the project's complete life cycle from the requirements to the production support.• Created test plan documents for all back-end database modules.• Used MS Excel, MS Access, and SQL to write and run various queries.