• Implemented SQL scripts to generate UAT metrics reports for the Teradata-to-Snowflake migration, leveraging Python to automate email alerts, ensuring the timely delivery of actionable insights.• Facilitated the transfer of structured data from Snowflake to AWS S3 through automated pipelines, ensuring high data quality and compatibility for machine learning model consumption.• Utilized shell scripting to automate S3 file manipulations, optimizing data structuring and organization for efficient processing in downstream workflows.• Converted existing Spark and Scala-based architecture code to a SQL-based, metadata-driven framework, streamlining the redesign and improving maintainability.• Utilized multiprocessing techniques to optimize data processing workflows, significantly reducing execution time for large-scale data tasks by parallelizing computationally intensive operations and enhancing overall system performance.• Leveraged AWS Lambda for event-driven architecture, automating data pipeline tasks, including data validation, transformation, and triggering downstream processes.• Developed a robust data quality framework supporting various metric functions and validations, implemented using stored procedures and designed for seamless integration across the data ecosystem.• Implemented comprehensive data quality checks across multiple use cases and data sources to ensure data accuracy, consistency, and reliability for critical business operations.• Utilized AWS CloudWatch for comprehensive monitoring and logging of data pipeline performance, enabling proactive issue resolution and system optimization.• Implemented AWS Lookout for anomaly detection in ad impressions, identifying unusual patterns, and enhancing data quality to improve ad performance and reporting accuracy.• Designed Airflow DAGs to orchestrate workflows, ensuring efficient scheduling, monitoring, and error handling for seamless data pipeline execution.

• Involved in writing Spark applications using Python to perform various data cleansing, validation, transformation, and summarization activities according to the requirement.• Developed multiple POCs using PySpark and deployed them on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata, and developed code in reading multiple data formats on HDFS using PySpark.• Loaded the data into Spark data frames and performed in-memory data computation to generate the output as per the requirements.• Worked on AWS Cloud to convert all on-premise, existing processes, and databases to AWS Cloud.• Designed and developed ETL processes in AWS Glue to migrate campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.• Used AWS Redshift, S3, Redshift Spectrum, and Athena services to query large amounts of data stored on S3 to create a virtual data lake without having to go through the ETL process.• Developed PySpark job to load the CSV files into the S3 buckets and created AWS S3 buckets, performed folder management in each bucket, and managed logs, and objects within each bucket.• Involved in file movements between HDFS and AWS S3 and extensively worked with S3 bucket in AWS.• Worked on importing metadata into Hive using Python and migrated existing tables and the data pipeline from legacy to AWS cloud (S3) environment and wrote lambda functions to run the data pipeline in the cloud.• Extensively worked with partitions, dynamic partitioning, and bucketing tables in Hive, designed both managed and external tables, and worked on the optimization of Hive queries.• Created generic Databricks notebooks for data cleansing.• Integrated Snowflake with other data analytics and BI tools such as PowerBI for effective data visualization and reporting.• Devised complex data architectures in Snowflake, including schema design, and table structures.• Maintained Snowflake databases, tables, and schemas for optimized performance.

• Worked with business users, business analysts, program managers, project managers, system analysts, and quality assurance analysts to review business requirements.• Created technical specification documents based on business requirements and collaborated with business analysts and other developers to discuss design and architecture. • Designed, developed, and created ETL (Extract, Transform, and Load) packages using Python, and SQL Server Integration Services (SSIS) to load data into a data warehouse (Microsoft SQL Server), from Excel workbooks and flat files into a database. • Implemented an application for cleansing and processing terabytes of data using Python and Spark.• Imported data from AWS S3 into spark RDD and performed transformations and actions on RDD.• Worked on AWS Cloud to convert all on-premise, existing processes, and databases to AWS Cloud.• Developed packages using Python, Shell scripting, and XML to automate some of the menial tasks. • Used Python to write data into JSON files for testing student item-level information. • Created scripts for data modeling.• Used Amazon CloudWatch to monitor and track resources on AWS.• Experience in deploying, configuring, and managing Hadoop clusters using AWS EMR, including customizing instance types, security settings, and software packages.• Proficient in writing AWS Kinesis applications using SQL-like queries to analyze streaming data in real-time and generate insights.• Demonstrated expertise in working with Apache Flume to ingest and transfer large volumes of data from various sources to Hadoop-based systems.• Proven ability to design, configure, and optimize Flume agents for efficient and reliable data ingestion.• Read the data from various Kafka topics and processed the data using Databricks.• Familiar with using Airflow's web UI and CLI to monitor pipeline executions, track data lineage, and manage DAGs.• Optimized queries by adding necessary non-clustered indexes and covering indexes.

• Responsible for developing a highly scalable and flexible authority engine for all of Groupon's customer data. • Worked on resetting customer attributes that provide insight about the customer, purchase frequency, marketing channel, and Groupon deal categorization. • Advocated different sources of data using SQL, Hive, and Scala.• Fetched gender, age, and other purchase histories from 3rd party data agencies and tried to integrate that data into the existing data store.• Used Kafka HDFS Connector to export data from Kafka Topic to HDFS files in a variety of formats and integrated with Apache Hive to make data immediately available for SQL querying.• Normalized the data according to the business needs by modifying the data types, performing data cleansing, and various transformations using Spark and Scala.•Implemented dynamic partitioning in BigQuery tables and used appropriate file format, and compression techniques to improve the performance of PySpark jobs in GCP Dataproc.• Built a system for analyzing the column names from all tables and identifying personal information columns of data across on-premise databases to GCP.• Processed and loaded bound and unbound data from Google Pub/Sub topic to BigQuery using Google Cloud Dataflow with Python.• Worked with Informatica Cloud for data integration projects, including cloud-to-cloud and cloud-to-on-premise integration.• Developed and maintained automated build, test, and deployment pipelines using industry-standard CI/CD tools such as Jenkins, Travis CI, or CircleCI.• Knowledge of Oozie workflow configuration files and parameters.• Designed and developed job flow using Apache Airflow in GCP.

• Collected business requirements from users and translated them into technical specifications and design documents for development.• Proficient in designing and implementing ETL workflows using industry-leading tools such as Informatica PowerCenter, Microsoft SSIS, and Talend.• Hands-on experience in managing metadata, creating and executing test cases, and implementing data lineage and governance using Talend Metadata Manager.• Designed and developed complex ETL mappings using Informatica PowerCenter to extract, transform and load data from various sources into target systems.• Responsible for creating ETL packages using SQL server integration services (SSIS).• Developed logging for ETL load at the package level and task level to log the number of records processed by each package and each task in a package using SSIS.• Responsible for deploying, scheduling jobs, alerting, and maintaining SSIS packages.• Designed SSIS packages using lookup transformations, merge joins fuzzy lookups, and derived columns with multiple Dataflow tasks.• Created complex stored procedures, triggers, functions, indexes, tables, views, and other SQL joins for applications following SQL coding standards.• Supported and maintained the existing SSRS reports and was responsible for source code fixes.• Created the reports and data extracts from the SQL server using Excel pivot functionality as per user requirements.• Worked on query optimization techniques for complex SQL hints statements. • Used Performance Monitor and Explain Plan for performance tuning to enhance the performance of databases.• Proficient in writing unit testing code using Unit Test/PyTest and integrating the test code with the build process worked on Python Testing frameworks.