● Designed and implemented ETL pipelines to efficiently extract, transform, and load large volumes of data using Python to migrate from on premises database systems to Bigtable and BigQuery.● Utilized data wrangling Python libraries like Pandas and NumPy for data manipulation and transformation tasks within ETL workflows.● Experienced in optimizing Extract, Transform, Load (ETL) processes using GCP DataFusion to improve data quality, reduce processing time, and enhance overall efficiency of data pipelines● Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow for ETL batch processing to load into BigQuery for analytical processes.● Implemented data quality checks and validations within Apache Airflow DAGs to ensure the accuracy, completeness, and integrity of data loaded into BigQuery.● Led the planning and execution of a large-scale migration project to move data from on-premises Hadoop clusters to GCP Google Cloud Storage ● Optimized table schemas, indexing strategies, and data partitioning techniques to maximize Bigtable's scalability and resource utilization for efficient data storage and retrieval.● Created Spark applications utilizing PySpark in conjunction with Python and Spark SQL in Databricks to modify and aggregate source data before importing it into BigQuery for reporting.● Developed and executed unit tests for ETL code of Python scripts using Pytest, increasing code reliability and maintainability.● Implemented Change Data Capture (CDC) incremental loading patterns in GCP DataFusion pipelines to capture and process only changed or updated data from the source systems, reducing data transfer costs and improving processing efficiency.● Designed and implemented static mapping processes for bulk data movement in GCP DataFusion data pipelines, ensuring accurate and consistent transformation of data from source to target systems.

● Designed and implemented data lake architectures using AWS Lake Formation to centralize and manage diverse data sources.● Implemented data governance and access control policies using AWS Lake Formation, ensuring compliance with security standards.● Migrate data from on-premise databases such as Oracle Database servers to AWS S3 storage buckets.● Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions.● Architected and implemented real-time data streaming pipelines using AWS Kinesis Data Streams, ensuring low-latency data processing.● Leveraged Scala for Spark SQL and DataFrame API to perform complex data transformations and aggregations for analytical purposes.● Developed custom scripts using Boto3 to interact with AWS services, ensuring seamless integration with existing data infrastructure.

● Imported data from sources like HDFS/HBase into Spark RDD. ● Usage of Spark Streaming and Spark SQL API to process the files.● Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice-versa loading data into HDFS.● Worked on Big Data Hadoop cluster implementation and data integration in developing large-scale system software.● Developing UDFs in Java for Hive and Pig and worked on reading multiple data formats on HDFS using Scala.● Developed workflow in Oozie to automate the tasks of loading data into HDFS and pre-processing with Hive.● Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.● Developed analytical component using Scala, Spark, and Spark Streaming.● Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis● Involved in creating Hive Tables, loading with data, and writing Hive queries which will invoke and run MapReduce jobs in the backend.

● Coordinated with front-end application developers for implementing database architecture and design.● Used various transformations in SSIS dataflow, control flow using for loop containers, and fuzzy lookups.● Develop parameterized reports, caching reports, sub reports and Ad Hoc reports using SSRS.● Implemented error handling and utilized event handlers for automated notifications using SSIS.● Write Complex SQL Queries to generate Reports based on the business requirement.● Redesigned the SSIS packages from the legacy DTS packages.● Execute SSIS package include a master package which include number of child packages.● Supporting ETL (Extract Transform and Load) for fetching data from multiple systems to single Data Warehouse.● Created complex Ad-Hoc reports, Sub reports, linked reports related to State compliance reporting. Used custom code in SSRS for row color, visibility, and masking.● Developed Full Analysis Cycle Project and created packages for extracting data from OLTP to OLAP. Created Multi-Dimensional Expression (MDX) scripts for OLAP data cubes.