Showcased expertise in Data Lakes and big data ecosystems, including Hadoop, Spark, and AWS services such as Amazon Redshift and AWS Lake Formation. Employed Amazon Kinesis and AWS Glue Streaming for real-time data processing, enhancing data flow and deriving actionable insights for diverse applications.Developed Scala pipelines with Spark SQL for data extraction, transformation, and loading into AWS data stores, facilitating interactive SQL-based analysis.Streamlined data import processes from MySQL to Amazon S3 using AWS Glue and custom ETL scripts, improving data management efficiency and accessibility.Developed data ingestion pipelines leveraging AWS services like Amazon Kinesis, AWS Glue, and Amazon Redshift Spectrum, with Sqoop used for transferring customer behavioral data into Amazon S3 for further analysis.Optimized and analyzed AWS EMR clusters, leveraging Spark and Hive for efficient data processing and analytics.Designed and implemented comprehensive data pipelines using Amazon Kinesis, AWS Glue, and Amazon Redshift, focusing on data ingestion, transformation, and large dataset analysis.Leveraged AWS services such as S3, EC2, and IAM to architect new solutions and create custom ETL processes for seamless data migration from on-premises systems to the cloud.Utilized Jupyter notebooks and AWS Glue Studio for interactive data exploration, transformation, and analysis, enabling scalable data processing.Created and managed data visualizations with Python and Amazon QuickSight, performing data extraction and analysis from AWS services using SQL queries.Implemented data transformation pipelines with Spark with Scala and Spark SQL on AWS EMR, enabling efficient querying and analysis via DataFrames.Developed Spark Streaming pipelines on AWS EMR for real-time data ingestion and processing, utilizing micro-batching to efficiently handle data streams.

Responsible for creating the GCP buckets, Datasets, Big Query tables in different layers of BQ projects. Developed SQL scripts to Load data into Big Query.Spearheaded the design and implementation of a robust batch data processing pipeline on Google Cloud Platform (GCP), leveraging services such as Cloud Storage, BigQuery, and Dataflow, to handle massive data volumes efficiently. Developed and optimized data ingestion processes, extracting data from diverse sources including relational databases, APIs, and flat files, ensuring data completeness, accuracy, and adherence to defined data standards. Architected and implemented complex data transformations and enrichment logic using Apache Beam and Dataflow, enabling seamless integration of disparate data sources and facilitating advanced analytics and reporting capabilities. Implemented partitioning and clustering strategies to optimize data querying performance in BigQuery, resulting in a 40% reduction in query execution time and significantly enhancing overall data processing efficiency.

Designed, developed Datastage type1 load jobs to load Warehouse and Mart tables.● Developed complicated jobs using various Datastage Stages such as Join, Lookup, Change Capture, Peek, Column generator, Transformer, Sort, Remove duplicates.● Developed and optimized SQL queries for data extraction, transformation, and reporting.● Convert SAS code to SQL to load data warehouses using ETL jobs.● Developed the Batch cycles for scheduling the jobs by using Control-M.

Responsibilities:● Assist with prototyping, schematic capture, debugging, hardware bug tracking, functional verification and signal characterization.● Efficiently develop complex software in the LabVIEW programming environment.● Accurately estimate the timelines required to develop LabVIEW programs.● Practice effective revision control of programs that are developed.● Develop a server environment where users are placed in a queue and only one user is allowed access to the hardware.● Modify the LabView programs suitable to the server program.● Wrote and executed various MYSQL database queries from python using Python-MySQL connector and MySQL dB package