Joshi G Email & Phone Number

• Designed and developed ETL (Extract, Transform, Load) pipelines using AWS services such as Glue, Lambda, and S3 to extract data from various sources, transform it and load it into a data warehouse.• Build and maintain scalable and highly available data solutions using AWS services such as Redshift, RDS, and DynamoDB.• Extensively worked with the application data present in AWS S3 storage for performing required transformations.• Configured and deployed production-ready data… Show more • Designed and developed ETL (Extract, Transform, Load) pipelines using AWS services such as Glue, Lambda, and S3 to extract data from various sources, transform it and load it into a data warehouse.• Build and maintain scalable and highly available data solutions using AWS services such as Redshift, RDS, and DynamoDB.• Extensively worked with the application data present in AWS S3 storage for performing required transformations.• Configured and deployed production-ready data pipelines using AWS Glue, S3 and Lambda to maximize data extraction, transformation, and loading.• Designed and implemented data ingestion pipelines using Apache Airflow to efficiently transfer and process data from various sources.• Created efficient DAGs to represent data workflows, defining task dependencies and execution order.• Integrated Apache Airflow with Amazon S3 to store and retrieve data, ensuring seamless data flow between different stages of the pipeline.• Optimized DAGs for parallel execution, improving data processing performance and reducing processing times.• Performed transformations on the data using Glue jobs with the pyspark script and stored transformed data to S3.• Integrated Apache Airflow with AWS Glue Data Catalog for centralized metadata management and cataloging of data assets within the data pipeline.• Optimized EMR usage and established scalable, fault-tolerant ETL pipelines to facilitate data extraction from multiple sources for storage in the S3 data lake, increasing output efficiency.• Set up CI/CD pipelines to automate testing and deployment of Airflow DAGs, enhancing the development and release process.• Implementing AWS Big Data services such as EMR, Kinesis, and Athena to process large volumes of data.• Utilized AWS Athena interactive querying service for running Ad-hoc queries to perform data analysis on transformed data Show less

Worked on Pandas DataFrame operations to clean and preprocess raw data, handling missingvalues and outliers.Developed an ETL process to transform raw data into a structured format using tools such asCloud Dataprep and Cloud Functions.Utilized Apache Beam to construct complex data transformation, ensuring data quality andconsistency. workflowsDeveloped and deployed Python-based data pipelines on Google Cloud Dataflow for high-volume data processing.Designed and… Show more Worked on Pandas DataFrame operations to clean and preprocess raw data, handling missingvalues and outliers.Developed an ETL process to transform raw data into a structured format using tools such asCloud Dataprep and Cloud Functions.Utilized Apache Beam to construct complex data transformation, ensuring data quality andconsistency. workflowsDeveloped and deployed Python-based data pipelines on Google Cloud Dataflow for high-volume data processing.Designed and implemented a data pipeline to process and store large volumes of data using GCPtechnologies such as Apache Beam and Dataflow.Utilized Pandas for efficient reading and loading of data from various sources, including CSVand databases, into Google Cloud Storage, BigQuery.Built a data warehouse on GCP using BigQuery and optimized it for performance and cost.Integrated new third-party data sources into the data pipeline using APIs and cloud storageservices.Developed and scheduled data processing jobs using Dataproc and Apache Spark.Implemented data validation and cleaning procedures to ensure integrity and improved dataquality by 20%.Designed CI/CD pipelines to scale efficiently to accommodate growing data workloads.Developed automated workflows using Cloud Composer to reduce manual intervention andimprove efficiency by 50%.Utilized Dataproc to process and analyze large-scale datasets.Utilized Python along with Google Cloud Storage (GCS) to store and retrieve data.Utilized Dataproc autoscaling to dynamically adjust cluster size based on workload demands.Developed data profiling and cleansing tools using GCP services such as Cloud Dataprep andData Loss Prevention (DLP) to ensure data accuracy and compliance.Created and managed a Data Catalog using Google Cloud Data Catalog.Worked with PowerShell and Linux scripts for file transfer, emailing and other file related tasks. Show less

• Migrated the Snowflake database to Windows Azure and updated the Connection Strings based on the requirement.• Developed Talend MDM jobs to populate the claims data to data warehouse - star schema, snowflake schema, Hybrid Schema.•Developing Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing& transforming the data to uncover insights into the customer usage patterns.• Creating pipeline… Show more • Migrated the Snowflake database to Windows Azure and updated the Connection Strings based on the requirement.• Developed Talend MDM jobs to populate the claims data to data warehouse - star schema, snowflake schema, Hybrid Schema.•Developing Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing& transforming the data to uncover insights into the customer usage patterns.• Creating pipeline jobs, scheduling triggers, Mapping data flows using Azure Data Factory(V2), and using Key Vaults to store credentials.•Creating Elastic pool databases and scheduling Elastic jobs for executing TSQL procedures.•Extract Transform and Load data from sources Systems to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data ingestion to one or more Azure services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks• Worked on creating correlated and non-correlated sub-queries to resolve complex business queries involving multiple tables from different databases.• Implemented Disaster Recovery and Failover servers in Cloud by replicating data across regions. Show less

• Hands on experience in Spark and Spark Streaming creating RDD & applying operations, transformations, and Actions.• Used Python with Apache Spark for distributed data processing on Hadoop clusters.• Used Spark and Spark-SQL to read the parquet data and create the tables in hive using the ScalaAPI.• Developed Spark code using Scala and Spark-SQL for faster processing and testing.• Designed and implemented complex data processing pipelines using Scala and Apache… Show more • Hands on experience in Spark and Spark Streaming creating RDD & applying operations, transformations, and Actions.• Used Python with Apache Spark for distributed data processing on Hadoop clusters.• Used Spark and Spark-SQL to read the parquet data and create the tables in hive using the ScalaAPI.• Developed Spark code using Scala and Spark-SQL for faster processing and testing.• Designed and implemented complex data processing pipelines using Scala and Apache Spark.• Experienced Good understanding of NoSQL databases and hands on work experience in writing applications No SQL Databases HBase, Cassandra and MongoDB.• Utilized Control-M to schedule and automate data pipelines, ensuring timely and reliable execution of data processing and ETL jobs.• Integrated version control systems Git into CI/CD pipelines for automated testing and deployment.• Worked on Hadoop cluster and data querying tools Hive to store and retrieve data.• While developing applications involved in complete Software Development Life Cycle (SDLC).• Involved in the design, development and testing phases of application using AGILE methodology. Show less

Master'S Degree, Computer And Information Sciences

Bachelor'S Degree, Electronics And Communications Engineering