Joshi G Email & Phone Number

 Designed and developed ETL (Extract, Transform, Load) pipelines usingAWS services such as Glue, Lambda, and S3 to extract data from varioussources, transform it and load it into a data warehouse. Build and maintain scalable and highly available data solutions using AWSservices such as Redshift, RDS, and DynamoDB. Extensively worked with the application data present in AWS S3 storage forperforming required transformations. Experience wif big data processing tools like CouchDB that allows running asingle logical database server on any number of servers. Used DataStax Cassandra along wif Pentaho for reporting. Implemented various Data Modeling techniques for Cassandra Improved teh query performance by transitioning log storage fromCassandra to Azure SQL Datawarehouse. Integrated on-premises data(MySQL, Cassandra) Configured and deployed production-ready data pipelines using AWS Glue,S3 and Lambda to maximize data extraction, transformation, and loading. Designed and implemented data ingestion pipelines using Apache Airflow toefficiently transfer and process data from various sources. Created efficient DAGs to represent data workflows, defining taskdependencies and execution order. Integrated Apache Airflow with Amazon S3 to store and retrieve data,ensuring seamless data flow between different stages of the pipeline. Optimized DAGs for parallel execution, improving data processingperformance and reducing processing times. Performed transformations on the data using Glue jobs with the pysparkscript and stored transformed data to S3. Integrated Apache Airflow with AWS Glue Data Catalog for centralizedmetadata management and cataloging of data assets within the data pipeline. Set up CI/CD pipelines to automate testing and deployment of Airflow DAGs,enhancing the development and release process. Utilized AWS Athena interactive querying service for running Ad-hoc queriesto perform data analysis on transformed data

Worked on Pandas DataFrame operations to clean and preprocess raw data,handling missingvalues and outliers.Developed an ETL process to transform raw data into a structured formatusing tools such asCloud Dataprep and Cloud Functions.Utilized Apache Beam to construct complex data transformation, ensuring dataquality andconsistency. workflowsDeveloped and deployed Python-based data pipelines on Google CloudDataflow for highvolume data processing.Designed and implemented a data pipeline to process and store large volumesof data using GCPtechnologies such as Apache Beam and Dataflow.Utilized Pandas for efficient reading and loading of data from various sources,including CSVand databases, into Google Cloud Storage, BigQuery.Built a data warehouse on GCP using BigQuery and optimized it forperformance and cost.Integrated new third-party data sources into the data pipeline using APIs andcloud storage services.Developed and scheduled data processing jobs using Dataproc and ApacheSpark.Implemented data validation and cleaning procedures to ensure integrity andimproved dataquality by 20%.Designed CI/CD pipelines to scale efficiently to accommodate growing dataworkloads.Developed automated workflows using Cloud Composer to reduce manualintervention andimprove efficiency by 50%.Utilized Dataproc to process and analyze large-scale datasets.Utilized Python along with Google Cloud Storage (GCS) to store and retrievedata.Utilized Dataproc autoscaling to dynamically adjust cluster size based onworkload demands.Developed data profiling and cleansing tools using GCP services such asCloud Dataprep andData Loss Prevention (DLP) to ensure data accuracy and compliance.Created and managed a Data Catalog using Google Cloud Data Catalog.Worked with PowerShell and Linux scripts for file transfer, emailing and otherfile related tasks.

• Migrated the Snowflake database to Windows Azure and updated theConnection Strings based on the requirement.• Developed Talend MDM jobs to populate the claims data to data warehouse -star schema, snowflake schema, Hybrid Schema.•Developing Spark applications using Spark-SQL in Databricks for dataextraction, transformation, and aggregation from multiple file formats forAnalyzing& transforming the data to uncover insights into the customer usagepatterns.• Creating pipeline jobs, scheduling triggers, Mapping data flows using AzureData Factory(V2), and using Key Vaults to store credentials.•Creating Elastic pool databases and scheduling Elastic jobs for executingTSQL procedures.•Extract Transform and Load data from sources Systems to Azure DataStorage services using a combination of Azure Data factory, T-SQL, SparkSQL, and U-SQL Azure Data Lake Analytics. Data ingestion to one or moreAzure services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) andprocessing the data in Azure Databricks• Worked on creating correlated and non-correlated sub-queries to resolvecomplex business queries involving multiple tables from different databases.• Implemented Disaster Recovery and Failover servers in Cloud by replicatingdata across regions

• Hands on experience in Spark and Spark Streaming creating RDD & applyingoperations, transformations, and Actions.• Used Python with Apache Spark for distributed data processing on Hadoopclusters.• Used Spark and Spark-SQL to read the parquet data and create the tables inhive using the ScalaAPI.• Developed Spark code using Scala and Spark-SQL for faster processing andtesting.• Designed and implemented complex data processing pipelines using Scalaand Apache Spark.• Experienced Good understanding of NoSQL databases and hands on workexperience in writing applications No SQL Databases HBase, Cassandra andMongoDB.• Utilized Control-M to schedule and automate data pipelines, ensuring timelyand reliable execution of data processing and ETL jobs.• Integrated version control systems Git into CI/CD pipelines for automatedtesting and deployment.• Worked on Hadoop cluster and data querying tools Hive to store and retrievedata.• While developing applications involved in complete Software DevelopmentLife Cycle (SDLC).• Involved in the design, development and testing phases of application usingAGILE methodology.

Master'S Degree, Computer And Information Sciences

Bachelor'S Degree, Electronics And Communications Engineering