• Enhanced performance of BigQuery and Tableau reporting solutions by implementing optimization techniques like partitioning key columns and conducting thorough testing across diverse scenarios.• Successfully transitioned Oracle SQL ETL processes to operate on GCP using Cloud Dataproc, BigQuery, and Cloud Pub/Sub for orchestrating Airflow jobs.• Deployed Apache Airflow within GCP Composer environment to build data pipelines, using various Airflow operators such as Bash Operator, Hadoop… Show more • Enhanced performance of BigQuery and Tableau reporting solutions by implementing optimization techniques like partitioning key columns and conducting thorough testing across diverse scenarios.• Successfully transitioned Oracle SQL ETL processes to operate on GCP using Cloud Dataproc, BigQuery, and Cloud Pub/Sub for orchestrating Airflow jobs.• Deployed Apache Airflow within GCP Composer environment to build data pipelines, using various Airflow operators such as Bash Operator, Hadoop Operators, Python Callable, and Branching Operators.• Constructed data pipelines using Cloud Composer to orchestrate processes and migrated existing Cloud Dataprep jobs to BigQuery. Used Cloud Dataflow for building scalable machine learning algorithms.• Demonstrated proficiency in Hive SQL, Presto SQL, and Spark SQL for performing ETL tasks, selecting appropriate technologies based on job requirements.• Acted as an integrator, facilitating collaboration between data architects, data scientists, and other data consumers. Translated SAS code into Python and Spark-based jobs for execution in Cloud Dataproc and BigQuery.• Monitored data engines, defined data requirements, and accessed data from relational and non-relational databases including Cassandra and HDFS. Created complex SQL queries and established JDBC connectivity for presales and secondary sales estimations.• Authored sophisticated Hive SQL scripts for creating tables with high-performance attributes such as partitioning, clustering, and skewing. Show less

• Implemented AWS Databricks security features, such as encryption, network security, and compliance with industry standards.• Involved in designing optimizing Spark SQL queries, Data frames, import data from Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS/AWS S3.• Collaborated with the DevOps team to deploy NiFi pipeline on EC2 nodes integrated with Spark, Kafka, and Postgres using SSL handshakes in QA and production… Show more • Implemented AWS Databricks security features, such as encryption, network security, and compliance with industry standards.• Involved in designing optimizing Spark SQL queries, Data frames, import data from Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS/AWS S3.• Collaborated with the DevOps team to deploy NiFi pipeline on EC2 nodes integrated with Spark, Kafka, and Postgres using SSL handshakes in QA and production environments.• Developed data pipelines using AWS Glue and AWS Step Functions to orchestrate complex ETL workflows, enabling workflow scheduling, error handling, and data dependency management.• Optimized Spark applications for performance and scalability by tuning configurations, leveraging data partitioning, and applying caching techniques.• Utilized AWS CLI to automate backups of ephemeral data stores to S3 buckets.• Deployed AWS Lambda code from Amazon S3 buckets and configured an AWS Lambda Deployment function to receive events from S3 buckets.• Utilized Terraform to manage infrastructure, execute scripts, and create alarms and notifications for EC2 instances using CloudWatch.• Worked on Big Data clusters on the cloud to ingest and write data from and to Amazon S3 storage and Redshift, optimizing data storage costs. Show less

• Utilized Azure Data Factory and Data Catalog to ingest and maintain data sources, and enabled security on HDInsight using Azure Active Directory.• Created Spark jobs to process the source files and performed various transformations using Spark DataFrame/Dataset and Spark SQL API.• Developed ETL solutions using Spark SQL in Azure Databricks for data extraction, transformation, and aggregation from multiple file formats and data sources to uncover insights into customer usage… Show more • Utilized Azure Data Factory and Data Catalog to ingest and maintain data sources, and enabled security on HDInsight using Azure Active Directory.• Created Spark jobs to process the source files and performed various transformations using Spark DataFrame/Dataset and Spark SQL API.• Developed ETL solutions using Spark SQL in Azure Databricks for data extraction, transformation, and aggregation from multiple file formats and data sources to uncover insights into customer usage patterns.• Collaborated with DevOps and development teams to integrate Terraform into CI/CD pipelines, enabling continuous deployment and reducing the time-to-market for new features.• Implemented data governance and security practices within Azure Data Factory, including data encryption, access controls, and data masking techniques, to ensure compliance with regulatory requirements and protect sensitive data.• Wrote shell scripts to get data from MySQL/EDW server to HDFS using Sqoop functionality.• Worked on Netezza Procedures to load data into the final table.• Implemented data governance and security measures in Snowflake, such as role-based access control, data masking, and encryption. Show less

• Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.• Improved performance by optimizing computing time to process the streaming data by optimizing the cluster run time.• Executed Extract, Transform, and Load (ETL) processes from Kafka and various source systems to Azure Data Storage services via a blend of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure… Show more • Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.• Improved performance by optimizing computing time to process the streaming data by optimizing the cluster run time.• Executed Extract, Transform, and Load (ETL) processes from Kafka and various source systems to Azure Data Storage services via a blend of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.• Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats, aimed at analyzing and transforming data to uncover insights into customer usage patterns.• Utilized SQL Server Import and Export Data tool extensively, working with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.• Executed Extract, Transform, and Load (ETL) processes from Kafka and various source systems to Azure Data Storage services via a blend of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.• Created builds and release pipelines in VSTS and deployed solutions using SPN (secure endpoint connection) for implementing Continuous Integration/Continuous Deployment (CI/CD).• Set up Azure infrastructure, including storage accounts, integration runtime, service principal ID, and app registrations to enable scalable and optimized utilization of business user analytical requirements in Azure. Show less

• Acquired practical knowledge of Amazon Web Services' (AWS) infrastructure, including S3 and EC2.• Managed and monitored the Hadoop cluster using Cloudera Manager.• Handled massive data sets in parallel across the Hadoop cluster, creating MapReduce applications in Java using the MapReduce programming framework.• Involved in defining Directed Acyclic Graph (DAG) of operations and control flows using Oozie for scheduling tasks to manage Apache Hadoop workloads.• Managed and… Show more • Acquired practical knowledge of Amazon Web Services' (AWS) infrastructure, including S3 and EC2.• Managed and monitored the Hadoop cluster using Cloudera Manager.• Handled massive data sets in parallel across the Hadoop cluster, creating MapReduce applications in Java using the MapReduce programming framework.• Involved in defining Directed Acyclic Graph (DAG) of operations and control flows using Oozie for scheduling tasks to manage Apache Hadoop workloads.• Managed and reviewed Hadoop Log Files, deploying and maintaining the Hadoop Cluster.• Used Tableau to generate reports, graphs, and charts that offer an overview of the presented data.• Developed a data ingestion pipeline from HDFS into AWS S3 buckets using Nifi.• Performs complex pattern recognition of financial time series data and forecast of returns through the ARMA and ARIMA models and exponential smoothening for multivariate time series data.• Wrote Hive queries for data analysis to meet the business requirements.• Expertise in Business Intelligence and data visualization using Tableau.• Expert in Agile and Scrum Process. Show less