Sunil S Email and Phone Number

 Evaluated business requirements and prepared detailed specifications adhering to project guidelines, including Python-based ETL processes. Responsible for Big data initiatives, including analysis, POC, and architecture, leveraging Python for data processing. Loaded and transformed large datasets using Azure HDInsight, incorporating Python scripts for ETL. Led the modern Data Architecture practice, delivering projects in Azure Cloud Technologies, with a focus on Python-based ETL solutions. Installed and Configured Azure HDInsight clusters, incorporating Python libraries for ETL tasks. Installed and configured Azure Data Lake Analytics and written U-SQL queries, integrating Python for ETL processes. Developed data pipeline using Azure Data Factory to ingest cargo data and customer histories, integrating Python scripts for ETL. Migrated existing on-premises code to Azure HDInsight cluster, ensuring compatibility with Python-based ETL workflows. Installed and configured Hadoop Ecosystem components and Azure HDInsight using Azure Databricks, with Python for ETL. Created automated pipelines in Azure DevOps to deploy Docker containers in Azure Kubernetes Service using Azure Blob Storage, incorporating Python for automation. Used Azure Cosmos DB for real-time access to data, integrating Python for ETL tasks. Extracted Real-time feed using Azure Stream Analytics and processed data into Data Frames and loaded data into Azure Cosmos DB, leveraging Python for data processing. Leveraged Azure Functions for event-driven data processing, including ETL tasks. Integrated Azure Functions with Azure Data Factory to orchestrate and automate ETL workflows, utilizing Python scripts. Managed data ingestion from various sources into Azure Event Hubs, incorporating Python for data transformation.

• Engineered automated build/deployment processes, improving user experience and implementing a continuous integration system.• Utilized AWS services (EC2, S3, DynamoDB, Lambda, RDS, etc.) for high availability, fault tolerance, and auto-scaling of multi-tier applications.• Deployed applications on AWS Cloud Formation, ensuring continuous storage using Elastic Block Storage, S3, and Glacier.• Migrated Data Pipeline from Cloudera Hadoop to AWS EMR clusters, implementing MapReduce programs for unstructured data.• Designed Big Data analytics platform using Hadoop, Hive, Pig, and Cloudera for processing customer interface preferences.• Managed ETL processes, transforming data from SQL Server, MySQL, PostgreSQL, and CSV into data frames using PySpark.• Developed AWS Lambda functions for serverless data pipelines integrated with Glue Catalog and Athena.• Configured S3 bucket rules, used Glacier and S3 for backup/storage, imported/exported data with Sqoop from Oracle to HDFS/Hive.• Read various data formats on HDFS using Python, converted Hive/SQL queries to Spark transformations with Spark RDDs.• Installed/configured Pig, developed Pig Latin scripts, and executed POCs comparing Spark, Hive, and SQL performance.• Worked on Spark SQL, migrated iterative MapReduce programs to Spark transformations using Python in AWS cloud environment.• Developed Spark jobs in Python for test environments, configured Spark streaming for real-time data from Kafka to HDFS.• Designed and implemented SOLR indexes for metadata, extensively used Pig for data cleansing, and developed a Kafka-Storm data pipeline.• Configured Kafka producers, created custom partitions, and implemented High-level consumers for data platform.• Created Hive tables, loaded data, and wrote Hive queries for internal MapReduce processing.• Maintained SQL-based analytics/reports using Power BI and Tableau, analyzed partitioned/bucketed data in Hive for reporting.

• Contributed to key data integration and advanced analytics solutions using Apache Hadoop for leading organizations.• Experienced in Agile methodologies, daily Scrum meetings, and Sprint planning.• Performed data transformations in HIVE, used partitions, and applied bucketing for performance improvements.• Developed business-specific Custom UDF's in Hive and implemented Spark structured streaming jobs on AWS EMR.• Loaded data into Amazon Redshift, monitored AWS RDS instances using AWS CloudWatch.• Worked as a Spark Expert and Performance Optimizer, handling Spark-SQL and Data Skewness.• Migrated an on-premises application to AWS and was part of Spark Center of Excellence at Cisco.• Implemented Spark using Python and Java, utilizing DataFrames and Spark SQL API for faster data processing.• Developed data pipeline using Kafka, HBase, Spark, and Hive for ingesting and analyzing customer behavioral data.• Executed a migration strategy to move Data Warehouse from Oracle to AWS Redshift.• Imported metadata into Hive, migrated tables and applications to work on Hive and Spark.• Implemented Sqooping from Oracle to Hadoop and enhanced traditional data warehouse based on STAR schema.• Used Spark for interactive queries, processing streaming data, and integration with NoSQL databases.• Imported data from various sources into HDFS using Sqoop, performed transformations using Hive, MapReduce, and loaded data into HDFS.• Collected and aggregated log data using Flume, designed and maintained test workflows for job management.• Collaborated with business stakeholders to architect, implement, and test Big Data analytical solutions.

• Designed and implemented SSIS jobs for data integration, focusing on Microsoft SQL Server.• Developed a strategy for applying predictive analytics to data warehouse projects based on business needs.• Implemented solutions for ingesting and processing Data-at-Rest using Hadoop, Map Reduce, HBase, and Hive.• Improved multi-node Hadoop Cluster performance and designed Big Data analytics platform.• Analyzed data sources from SQL Server and Oracle, leading reporting and analysis projects in Tableau Desktop.• Collaborated with teams to ensure product quality, emphasizing Development and Quality Assurance.• Migrated data warehouse from Azure Synapse to Big Query, optimizing schema and ETL processes.• Responsible for data extraction and ingestion into Hadoop Data Lake, creating ETL pipelines with Apache Hive.• Analyzed Hadoop cluster and Big Data tools, handling large volumes of XML and JSON data using Hadoop streaming.• Installed YARN Capacity Scheduler, fine-tuned settings for relevant workloads, and utilized Azure Data Factory, T-SQL, Spark SQL, and U-SQL for data analytics.• Created Oozie workflow engine for scheduling Hive and Pig operations and used SQL, PL SQL, and Spark SQL for performance optimization.• Developed SSIS packages for ETL transformation tasks, wrote T-SQL stored procedures, and managed incremental changes in source systems.• Created complex mappings involving Slowly Changing Dimensions and implemented Business Logic.• Developed SQL stored procedures for database updates and index creation in target tables.