Surendra M Email & Phone Number

Southfield, Michigan, Us

• Worked with data transfer from on-premises SQL servers to cloud databases (Azure Synapse Analytics (DW) and Azure SQL DB). • Created Pipelines that were built in Azure Data Factory using Linked Services/Datasets/Pipeline/ to extract, transform, and load data from a variety of sources including Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and reverse. • Created CI-CD Pipelines using Azure devOps.• Used a blend of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics, gather, convert, and load the data from source systems to Azure Data Storage services. • Processed structured and semi-structured data into Spark Clusters using Spark SQL and Data Frames API. • Created Spark apps with Azure Data Factory and Spark-SQL for data extraction, transformation, and aggregation from various file formats in order to analyze and transform the data in order to reveal insights into consumer usage patterns. • Ingestion of data into one or more Azure Services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing of data in Azure Databricks. • Monitored the SQL scripts and modified them for improved performance using PySpark SQL. • Managed relational database service in which Azure SQL manages scalability, stability, and maintenance. • Integrated data storage options with Spark, notably with Azure Data Lake Storage and Blob storage. • Experience in ETL jobs and developing and managing data pipelines. • Experience in tuning hive jobs by performing partitioning, bucketing, and optimized joins on hive tables. • Performed data transformations and analytics on large dataset using Spark. • Created highly interactive Data Visualization Reports and Dashboards using feature such as Data Blending, Calculations, Filters, Actions, Parameters, Maps, Extracts, Context Filters, Sets and Aggregate measures using Tableau.• Combined views and reports into interactive dashboards in Tableau Desktop.

Chicago, Il, Us

• Worked on building centralized Data Lake on AWS Cloud utilizing primary services like S3, EMR, Redshift and Athena. • Worked on migrating datasets and ETL workloads from On-prem to AWS Cloud services. • Built series of Spark Applications and Hive scripts to produce various analytical datasets needed for digital marketing teams. • Created a Data Pipeline utilizing Processor Groups and numerous processors in Apache Nifi for Flat File, RDBMS as part of a Proof of Concept (POC) on Amazon EC2. • Created Apache presto and Apache drill configurations on an AWS EMR (Elastic Map Reduce) cluster to integrate different databases such as MySQL and Hive. This allows for the comparison of outcomes such as joins and inserts on many data sources controlled by a single platform. • Terraform scripts to automate deployment and configuration of various data lake components on AWS such as Cassandra clusters, Couchbase on EKS, Airflow, Glue, AWS RDS, Redshift, EMR, Tableau and SAS on RHEL with stripped volume at RHEL• Developed data ingestion modules (both real time and batch data load) to data into various layers in S3, Redshift and Snowflake using AWS Kinesis, AWS Glue, AWS Lambda and AWS Step Functions• Worked on automating the infrastructure setup, launching and termination EMR clusters etc,.• Created Hive external tables on top of datasets loaded in S3 buckets and created various hive scripts to produce series of aggregated datasets for downstream analysis. • Involved in the development of the new AWS Fargate API, which is comparable to the ECS run task API. • Worked on the code transfer of a quality monitoring application from AWS EC2 to AWS Lambda, as well as the construction of logical datasets to administer quality monitoring on snowflake warehouses.

Oxford, Mississippi, Us

• Involved in analysis, specification, design, and implementation and testing phases of Software Development Life Cycle (SDLC) and used agile methodology for developing the application • Upgraded existing UI with HTML, CSS, jQuery, and Bootstrap.• Designed and developed, Use-Case Class and Object Diagrams using UML Rational Rose for Object-Oriented Analysis/Object Oriented Design techniques. • Designed and developed data management system using MySQL. Built application logic using Python 2.7.• Used Django APIs for database access. • Developed RESTful services using Django.• Designed and developed communication between client and server using Secured Web services. • Developed and tested many features for a dashboard using Python, Java, Bootstrap, CSS, JavaScript, and jQuery.• Designed and developed data management system using MySQL. Involved in Agile Methodologies and SCRUM Process.• Participated in the complete SDLC process.• Create and enhanced existing automated solutions. • Parse JSON and XML data using Python3.X. • Responsible for debugging and troubleshooting the web application.• Experienced in Writing Indexes, Views, Constraints, Stored Procedures, Triggers, Cursors and user-defined functions or subroutines in MySQL. • Utilized Subversion control tool to coordinate teamwork.• Used Selenium Libraries to write fully functioning test automation process.