• Involved in requirements gathering, analysis, design, development, change management, deployment.• Involved in the development of real time streaming applications using PySpark, Apache Flink, Kafka, Hive on distributed Hadoop Cluster.• Utilizing Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine learning use cases under Spark ML and Mllib.• Extracting data from heterogeneous sources and performed complex business logic on network data to normalize raw data which can be utilized by BI teams to detect anomalies.• Designing and developing Flink pipelines to consume streaming data from Kafka and applied business logic to massage and transform and serialize raw data.• Developing common Flink module for serializing and deserializing AVRO data by applying schema.• Developing Spark streaming pipeline to batch real time data, detect anomalies by applying business logic and write the anomalies to HBase table.• Implementing layered architecture for Hadoop to modularize design. Developed framework scripts to enable quick development. Designed reusable shell scripts for Hive, Sqoop, Flink and PIG jobs. Standardize error handling, logging and metadata management processes.• Indexed processed data and created dashboards and alerts in splunk to be utilized/ action by support teams.• Responsible for operations and support of big data Analytis platform, Splunk and Tableau visualization.• Managed, developed, and designed a dashboard control panel for customers and Administrators using Tableau, PostgreSQL and RESTAPI calls. Skills: Apache Spark · PySpark · Scala · Extract, Transform, Load (ETL) · hdfs · Hadoop · Continuous Integration and Continuous Delivery (CI/CD) · Google BigQuery · Microsoft Azure ·

• Designed Batch Audit Process in batch\shell script to monitor each ETL job along with reporting status which includes table name, start and finish time, number of rows loaded, status, etc.• Developed python scripts to automate data ingestion pipeline for multiple data sources and deployed Apache Nifi in AWS.• Design and develop Tableau visualizations which include preparing Dashboards using calculations, parameters, calculated fields, groups, sets and hierarchies.• Involved in implementation of new statistical algorithms and operators on Hadoop and SQL platforms and utilized optimizations techniques, linear regressions, K-means clustering, Native Bayes and other approaches. • Developed Spark batch job to automate creation/metadata update of external Hive table created on top of datasets residing in HDFS. • Developed Data Serialization spark common module for converting Complex objects into sequence bits by using AVRO, PARQUET, JSON, CSV formats. • Worked on ER Modeling, Dimensional Modeling (StarSchema, Snowflake Schema), Data warehousing and OLAP tools. • Populated HDFS and PostgreSQL with huge amounts of data using Apache Kafka. • Design and develop Rest API (Commerce API) which provides functionality to connect to the PostgreSQL through Java services. • Designed Batch Audit Process in batch\shell script to monitor each ETL job along with reporting status which includes table name, start and finish time, number of rows loaded, status, etc. • Developed Spark jobs in PySpark to perform ETL from SQL Server to Hadoop. • Responsible for continuous monitoring and managing Elastic MapReduce (EMR) cluster through AWS console.Skills: PySpark · Scala · Azure Databricks · Bash · SQL · Hive · Amazon Web Services (AWS) · Python (Programming Language) · JSON · MapReduceSkills: PySpark · Scala · Azure Databricks · Bash · SQL · Hive · Amazon Web Services (AWS) · Python (Programming Language) · JSON · MapReduce

 • Responsible for analysis of requirements and designing generic and standard ETL process to load data from different source systems• The developed objects are tested for unit/component testing and prepared test cases document for mappings/sessions/workflows.• Involved in the daily status meeting and interacting with onshore team through mails/calls to follow up the module and to resolve the data/code issues.• Handled Classification System part of the project which involved loading of the data based on some pre conditions• Understand the existing business model and customer requirements.• Involved in developing and documenting the ETL (Extract, Transformation and Load) strategy to populate the Data Warehouse from various source systems• Involved in Data Extraction, Staging, Targeting Transformation and Loading.• Involved in testing at the data base end and reviewing the Informatica Mappings as per the business logic• Listed out the issues that was not according to business requirement, developed some maps and changes for other maps.• Wrote several test cases, identifying the issues that can occur, understanding the date merge, match process.