Implemented multiple streaming pipelines to load data from Kafka and AWS kinesis to AWS S3 using Apache Spark Structured Streaming Developed batch jobs to transform raw data into meaningful insights per business reporting requirements using Apache Spark, and eventually, the insights are visualized using Looker. Enriched datasets by joining with third-party datasets to derive business outcomes for the newly released products at Comcast.Configured AWS Glue crawlers to update metastore for newly added partitions in S3. Did ad-hoc analysis and debugging of batch insights using AWS Athena.Worked with BI teams in analyzing the data and designing ETL workflows.Developed unit tests using Spark testing base library for batch jobs to improve code quality. Built internal tools based on the Databricks platform to automate the orchestration of workflows from Git Hub using Jenkins. Did performance testing for Spark batch and streaming jobs by reviewing the Spark planning and identifying bottlenecks. Leveraged AWS Glue for metadata management in data catalog, architected Amazon Athena to execute historical ad hoc queries, utilizing AWS Glue metadata and transferring the result to Amazon S3.Created a de-normalized data model in AWS S3 using Spark and Athena for reporting stakeholders. Expertise in developing spark jobs using window functions, Dataset APIs, accumulators, and custom UDFs.Developed an internal tool to parse spark job logs from the Databricks platform to analyze the input records, and optimize spark resources for all the jobs running on the platform.Developed a framework to drop the partitions for each dataset per the retention policy to adhere to Comcast Network Industry Standards.Expertise in performing initial and incremental data load from Oracle to AWS S3. Collaborated with cross-functional teams to design and implement multiple data pipelines using Spark on the Databricks platform.

Led the design and implementation of a robust data pipeline to extract healthcare data from an on-prem SQL database and Oracle databases to HDFS using Sqoop, utilizing Kafka for real-time data streaming to Hadoop HDFS.Created de-normalized data model in HDFS to achieve better HIVE performance.Created managed and external tables in Hive and implemented partitioning and bucketing techniques for space and performance efficiency.Developed Java-based custom Hive UDFs to safeguard sensitive data during processing.Orchestrated the integration of the data pipeline with Azure Data Factory to perform Extract, Transform, Load (ETL) operations, ensuring efficient data ingestion into Azure Data Lake Gen2.Developed a layer-based architecture for data transformation to maintain original data copies, perform fundamental transformations, and refine data quality for advanced analytics using pyspark.Utilized Azure Databricks to execute complex data transformations, including data cleansing, normalization, and enrichment, resulting in high-quality data ready for analysis.Collaborated with data analysts and domain experts to understand healthcare data insurances, enabling accurate translation of raw data into meaningful insights.Designed and established tables and columns within Azure Synapse Analytics, facilitating optimized storage and retrieval of healthcare data.Ensured data security and access control by integrating Azure Active Directory for identity and access management and Azure Key Vault for safeguarding sensitive information.Empowered healthcare professionals with data-driven decision-making capabilities by creating interactive and insightful visualizations using Power BI.Orchestrated seamless data migration from Azure Synapse Analytics with Azure Blob Storage to Snowflake through Azure Data Factory, enabling efficient data transfer while preserving data integrity.

Demonstrated expertise in data handling and migration, successfully transferring data from diverse sources including transaction systems, payment gateways, Financial Data, customer databases, mainframe, and MySql, into the Hadoop environment using Sqoop. Leveraged Flume for real-time data streaming, enabling seamless migration of data into the Hadoop ecosystem.Gather clickstream data by sending events to Kafka topics. Use producers to publish events and consumers to subscribe and process the data streams.Collect service logs by configuring log streams in CloudWatch Logs. Monitor and analyze the logs using CloudWatch metrics and alarms.Retrieve social media data by interacting with Python libraries like tweepy for Twitter or requests for Facebook APIs.Implemented Hive's advanced features like Partitioning, Dynamic Partitions, and Buckets to enhance query performance and efficiently manage large-scale data.Executed rigorous data validation queries in Hive to ensure accuracy of migrated data.Developed custom UDF’s to handle exponential values and to convert different date formats to desired format.Configured and used HCatalog to access the table data maintained in the Hive megastore and use the same table information for processing in Pig.Utilized PySpark for Extract, Transform, Load (ETL) transformations, meeting the business requirements for generating reports on customer churn and credit card offered to customer report generation.Conducted data profiling and data cleansing activities using Apache spark to identify and rectify anomalies, inconsistencies, and missing data within card-related datasets.Ensured transformed and cleaned data was stored efficiently in Hive and AWS S3, facilitating downstream teams' access for report generation and analysis.Implemented automated workflows using Apache Airflow to execute key business processes on a weekly basis, aligning with data generation cycles and enhancing operational efficiency.

Involved in analysis and design phases of the Software Development Life Cycle (SDLC).Implemented design patterns and OO design concepts to build the code.Participated in planning and developing UML diagrams like Use Case Diagrams, Object Diagrams, Class Diagrams, and Sequence Diagrams to represent the detailed design phase. Implemented various J2EE Design patterns like Singleton, Service Locator, DAO, and SOA.Configured Maven dependencies for application-building processes that created Pom.xml files. Developed Utility Classes in Java for accessing databases using Stored Procedures.Developed Exception handling framework and used log4J for logging and JUnit for unit testing.Writing Regular expressions for verification of the data.Created a database schema for using SQL and also implemented the same through JDBC.Involved in writing complex SQL queries using JDBC and stored procedures for the application in Oracle 10g.Used Web services - WSDL and SOAP using Apache-AXIS to communicate between the systems. Performed unit testing, system testing, and user acceptance test.Used SVN as a version control system for giving version labels and checking the changes