Monika K Email and Phone Number

● Developed Spark Applications to implement various data cleansing/validation and processing activity of large scale datasets ingested from traditional data warehouse systems.● Worked both with batch and real time streaming data sources.● Developed custom Kafka producers to write the streaming messages from external Rest applications to Kafka topics.● Developed spark streaming applications to consume the streaming json messages from Kafka topics.● Developed data transformations job using Spark Data frames to flatten JSON documents to csv.● Worked with the Spark for improving performance and optimization of the existing transformations.● Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data -from Kafka in Near real time and persist it to HBase.● Worked and learned a great deal from AWS Cloud services like EMR, S3, RDS, Redshift, Athena, Glue.● Migrated an existing on-premises data pipelines to AWS.● Worked on automating provisioning of AWS EMR clusters.● Used Hive QL to analyze the partitioned and bucketed data, executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business specification logic.● Experience in using Avro, Parquet, ORC file and JSON file formats, developed UDFs in Hive.● Worked with Log4j framework for logging debug, info & error data.● Used Jenkins for Continuous integration.● Generated various kinds of reports using Tableau based on client specification.● Used Jira for bug tracking and Git to check-in and checkout code changes.● Responsible for generating actionable insights from complex data to drive real business results for various application teams and worked in Agile Methodology projects extensively.● Worked with Scrum team in delivering agreed user stories on time for every Sprint.

• Involved in writing Spark applications using Scala to perform various data cleansing, validation, transformation, and summarization activities according to the requirement.• Involved in creating data lake in Google Cloud Platform (GCP) for allowing business teams to perform data analysis in BigQuery.• Responsible for developing data pipelines involving ingesting raw json files, transactional and user profile information from on prem data warehouses and processing them using spark and finally loading the processed data to BigQuery.• Automated launch of Dataproc clusters and autoscaling the clusters and submitted spark jobs to dataproc clusters.• Utilized cloud sql as external hive metastore for dataproc clusters so that metadata is persisted across multiple dataproc clusters.• Utilized Spark-Bigquery Connector for writing the processed data from spark to Bigquerydirectly.• Utilized Google Cloud Storage as data lake and ensured all the processed data is written to GCS directly from spark and hive jobs.• Written kafka producers for streaming real time json messages to kafka topics and processed them using spark streaming and performed streaming inserts to Bigquery.• Worked extensively on performance tuning of Spark application to improve job execution times and troubleshooting failures.• Worked on different file formats like Text, Avro, Parquet, JSON and Flat files using Spark.• Developed daily process to do incremental import of data from Teradata into Hive tables using Sqoop.• Work with cross functional consulting teams within the data science and analytics team to design, develop and execute solutions to derive business insights and solve client’s operational and strategic problems.• Extensively worked with Partitions, Dynamic Partitioning, bucketing tables in Hive, designed both Managed and External tables, also worked on optimization of Hive queries.

• Involved in creating data ingestion pipelines for collecting health care and providers data from various external sources like FTP Servers and S3 buckets.• Involved in migrating existing Teradata Datawarehouse to AWS S3 based data lakes.• Involved in migrating existing traditional ETL jobs to Spark and Hive Jobs on new cloud data lake.• Wrote complex spark applications for performing various de-normalization of the datasets and creating a unified data analytics layer for downstream teams.• Primarily responsible for fine-tuning long running spark applications, writing custom spark udfs, troubleshooting failures etc.,• Involved in building a real time pipeline using Kafka and Spark streaming for delivering event messages to downstream application team from an external rest-based application.• Involved in creating Hive scripts for performing adhoc data analysis required by the business teams. • Worked extensively on migrating on prem workloads to AWS Cloud.• Worked on utilizing AWS cloud services like S3, EMR, Redshift, Athena and Glue Metastore.• Used broadcast variables in spark, effective & efficient Joins, caching and other capabilities for data processing.• Involved in continuous Integration of application using Jenkins.

● Involved in writing Spark applications using Scala to perform various data cleansing, validation, transformation, and summarization activities according to the requirement.● Load the data into Spark RDD and perform in-memory data computation to generate the output as per the requirements.● Developed data pipelines using Spark, Hive and Sqoop to ingest, transform and analyze operational data.● Developed Spark jobs, Hive jobs to summarize and transform data.● Worked on performance tuning of Spark application to improve performance.● Performance tuning the Spark jobs by changing the configuration properties and using broadcast variables.● Real time streaming the data using Spark with Kafka. Responsible for handling Streaming data from web server console logs.● Worked on different file formats like Text, Sequence files, Avro, Parquet, JSON, XML files and Flat files using Map Reduce Programs.● Developed daily process to do incremental import of data from DB2 and Teradata into Hive tables using Sqoop.● Analyzed the SQL scripts and designed the solution to implement using Scala.● Solved performance issues in Hive and Pig scripts with understanding of Joins, Group and Aggregation and how does it translate to MR jobs.● Work with cross functional consulting teams within the data science and analytics team to design, develop and execute solutions to derive business insights and solve clients operational and strategic problems.● Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.● Extensively used Hive/HQL or Hive queries to query data in Hive Tables and loaded data into HBase tables.● Extensively worked with Partitions, Dynamic Partitioning, bucketing tables in Hive, designed both Managed and External tables, also worked on optimization of Hive queries.● Involved in collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.

● Actively participated in requirements gathering, analysis, design, and testing phases.● Designed use case diagrams, class diagrams, and sequence diagrams as a part of Design Phase.● Developed the entire application implementing MVC Architecture integrating JSF with Hibernate and Spring frameworks.● Created and implemented stored procedures, functions, triggers, using SQL.● Setting up client-side validations using JavaScript.● Developed the Enterprise Java Beans (Stateless Session beans) to handle different transactions such as online funds transfer, bill payments to the service providers.● Developed XML documents and generated XSL files for Payment Transaction and Reserve Transaction systems.● Developed Web Services for data transfer from client to server and vice versa using Apache Axis and SOAP.● Implemented various J2EE Design patterns like Singleton, Service Locator, DAO, and SOA.● Worked on AJAX to develop an interactive Web Application and JavaScript for Data Validations.