Manisha M Email & Phone Number

California, United States

• Working in big data technologies like spark 2.3 & 3.0 Scala, Hive, Hadoop cluster (Cloudera platform).• Worked in AWS environment for development and deployment of Custom Hadoop Applications.• Installed Hadoop, Map Reduce, HDFS, and AWS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.• Design & implement Spark Sql tables, Hive scripts job with stone branch for scheduling and create workflow and task flow.• We generally used partitions and bucketing for data in hive to get query faster. This part of hive optimization• Write programs using Spark to move data from Storage input location to output location by running data loading, validation, and transformation to the data• Used Scala function, dictionary and data structure (array, list, map) for better code reusability• Based on Development, we need to do the Unit Testing.• Prepare the Technical Release Notes (TRN) for the application deployment into the• DEV/STAGE/PROD environment.• Developed report layouts for Suspicious Activity and Pattern analysis under AML regulations• Prepared and analyzed AS IS and TO BE in the existing architecture and performed Gap Analysis. Created workflow scenarios, designed new process flows and documented the Business Process and various Business Scenarios and activities of the Business from the conceptual to procedural level.• Analyzed business requirements and employed Unified Modeling Language (UML) to develop high-level and low-level Use Cases, Activity Diagrams, Sequence Diagrams, Class Diagrams, Data-flow Diagrams, Business

New York, United States

• Experience in Job management using Fair scheduler and Developed job processing scripts using Oozie workflow.• Used Spark, Hive for implementing the transformations need to join the daily ingested data to historic data.• Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time.• Developed Spark scripts by using Scala shell commands as per the requirement.• Used Spark API over EMR Cluster Hadoop YARN to perform analytics on data in Hive.• Developed Scala scripts, UDFs using both Data frames/SQL/Data sets and RDD in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.• Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.• Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark

Tampa, Florida, United States

• Created and executed Hadoop Ecosystem installation and document configuration scripts on Google Cloud Platform.• Transformed batch data from several tables containing tens of thousands of records from SQL Server, MySQL, PostgreSQL, and csv file datasets into data frames using PySpark.• Researched and downloaded jars for Spark-avro programming.• Developed a PySpark program that writes data frames to HDFS as avro files.• Utilized Spark's parallel processing capabilities to ingest data.• Created and executed HQL scripts that creates external tables in a raw layer database in Hive.• Developed a Script that copies avro formatted data from HDFS to External tables in raw layer.• Created PySpark code that uses Spark SQL to generate data frames from avro formatted raw layer and writes them to data service layer internal tables as orc format.• In charge of PySpark code, creating data frames from tables in data service layer and writing them to a Hive data warehouse.• Developed MapReduce/Spark Python modules for machine learning & predictive analytics in Hadoop on AWS.• Doing data synchronization between EC2 and S3, Hive stand-up, and AWS profiling.• Installed Airflow and created a database in PostgreSQL to store metadata from Airflow.• Configured documents which allow Airflow to communicate to its PostgreSQL database.• Developed Airflow DAGs in python by importing the Airflow libraries.

Noida, Uttar Pradesh, India

• Responsible for SDLC process in gathering requirements, system analysis, design, development, testing and deployment.• Designed the front-end applications, user interactive (UI) web pages using web technologies like HTML, XHTML, CSS, JavaScript and JQuery.• Work with team of developers on python applications for RISK management. • Design, develop, test, deploy and maintain the website. • Designed and developed data management system using MySQL. • Rewrite existing Python/Django modules to deliver certain format of data. • Developed entire frontend and backend modules using Python on Django Web Framework. • Responsible for debugging and troubleshooting the web application. • Using Subversion control tool to coordinate team-development. • Used Django Database API’s to access database objects. • Developed server-based web traffic statistical analysis tool using Flask, Pandas.• Wrote python scripts to parse XML documents and load the data in database. • Handled all the client-side validation using JavaScript.• Automated the existing scripts for performance calculations using Numpy and sql alchemy.