Manisha R Email & Phone Number

Eagan, Minnesota, United States

• Involved in complete Big Data flow of the application starting from data ingestion upstream to HDFS, processing the data in HDFS and analyzing the data and involved. Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Cosmos Activity.• Developed Automation Regressing Scripts for validation of ETL process between multiple databases like AWS Redshift, Oracle, Mongo DB, T-SQL, and SQL Server using Python Collaborated with team members and stakeholders in design and development of data environment.• Experienced knowledge over designing Restful services using java-based APIs like JERSEY. Used airflow Operational Services for batch processing and scheduling workflows dynamically.• Experience in developing customized UDF’s in Python to extend Hive and Pig Latin functionality. Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.• Worked on performance tuning of data bricks delta tables. Stage the API or Kafka Data (in JSON file format) into Snowflake DB by Flattening the same for different functional services.• Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark databricks cluster. Used SSIS and T-SQL stored procedures to transfer data from OLTP databases to staging area and finally transfer into data-mart.

Foster City, California, United States

• Responsible for analyzing large data sets and derive customer usage patterns by developing new MapReduce programs using Java. Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources.• As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake. Use Spark SQL to load JSON data and create Schema RDD and loaded it into Hive Tables and handled structured data using Spark SQL.• Perform structural modifications using MapReduce, Hive and analyze data using visualization/reporting tools (Tableau). Develop solutions to leverage ETL tools and identify opportunities for process improvements using Informatica and Python.• Designed Kafka producer client using Confluent Kafka and produced events into Kafka topic. Evaluated big data technologies and prototype solutions to improve our data processing architecture. Data modeling, development, and administration of relational and NoSQL databases.• Experience in Developing ETL solutions using Spark SQL in Azure Databricks for data extraction, transformation and aggregation from multiple file formats and data sources for analyzing. Using Python in spark to extract the data from Snowflake and upload it to Salesforce on Daily basis.• Use python to write a service which is event based using AWS Lambda to achieve real time data to One-Lake (A Data Lake solution in Cap-One Enterprise). Worked on data processing and transformations and actions in spark by using python (PySpark) language.

Englewood, Colorado, United States

• Analyze the existing application programs and tune SQL queries using execution plan, query analyzer, SQL Profiler and database engine tuning advisor to enhance performance. Migrated on premise database structure to Confidential Redshift data warehouse.• Was responsible for ETL and data validation using SQL Server Integration Services. Wrote various data normalization jobs for new data ingested into Redshift. Implementing and Managing ETL solutions and automating operational processes.• Used Kafka functionalities like distribution, partition, replicated commit log service for messaging systems by maintaining feeds and Created applications using Kafka, which monitors consumer lag within Apache Kafka clusters.• Authoring Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks. Managed security groups on AWS, focusing on high-availability, fault-tolerance, and auto scaling using Terraform templates. • Along with Continuous Integration and Continuous Deployment with AWS Lambda and AWS code pipeline. Enforced referential integrity in the OLTP data model for consistent relationship between tables and efficient database design. • Installing and configuring Nagios check postgresql.pl plugins for monitoring PostgreSQL instances effectively. Used Hive SQL, Presto SQL, and Spark SQL for ETL jobs and using the right technology for the job to get done. Created various complex SSIS/ETL packages to Extract, Transform and Load data.

India

• Involved in complete Software Development Lifecycle (SDLC). Worked on different dataflow and control flow task, for loop container, sequence container, script task, executes SQL task and Package configuration.• Extensive use of Expressions, Variables, Row Count in SSIS packages. Created SSIS packages to pull data from SQL Server and exported to Excel Spreadsheets and vice versa. Created batch jobs and configuration files to create automated process using SSIS.• Data validation and cleansing of staged input records was performed before loading into Data Warehouse. Automated the process of extracting the various files like flat/excel files from various sources like FTP and SFTP (Secure FTP).• Design and implement multiple ETL solutions with various data sources by extensive SQL Scripting, ETL tools, Python, Shell Scripting, and scheduling tools. Data profiling and data wrangling of XML, Web feeds and file handling using python, Unix, and SQL. Interacted with Business analysts and stake holders to gather business requirements. Transferred data from various data sources /business systems including DB2, MS EXCEL,FLAT FILES ETC. Experienced in Data modelling and Data design processing.• Developed spark applications in python (PySpark) on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables. Deploying and scheduling reports using SSRS to generate daily, weekly, monthly, and quarterly reports. Loading data from various sources like OLEDB, flat files to SQL Server database Using SSIS Packages and created data mappings to load the data from source to destination.