Divya P Email & Phone Number

Tampa, Florida, United States

*Involved in writing custom Map Reduce, Pig and Hive programs.* Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions* Created Yaml files for each data source and including glue table stack creation.* Develops training classes for new telecommunications products.* Maintains database inventory for telecommunications equipment, telephone lines, and circuit.* Worked on a python script to extract data from Netezza databases and transfer it to AWS S3* Developed Lambda functions and assigned IAM roles to run python scripts along with various triggers(SQS, Event Bridge, SNS)* Experience developing Kafka producers and Kafka Consumers for streaming millions of events persecond on streaming data.* Experience in working with products teams to create various store level metrics and supporting datapipeline written in GCP's big data stack.* Experience in GCP Dataproc, GCS, Cloud functions, Cloud SQL & Big Query.* Implement slowly changing dimensions (SCD), Change data capture(CDC)using data integrator.* Develop and deploy ETL logic with Oracle Data Integrator.* Experience in Loading Essbase metadata with Oracle Data Integrator.* Involved in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Zookeeper, SQOOP, flume, Spark, Impala, and Cassandra with Horton work distribution.

Vernon Hills, Illinois, United States

* Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines* Migrate databases to cloud platform SQL Azure and as well the performance tuning.* Installed and Configured Sqoop to import and export the data into Hive from Relational databases. * Expertise in Python and Scala, user-defined functions (UDF) for Hive and Pig using Python. Involved in creating HiveQL on HBase tables and importing efficient work order data into Hive tables* Extensive experience on Hadoop ecosystem components like Hadoop, Map Reduce, HDFS, HBase, Hive, Sqoop, Pig, Zookeeper and Flume. * Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in in Azure Data bricks.* Experienced in ETL concepts, building ETL solutions and Data modelling* Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in java andScala for data cleaning and pre-processing* Extensive usage of Azure Portal, Azure PowerShell, Storage Accounts, Certificates and Azure DataManagement.* Used OOZIE Operational Services for batch processing and scheduling workflows dynamically.* Worked on architecting the ETL transformation layers and writing spark jobs to do the processing.* Configured Zookeeper, worked on Hadoop High Availability with Zookeeper failover controller, addsupport for scalable, fault-tolerant data solution.* Aggregated daily sales team updates to send report to executives and to organize jobs running on Spark clusters

Columbia, Missouri, United States

* Extensively used Agile methodology as the Organization Standard to implement the data Models * Created several types of data visualizations using Python and Tableau.* Extracted Mega Data from AWS using SQL Queries to create reports. * Performed reverse engineering using Erwin to redefine entities, attributes and relationships existing database.* Implemented the Spark Scala code for Data Validation in Hive* Implemented the automated workflows for all the jobs using the Oozie and shell script.* Used Spark SQL functions to move data from stage hive tables to fact and dimension tables.* Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results. * Defined Kafka Zookeeper offset storage.* Involved in Data modelling and design of Data Warehouse in star schema methodology with conformed and granular dimensions and FACT tables.* Migrate data from on-premises to AWS storage buckets

Dallas, Texas, United States

* Worked on development of data ingestion pipelines using ETL tool, Talend & bash scripting with big data technologies including but not limited to Hive, Impala, Spark, Kafka, and Talend.* Worked with multiple storage formats (Avro, Parquet) and databases (Hive, Impala, Kudu).* Working knowledge of cluster security components like Kerberos, Sentry, SSL/TLS etc.* Experience in developing scalable & secure data pipelines for large datasets.* Gathered requirements for ingestion of new data sources including life cycle, data quality check, transformations, and metadata enrichment.* Supported data quality management by implementing proper data quality checks in data pipelines.* Delivered data engineer services like data exploration, ad-hoc ingestions, subject-matter-expertise to Data scientists in using big data technologies.* Build machine learning models to showcase Big data capabilities using Pyspark and MLlib.* Enhancing Data Ingestion Framework by creating more robust and secure data pipelines.* Implemented data streaming capability using Kafka and Talend for multiple data sources.* Involved in the development of agile, iterative, and proven data modeling patterns that provide flexibility.

Hyderabad, Telangana, India

* Responsible for building scalable distributed data solutions using Hadoop.* Developed Spark jobs and Hive Jobs to summarize and transform data.* Developed a different kind of custom filters and handled pre-defined filters on HBase data using API.* Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.* Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive and then loading data into HDFS.* Exporting of a result set from HIVE to MySQL using Sqoop export tool for further processing.* Collecting and aggregating large amounts of log data and staging data in HDFS for further analysis.* Experience in managing and reviewing Hadoop Log files.* Involved in converting Hive/SQL queries into Spark transformations using Spark data frames, Scala and Python.* Experienced in developing Spark scripts for data analysis in both python and Scala.* Wrote Scala scripts to make spark streaming work with Kafka as part of spark Kafka integration efforts.* Created reports in TABLEAU for visualization of the data sets created and tested Spark SQL connectors.