*Involved in writing custom Map Reduce, Pig and Hive programs.* Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions* Created Yaml files for each data source and including glue table stack creation.* Develops training classes for new telecommunications products.* Maintains database inventory for telecommunications equipment, telephone lines, and circuit.* Worked on a python script to extract data from Netezza databases and transfer it to AWS S3* Developed Lambda functions and assigned IAM roles to run python scripts along with various triggers(SQS, Event Bridge, SNS)* Experience developing Kafka producers and Kafka Consumers for streaming millions of events persecond on streaming data.* Experience in working with products teams to create various store level metrics and supporting datapipeline written in GCP's big data stack.* Experience in GCP Dataproc, GCS, Cloud functions, Cloud SQL & Big Query.* Implement slowly changing dimensions (SCD), Change data capture(CDC)using data integrator.* Develop and deploy ETL logic with Oracle Data Integrator.* Experience in Loading Essbase metadata with Oracle Data Integrator.* Involved in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Zookeeper, SQOOP, flume, Spark, Impala, and Cassandra with Horton work distribution.

* Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines* Migrate databases to cloud platform SQL Azure and as well the performance tuning.* Installed and Configured Sqoop to import and export the data into Hive from Relational databases. * Expertise in Python and Scala, user-defined functions (UDF) for Hive and Pig using Python. Involved in creating HiveQL on HBase tables and importing efficient work order data into Hive tables* Extensive experience on Hadoop ecosystem components like Hadoop, Map Reduce, HDFS, HBase, Hive, Sqoop, Pig, Zookeeper and Flume.  * Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in in Azure Data bricks.* Experienced in ETL concepts, building ETL solutions and Data modelling* Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in java andScala for data cleaning and pre-processing* Extensive usage of Azure Portal, Azure PowerShell, Storage Accounts, Certificates and Azure DataManagement.* Used OOZIE Operational Services for batch processing and scheduling workflows dynamically.* Worked on architecting the ETL transformation layers and writing spark jobs to do the processing.* Configured Zookeeper, worked on Hadoop High Availability with Zookeeper failover controller, addsupport for scalable, fault-tolerant data solution.* Aggregated daily sales team updates to send report to executives and to organize jobs running on Spark clusters

* Extensively used Agile methodology as the Organization Standard to implement the data Models * Created several types of data visualizations using Python and Tableau.* Extracted Mega Data from AWS using SQL Queries to create reports. * Performed reverse engineering using Erwin to redefine entities, attributes and relationships existing database.* Implemented the Spark Scala code for Data Validation in Hive* Implemented the automated workflows for all the jobs using the Oozie and shell script.* Used Spark SQL functions to move data from stage hive tables to fact and dimension tables.* Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results. * Defined Kafka Zookeeper offset storage.* Involved in Data modelling and design of Data Warehouse in star schema methodology with conformed and granular dimensions and FACT tables.* Migrate data from on-premises to AWS storage buckets

* Worked on development of data ingestion pipelines using ETL tool, Talend & bash scripting with big data technologies including but not limited to Hive, Impala, Spark, Kafka, and Talend.* Worked with multiple storage formats (Avro, Parquet) and databases (Hive, Impala, Kudu).* Working knowledge of cluster security components like Kerberos, Sentry, SSL/TLS etc.* Experience in developing scalable & secure data pipelines for large datasets.* Gathered requirements for ingestion of new data sources including life cycle, data quality check, transformations, and metadata enrichment.* Supported data quality management by implementing proper data quality checks in data pipelines.* Delivered data engineer services like data exploration, ad-hoc ingestions, subject-matter-expertise to Data scientists in using big data technologies.* Build machine learning models to showcase Big data capabilities using Pyspark and MLlib.* Enhancing Data Ingestion Framework by creating more robust and secure data pipelines.* Implemented data streaming capability using Kafka and Talend for multiple data sources.* Involved in the development of agile, iterative, and proven data modeling patterns that provide flexibility.

* Responsible for building scalable distributed data solutions using Hadoop.* Developed Spark jobs and Hive Jobs to summarize and transform data.* Developed a different kind of custom filters and handled pre-defined filters on HBase data using API.* Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.* Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive and then loading data into HDFS.* Exporting of a result set from HIVE to MySQL using Sqoop export tool for further processing.* Collecting and aggregating large amounts of log data and staging data in HDFS for further analysis.* Experience in managing and reviewing Hadoop Log files.* Involved in converting Hive/SQL queries into Spark transformations using Spark data frames, Scala and Python.* Experienced in developing Spark scripts for data analysis in both python and Scala.* Wrote Scala scripts to make spark streaming work with Kafka as part of spark Kafka integration efforts.* Created reports in TABLEAU for visualization of the data sets created and tested Spark SQL connectors.