Mani K Email & Phone Number

United States

Designed and set up Enterprise Data Lake to provide support for various use cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing data.Spearheading data engineering initiatives on GCP, including BigQuery, Dataproc, and GCS.Implementing ETL processes using PySpark, Python, and integrating with various data sources such as Hadoop, Oracle, and MySQL.Leveraging Snowflake for data warehousing and ensuring efficient data processing on Databrick.Collaborating in the development of comprehensive data pipelines and workflows.Managing and optimizing data storage in GCS and orchestrating data workflows using Apache Airflow

California, United States

Develop an ingestion and processing platform for terabytes of data using a variety of tools such as Spark, Spark Streaming, Kafka, Cassandra.Designing architecture and building new features into the existing product.Develop an ingestion and processing platform for terabytes of data using a variety of tools such as Spark, Spark Streaming, Kafka, Cassandra.Worked with the Data Lake team to ingest data from various sources into a data lake using tools like Teradata, Hive, Spark, and make it available for different ML applications using Hortonworks Data Platform.Involved in gathering business requirements, analyzing the project, and creating Use Cases and Design Document for new requirements.Converting long-running Hive queries to Spark Scala to improve performance.Worked with DevOps team to automate EMR cluster creation and destruction. Using DevOps tools like Jenkins, JFrog for automation, deployment, and management CI/CD pipelines

Hyderabad Area, India

Used HIVE to do transformations, joins and some pre-aggregations before storing the data into HDFS. Developed complex queries in HIVE to validate data migration of historical and incremental data between source and target database. Designed and implemented HIVE queries and functions for evaluation, filtering, loading, and storing of data. Written stored procedure for those reports which use multiple data sources. Added indexes to improve performance on tables. Provided L2 production support, monitored jobs using AutoSys for data warehousing projects. Coordinated with the L1 and L3 support team and resolved the incidents in a timely manner. Extensive experience in writing UNIX shell scripts and automation of the ETL processes using shell scripting.

Hyderabad Area, India

Analyzed more than 3000 cases of KPIs degradation. Identified root cause using data of counters and event/alarm logs. Managed and analyzed business problems to identify efficient cost-effective technical solutions. Analyzed patterns of traffic and KPIs for identification of degradation from data logs. Involved in Data Analysis, Data Mapping and Data Modeling and Performed Data integrity, Data Portability testing by executing SQL statements for the customer Data