Developed Sqoop scripts to import, export data from relational sources and handled incremental loading on the customer data by date. Utilized Matillion ETL solution to develop pipeline that extract and transform data from multiple sources and loading to Snowflake. Developed and maintained data pipelines for real-time streaming data processing using technologies like Apache Kafka, Apache Flink, or AWS Kinesis.

Developed and optimized Flink jobs for high-throughput and low-latency requirements, handling large volumes of streaming data. Designed and implemented the infrastructure for the Data Lake, considering scalability, performance, and cost-effectiveness. Utilized Flink's capabilities to perform windowed aggregations, event time processing, and stateful computations in real-time ETL workflows.

Responsible for Cluster maintenance, adding and removing cluster nodes, Cluster Monitoring and Troubleshooting, Manage and review data backups, Manage and review Hadoop log files. Importing & exporting database using SQL Server Integrations Services (SSIS) and Data Transformation Services (DTS Packages). Responsible for maintaining data warehouse on Teradata, optimize index, modify table structure and etc. Worked on developing SQL DDL to create, drop and alter the tables.

Involved in performance tuning of Spark Applications for setting the right level of Parallelism and memory tuning. Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the R&D team. Utilized SQL scripts for automation purposes.