1.) Utilised PySpark and Spark SQL to develop complex ETL workflows for data transformation, aggregation, and preparation for reports2.) Developed end-to-end data pipelines for daily report generation tasks, automating scheduling and execution and data movement to PostgreSQL and MS SQL Server databases.3.) Utilised Azure Serverless SQL Pool to create views for tables and aggregated reports, provide required access to the concerned users and query the files directly, enabling data-driven insights.4.) Engineered the enhancement of existing batch and real-time Change Data Capture (CDC) data pipelines, optimizing processing efficiency and ensuring data quality.5.) Used different data file formats i.e parquet, delta, CSV in accordance with the most suitable file format for different use cases.6.) Conducted data quality checks and assisted in monitoring processes, resolving anomalies and discrepancies to maintain data accuracy and consistency.

1.) Trade Data Volume Prediction - Used time-series analysis methods (i.e Exponential Smoothing, ARIMA,Facebook Prophet) to predict trade data volume for next 7 years.2.) Generation of Mutual Fund Alerts - Assisted in generating Mutual Fund Alerts of SEBI by implementing the logic using SQL and Shell-Scripting.3.) Identifying Connection between entities using Demographic data - Used record linkage method to identify connection between entities using SEBI’s demographic data and grouped the similar records into one group.