API Development: Developed API's for various reports over 4 different social media platforms that fetch around 100 GB of data per day for our client base. Used REST API's in combination with Apache Airbyte.Schema Evolution for Diverse Sources:Developed flexible schema evolution scripts in PySpark for both JSON and Starburst data sources, enabling seamless handling of schema changes, data type casting, and integration into Starburst tables.Iceberg Tables scheduled maintenance:Implemented and productionized a Glue job to optimize Iceberg tables by merging small partitions, removing old snapshots, and maintaining metadata, enhancing table efficiency and query performance

AWS end-to-end PySpark Bulk ingestion pipeline• Developed a Data Ingestion Pipeline integrating Excel frontend with Pysparkand Snowflake backend technologies for seamless bulk data ingestions.• Implemented Spark optimization techniques such as caching and broadcastjoins resulting in a 30% decrease in processing time for handling a daily load ofaround 30,00,000 records from 13 data sources.• Implemented scheduling capabilities, allowing forthe efficient orchestration ofdata pipelines, optimizing data processing workflows.Tech Stack Used: Pyspark, AWS EMR, AWS Lambda, AWS Glue, AWS S3, AWSEventBridge, AWS Step Functions, Snowflake Cloud, Github CI/CDOneprem to Snowflake CLoud Migrations• Led seamless migration of on-premises data pipelines (Alteryx, Python, Excel)ETL Processes to Snowflake SQL Views, enabling nearreal-time processing for13 sources.• Implemented Tableau Server and Dashboards for Batch and Near Real TimeBI needs Improved Data Analytics Process from 4 hours perreport to 10minutes,resulting in a 96% reduction in manual work.Tech Stack Used: Data modeling, Snowflake, Alteryx, AWS Redshift, PySpark, AWSEMR, AWS LambdaAutomated Snowflake View Generator• Created an Python API service to generate dynamic SQL views on Snowflake.Tech Stack Used: Python, Snowflake

Handling all social media accounts, blogs and landing pages for the rotract club of JaiHind college