Working on Meta (Facebook and Instagram) Direct Integration and currently focusing on new features and enhancements in the integration pipeline.Accomplished Tasks:-- Implemented integration with Meta (Facebook and Instagram).-- Developed deep-level monitoring to extract valuable pipeline insights, and created Grafana dashboards with alerts for stakeholders.-- Successfully migrated the initial hive-based ETL implementation to a more efficient spark-based solution.-- Transitioned jobs from AWS EC2 and ECS to Kubernetes pods, improving scalability and maintainability.-- Optimized the AWS Athena table update module, significantly reducing processing time from hours to seconds.-- Led the migration of the entire system from AWS Cloud to Oracle Cloud Infrastructure (OCI).

Observability platformResponsibilities-- Developed API to ingest log, metric, and trace events into AWS Kinesis streams feeding Elasticsearch clusters. Handling petabytes of log data weekly, 400 billion metrics daily.-- Created API for centralized user access and authorization management in Grafana and Kibana for observability users.-- Automated tenant onboarding across all observability tools, including custom module development and Rundeck job creation.-- Automated tasks such as auth token generation, organization creation in Grafana, index pattern setup in Kibana, and access revocation.

Designed and Implemented a Datamart for One of India's Largest Equity Firms:Responsibilities:-- Collaborated with the client to understand requirements, and finalize the schema for the datamart-- Built custom ETL pipelines for various data sources(Oracle DB, PostgreSQL, MySQL, and Excel files)-- Developed custom data quality checks and data transformation modules -- Developed a module to store logically joined data in the datamartAdverse News Analysis:Responsibilities:-- Collaborated with the client to understand requirements.-- Implemented a crawler to download financial news articles-- Developed a custom filter and entity extraction module-- Built a module to verify if extracted entities or related entities are active customers.Impact: Automated manual news analysis, enabling proactive actions against risky clients.

Designed and developed an AI-based smart scan solution for India's largest equity firmInput: Scanned images, photos, or PDF files of financial statements and invoicesResponsibilities:-- Utilized open-source OCR for text extraction from scanned documents-- Implemented text extraction module for PDF documents using pdfminer-- Developed a data enrichment module to enhance the extracted data-- Created a custom layout identification and formatting module-- Built a module to organize and consolidate processed data into a specified tabular format