1. Designed and developed a sophisticated data integration platform using Snowflake and Apache Kafka for streaming data ingestion, coupled with dbt for data transformations, achieving near real-time analytics capabilities.2. Led the end-to-end design and implementation of a multi-terabyte enterprise data warehouse in Snowflake, incorporating advanced partitioning and clustering techniques to optimize query performance for over 100 concurrent users.3. Automated end-to-end data pipeline monitoring and alerting using a combination of Snowflake’s Information Schema, SnowAlert, and custom Python scripts, leading to a 70% reduction in pipeline downtime and faster incident resolution.4. Established data validation checkpoints within ETL workflows using Snowflake’s task and stream features, ensuring that only validated and error-free data progressed through the pipeline.5. Managed and accommodated over 100 TB of data within BigQuery, facilitating analytical queries for 50+ concurrent users, and enhancing decision-making processes.6. Designed, developed, and deployed a specialized data pipeline managing 2 TB/day from 50+ sources, achieving a 40% reduction in processing time. Leveraged Iceberg tables for data analytics, significantly enhancing read performance by 10 times, streamlining data processing, and accelerating insights retrieval.7. Designed, developed, and deployed real-time replication of database changes to Apache Iceberg tables, efficiently managing 500+ events per second without the need for Spark Kafka or a streaming platform.8. Engineered and orchestrated a robust production-level project, leveraging Spark, Kafka, and NoSQL to process 100K+ records per second. Developed Python programs to facilitate real-time message handling, email registration, and authentication processes, ensuring secure multi-producer and multi-consumer interactions.

1. Designed and developed a comprehensive catalog comparison tool in an AWS environment, harnessing Spark and Scala's robust functionalities. Orchestrated the process using Airflow, enabling automated comparison among e-support, platform, and SDP catalogs. Successfully identified discrepancies in software bundles (SWBs) within a dataset of 10+ million records, ensuring catalog consistency and data accuracy.2. Engineered a comprehensive framework for the seamless ingestion of diverse database sources into AWS S3.3. Processed and standardized data from 15+ different formats, ingesting 1.5 TB of data daily, supporting analytical queries efficiently.4. Established a metadata management infrastructure using AWS Glue and Trino. Cataloged ingested data, optimizing analytical query efficiency and facilitating streamlined information retrieval, improving query response times by 30%.5. Implemented optimization techniques in Spark jobs, fine-tuning through broadcasts, caching, and resource allocation adjustments. Rigorously tested and identified optimal resources for each job, resulting in approximately 15% cost savings on AWS.6. Designed, developed, and deployed a robust data ingestion architecture using Cloud Pub/Sub, enabling seamless and scalable streaming data intake at a rate of 1 TB/hour, ensuring real-time availability for downstream processes.7. Developed and Deployed Apache Kafka for real-time data processing, enabling seamless ingestion, processing, and analysis of 100 million records per batch processing cycle, optimizing system performance and enabling real-time decision-making.

1. Developed and deployed data architecture system enabling seamless ingestion of 1 TB/day from 10+ diverse sources into AWS S3. Utilized Spark RDD for complex transformations across datasets, processing over 100 million records daily.2. Employed Python's Plotly and Matplotlib libraries to craft comprehensive data visualizations and generate insightful reports.3. Leveraged AWS Glue to efficiently inspect and analyze the ingested data, enhancing visibility and accessibility for further insights.4. Created Python-based data validation tools, ensuring data accuracy and compliance, resulting in a 90% improvement in data quality.5. Migration of Spark RDD-based code to Spark DataFrame API to optimize functionality and capitalize on robust capabilities.6. Orchestrated 15+ data pipelines efficiently with Apache Airflow, managing diverse data sources and destinations reliably.

1. Leveraged Java as the primary programming language, windows PowerShell for scripts, Git for version control, and Jenkins as the automation tool to demonstrate the steps to automate the manual parts of configuration, integration, and deployment processes of D2 (an advanced, intuitive and configurable content-centric client for Documentum).2. Developed Python automation script to automate GIT functionalities like cloning and updating the repository.3. Provided support to teams by demonstrating best practices for utilizing Git and GitHub, including branching techniques for optimal code management.4. Assisted teams and conducted demonstrations on leveraging Jenkins for implementing Continuous Integration/Continuous Delivery (CI/CD) within projects.

During my internship at Bengal Chemicals & Pharmaceuticals Ltd., I had the opportunity to work on the implementation of an Enterprise Resource Planning (ERP) system, focusing on the modules related to purchase, production, and sales.