● Optimized ETL pipeline by migrating from Compute Engine (VM) to managed services with Cloud Composer and Dataproc, enhancing performance, monitoring, and scalability. Leveraged Composer for improved data flow orchestration and Dataproc for parallelized data ingestion and transformation, reducing process time from 4 hours to 45 minutes● Implemented Oracle VPD in BigQuery using BigQuery AEAD and Low-level access policy to filter out the sensitive records while querying based on… Show more ● Optimized ETL pipeline by migrating from Compute Engine (VM) to managed services with Cloud Composer and Dataproc, enhancing performance, monitoring, and scalability. Leveraged Composer for improved data flow orchestration and Dataproc for parallelized data ingestion and transformation, reducing process time from 4 hours to 45 minutes● Implemented Oracle VPD in BigQuery using BigQuery AEAD and Low-level access policy to filter out the sensitive records while querying based on persona● Orchestrated data pipelines using Apache Airflow to load incremental data from Cloud Storage landing zone into BigQuery by applying transformations using Apache PySpark and tokenization to mask sensitive/PII data● Developed a custom environment refresh solution to refresh data in non-production environments. The solution was built to refresh many BigQuery datasets concurrently using ThreadPoolExecutor. This reduced the 30% manual work of the operations team.● Developed terraform scripts to automate infrastructure creation in GCP using GitLab CI/CD for Data Engineering workflows Show less

● Designed and developed an Inventory Visibility solution to track purchase order movements at the PO line level for Ocean & Air shipment mode. The solution helped to track PO performance, containers current locations, milestone delays, sending alerts, and generating reports on near real-time events which were earlier delayed by 1 day● Involved in a Data Foundation setup on BigQuery for one of the Retail and Supply Chain customers to carry out planning, and managing inventory… Show more ● Designed and developed an Inventory Visibility solution to track purchase order movements at the PO line level for Ocean & Air shipment mode. The solution helped to track PO performance, containers current locations, milestone delays, sending alerts, and generating reports on near real-time events which were earlier delayed by 1 day● Involved in a Data Foundation setup on BigQuery for one of the Retail and Supply Chain customers to carry out planning, and managing inventory effectively which reduces the manual process and increased the planning process efficiency by 45%● Wrote high-performance SQL scripts to convert business logic for UI and dashboard display, and implemented SCD Type 2 to maintain historical data in the data warehouse for reporting● Created pipelines for different load types like a Snapshot, Incremental delta load, and Appending delta load● Handled pipeline monitoring, package upgrades, issue resolution, and enhancements. Participated in requirements gathering, data modeling, and creating technical design documents (TDD) for customer use cases. Show less

● Designed and developed automated data pipelines using ELT/ETL processes for Jewellery domain customers to replicate data from various source systems such as MS SQL Server, and PostgreSQL into BigQuery using Cloud Composer and Dataproc● Built pipelines to handle Full and Incremental load data from different data sources by scheduling Airflow DAGs to trigger daily and weekly.● Built data pipelines using ELT process to replicate data from the SAP ECC system into BigQuery using… Show more ● Designed and developed automated data pipelines using ELT/ETL processes for Jewellery domain customers to replicate data from various source systems such as MS SQL Server, and PostgreSQL into BigQuery using Cloud Composer and Dataproc● Built pipelines to handle Full and Incremental load data from different data sources by scheduling Airflow DAGs to trigger daily and weekly.● Built data pipelines using ELT process to replicate data from the SAP ECC system into BigQuery using Data Fusion for one of the UAE customers● Writing high-performant SQL scripts to convert business logic into the desired data format to display on the web application(UI) and dashboards. SCD type-2 to maintain historical dimensions data in the data warehouse for reporting purposes● Regularly worked on pipeline monitoring, upgrading packages, resolving the issues, and enhancements if needed Show less

● Migrated an internal solutions infrastructure from IaaS to PaaS using services such as App Engine, Cloud Functions, Cloud Datastore, and integrated operational tools such as Monitoring and Logging● Developed a near real-time vision model using the Vertex-AI platform to detect and notify if objects on the shop shelf are misplaced for one of the Retail customers