Contributed to the modernization of data solutions and the data stack at a multinational financial services corporation, one of the world's largest asset managers. Utilized stream and event-based approaches for enhancing data pipelines.Tech Stack:- AWS- Snowflake, MS SQL Server 2016- CDC (Oracle Golden Gate)- MSK (Kafka), Kafka Connect- Lambda Functions, S3, EC2, VPC, CloudWatch- Terraform, JenkinsAchievements:- Engineered a Change Data Capture (CDC) flow to Kafka (MSK) service, reducing the time to replicate any changes to as low as 10 seconds.- Transformed a scheduled job into a near real-time data feed with Kafka Connect, Snowpipe Streaming, and Dynamic tables, cutting the total integration time for critical business data from 12 hours to 5 minutes.- Maintained cross-team communication, engaging multiple teams to ensure the project's readiness for production.- Successfully curated the onboarding process for new team members and provided leadership for a team of 2 data engineers.

As a senior data engineer, took a leading role in building a data lake for a startup focused on digital identity creation. Integrated data from diverse systems (Jira, Github, HubSpot, Greenhouse, Helpdesk, etc.), and generated KPI metrics for teams and employees.Tech Stack:- AWS- Redshift, Redshift Spectrum, Athena- Glue (Jobs, Crawlers, Catalog, Workflows)- Lambda Functions, S3, Step Functions, CloudWatch, EventBridge, IAM- Terraform (Terragrunt), Gitlab CI\CD pipelinesAchievements:- Successfully pitched a lakehouse solution for a customer's use case during pre-sales, including estimating the timeline for the PoC phase.- Established a cost-efficient data platform using AWS serverless stack(Glue, Athena, Lambda), leading to a 75% reduction in monthly expenses.- Leveraged AWS Glue DynamicFrames to create a robust PySpark job, swiftly onboarding new raw datasets to the cleansed layer within 10-30 minutes, applying standard transformations for data preparation & metadata unification.- Automated platform component setup with Terraform for swift deployment of the entire platform within minutes.

Led the development of a lakehouse solution for the company's analytical platform using Azure toolset & Databricks. Additionally, played a key role in creating a media monitoring solution using the Databricks platform.Tech Stack:- Azure- Azure Synapse, Azure SQL Database (MS SQL Server)- Azure Databricks, Azure Data Factory (ADF)- Azure Data Lake Storage Gen2, Event Hubs, Azure Functions, Azure Monitor- Databricks MLflow- Terraform, Azure DevOpsAchievements:- Coordinated the development of a three-layered lakehouse on Azure Databricks, boosting integration speed and reducing costs by eliminating the need for Azure Synapse service.- Successfully migrated ML workloads to Databricks, optimizing notebook performance with Spark ML libraries, and integrating ML pipelines with the MLflow platform for better observability.- Addressed performance issues in the duplicate's definition pipeline by optimising existing logic with O(n²) complexity into an O(n) complexity hashing for images and LSH algorithm for similarity search within text messages.- Led and mentored a team of 3 data engineers and 1 data scientist, creating a structured learning path for Spark and Cloud competencies, fostering team development and knowledge sharing.

Designed and implemented data pipelines utilizing Google Cloud Platform's data offerings for the UK leading organization in the postal and logistics sector.Tech Stack:- GCP- BigQuery, Cloud SQL (PostgreSQL)- Dataflow (Apache Beam), Dataproc (Apache Spark), Apache NiFi- Cloud Composer (Apache Airflow)- Cloud Functions, Cloud Storage, PubSub, GKE- Terraform, JenkinsAchievements:- Engineered ingestion logic using Cloud Functions & Dataflow jobs, enabling event-based ingestion. Data added to Cloud Storage was ingested into BigQuery within 4 minutes range.- Redesigned complex translated Teradata ETL queries by using Airflow DAGs and appropriate BigQuery SQL scripts, drastically reducing job execution time from 3-10 hours to 10 minutes.- Optimized GCS and BigQuery storage models by implementing partitioning and nesting strategies, resulting in 15-40% improvement in query speed and cost reduction through minimized bytes scanned.- Integrated Terraform BigQuery modules into Jenkins CI/CD pipeline, streamlining schema control and modifications for BQ datasets and tables.- Implemented unit tests for Airflow jobs, significantly reducing the number of errors created during development.

Contributed to cross-functional initiatives, including the development of an analytical database from scratch and optimization of existing data warehouse solution for one of the biggest American security company.Tech Stack:- MS SQL Server 2012-2016, T-SQL- SSIS for data integrations, C# for advanced scripts- SSRS as the main reporting tool- Git (VSTS Repository), Azure DevOps for CI/CDAchievements:- Enhanced system efficiency tuning stored procedures, reducing the call stack from 20 to 3, and slashing execution time from 2 hours to 7 minutes.- Led the development and implementation of a Data Warehouse solution and integration flow, enabling the generation of B-level reports within a 10-second timeframe.- Conducted a comprehensive analysis of query patterns on the SQL Server database, implementing both row and column-oriented indexes to achieve nearly twice the performance improvement for existing queries.- Initiated and coordinated cross-team knowledge-sharing meet-ups to share knowledge and insights among engineers in different tech areas.- Collaborated closely with the back-end team to optimize Entity Framework generated queries for improved system performance.

Aided in migrating reports from legacy systems and collaborated with the mobile team to develop a synchronization mechanism for a healthcare software provider serving hundreds of hospices in the USTech Stack:- Sybase, MS SQL Server 2008-2012, T-SQL- DevExpress reports (C#)- Git (TFVC), Jenkins for CI/CD- Regdate tools: SQL Source Control, SQL Monitor, SQL Compare, SQL TestAchievements:- Engineered a synchronization mechanism, boosting sync process in ~20 times.- Accelerated development of stored procedures by creating a dynamic SQL script responsible procedures core logic definition.- Implemented robust unit tests for SQL Server procedures, decreasing the number of errors caused by code modifications.- Established source control for all database objects using RedGate, enabling version control for database objects.