* Developed spark-Scala components and successfully integrated with EDGE framework to handle around 6000+ spark jobs per day.* Developed Automatic Airflow DAG builder using Python to generate the Airflow DAGs dynamically.* Optimized the EDGE framework performance by creating materialized views in PostgreSQL.* Successfully integrated the spark-scala components with Azure Synapse.* Ingested huge volumes of clinical research data into Snowflake as per customer needs.* Developed a custom component that can generate complex XML and JSON files. Also, this will support parquet and CSV files.* Developed Python script to flatten heavily nested JSON files.* Configured custom GitLab runners and successfully implemented the CICD pipeline to deploy the builds and improved its performance by leveraging the GitLab cache.* Developed spark-Scala component to record the change data capture.* Successfully migrated and productized the legacy systems to the Big data platform.* Developed Snowpark(snowflake) application using Scala to generate complex xml&json files.* Engineered DBT models and optimized DBT macros for efficient XML and JSON file generation, significantly enhancing performance and throughput.* Led the seamless migration of a Native Spark Scala application onto the Snowflake environment.* Implemented Python User-Defined Functions (UDFs) within Snowflake.

* Architected the Airflow 2.0 implementation in the CDS Platform.* Migrated the Airflow DAGs to Airflow 2.0 and optimized the DAGs by using the latest upgrades in the DAGs.* Implemented the Automated CICD pipeline for DAG deployment.* Written Airflow DAG to clean up the Airflow logs from the docker containers to improve performance.* Developed terraform modules to provision the AWS resources.* Imported the legacy AWS infrastructure to terraform code and brought it under terraform control.* Created spark-scala data pipelines using Databricks to conduct exploratory data analysis and build automated report generation.

• Quantexa• Spark• Gradle• Scala• Oracle• Entity Resolution• Airflow• S3• Elasticsearch• Docker• Kubernetes• GitLab CI/CD• AppDynamics• Azure Databricks• Azure Blob• Splunk• Grafana* Coordinated the design, development, and integration of the Quantexa project, consolidating internal bank data and external third-party data to calculate risk scores and facilitate entity resolution.* Developed robust ETL pipelines, extracting internal banking data from Oracle using Spark and Scala, and ingesting it into the S3 bucket, resulting in a streamlined data processing workflow.* Transformed raw data into Parquet format, implemented data validations, created Scala case class models, and executed custom Scala functions to cleanse address and name fields, ensuring data consistency.* Orchestrated end-to-end ETL processes and successfully deployed them into production using Airflow, optimizing operational efficiency.* Established a CI/CD pipeline, automating build processes and application deployment, contributing to a more agile and efficient development environment.* Utilized Spark and Scala to ingest processed data into Elasticsearch, enhancing data accessibility and search capabilities.* Deployed Spark Docker images to Kubernetes containers, demonstrating proficiency in containerized application deployment.* Developed Java applications using AWS SDK to seamlessly transfer files from S3 to various target locations, improving data distribution processes.* Integrated AppDynamics to monitor Microservices running on Kubernetes containers, ensuring optimal performance and proactively addressing any potential issues.* Contributed to the creation of elements and compounds integral to the entity resolution process, further enhancing the project's overall success.* Developed a hybrid data pipeline leveraging both an on-premise Hadoop cluster and Azure Databricks to process sensitive and non-sensitive data separately.

Data Engineer, Capgemini Ltd: DBS Bank, Hyderabad, IndiaWorking with DBS bank (Client) on Digibank App, which helps customers move from traditional banking to paperless banking. In this project, we migrate a traditional warehouse system to a new era data mart built on a newer data platform stack. This transformation helps the bank bring business logic to the presentation layer, which can be used to visually track, analyze, and display key performance indicators (KPI).* Ingested massive volumes of data from various core banking source systems like Finacle, Kony, etc., using Spark ingestion framework into ADA platform's Storage in S3.* Developed data pipelines in PySpark to support the needs of Data Science and Business Analyst teams.* Involved in data mapping and metadata preparation for the new data models using an enterprise metadata Governance tool called collibra.* Written pyspark and Teradata TPT scripts to extract and migrate the Legacy data from the old platform (BIP) to the new platform's (ADA) S3 storage.* Provisioned the GCP Dataproc cluster using the Airflow DAG for running the spark jobs and deleted the cluster once the job is done.* Created Airflow DAGS for the spark jobs and successfully deployed them into the production environment.* Written data reconciliation scripts to identify discrepancies in the legacy data.* Used Presto on top of Hive tables to achieve significantly faster performance for OLAP queries.* Used alluxio file system for speeding up the spark reading and writing operations.* Involved in tuning the performance of the long-running spark jobs.* Successfully implemented the SCD type-2 logic in hive tables.* Involved in unit testing, deployment, and change request management activities.