• Worked with users to identify the most appropriate source of record required to define the asset data for financing.• Enhanced processing of large datasets of over 10 TB using Hadoop, Apache Spark, PySpark, and Kafka achieving a 40% increase in efficiency and a 30% reduction in server load.• Implemented and supported complex Airflow DAGs, scheduling and automating ETL processes with Python in AWS Managed Workflows for Apache Airflow (MWAA) ecosystem, significantly reducing manual intervention and enhancing data pipeline efficiency.• Streamlined data processing to AWS, performing extensive data analysis on complex datasets using AWS Redshift, and Optimized SQL queries to reduce processing time and costs.• Designed and Developed ETL Processes in AWS Glue to migrate data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.• Responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark.• Utilized Spark's RDDs and Data Frames for high-performance parallel processing and used Spark Streaming to process and analyze real-time data feeds on the Hadoop platform, resulting in real-time analytics and insights.• Ingested customer data from Data Lake, and Apache Hive into the Oracle database, ensuring smooth data integration for the company's analytical needs.• Handled various data formats including YAML, JSON, Parquet, XML, and CSV across AWS platforms ensuring seamless data integration and analysis in security-related datasets.• Specialized in data warehousing solutions, managing both cloud-based (AWS Redshift, Big Query, Azure SQL Data Warehouse) and on-premises (Oracle, PostgreSQL, MYSQL, MongoDB) enterprise data warehouses and data marts, incorporating star and snowflake schema modeling and implementing Type 1 and Type 2 SCDs for robust historical data.

• Led the design and implementation of a complete ETL pipeline using Azure services, ensuring data accuracy and availability for critical banking operations.• Developed and maintained a centralized data warehouse in Azure Synapse Analytics, integrating data from various sources and enabling advanced analytics and reporting capabilities for the banking team.• Worked on migrating two data-intensive applications from on-prem servers to Azure App Service environments while utilizing Azure SQL, Blob Store, and Databricks for data storage and processing.• Proficient in Databricks for Lakehouse architecture implementation. Skilled in metadata optimization for enhanced data governance.• Advanced Databricks proficiency for big data processing. Experienced in metadata management for reliable insights.• Utilized the capabilities of Azure Functions to perform event-driven data transformations and data transfers from Oracle to Azure SQL DB.• Utilized a range of tools (Sqoop, Spark-Streaming/SQL, Kafka, Flume) for data ingestion and processing, ensuring seamless integration of structured, semi-structured, and unstructured data.• Experienced with NoSQL Databases like HBase as well as other ecosystems like Zookeeper, Oozie, Impala, Strom, Spark-Streaming/SQL, Kafka, and Flume.• Designed and implemented scalable and secure data solutions using Azure Data Lake, Azure Synapse Analytics, and Azure Data Factory.• Experienced extensively in creating and managing Hive tables and implementing partitions and buckets for efficient data loading and analysis.• Proficient in designing database schemas in Hive, MySQL, and other systems, and in processing large datasets via importing and exporting between databases and HDFS.• Developed Hive and Bash scripts for source data validation and transformation.

• Utilized Elasticsearch to design and implement a scalable log data storage and retrieval system, enabling efficient querying and analysis of log data for application performance testing purposes.• Configured Logstash pipelines to ingest, parse, and enrich log data from trading applications and infrastructure components, ensuring seamless integration with Elasticsearch for centralized log management.• Collaborated with cross-functional teams to define and prioritize requirements for new features and enhancements in trading applications, leveraging Elasticsearch for real-time data retrieval and visualization.• Implemented log-based performance testing strategies using Elasticsearch and Logstash, enabling the identification of performance bottlenecks, latency issues, and resource constraints in trading systems.• Automated log data ingestion and processing workflows using Logstash input plugins and scheduled Logstash pipelines, reducing manual intervention and improving operational efficiency.• Developed custom monitoring dashboards and alerts to track key metrics, detect anomalies, and proactively identify issues in data ingestion, processing, and storage.• Experienced in designing and implementing Extract, Transform, and Load (ETL) processes using ADF.• Implemented and configured data monitoring tools such as Grafana to monitor the health, performance, and availability of data pipelines and systems.• Enhanced efficiency of data movement and transformation by building and optimizing data pipelines in Azure Data Factory, conducting performance tuning to identify bottlenecks, and implementing enhancements for improved throughput and reduced latency.• Established CI/CD pipelines for Databricks notebooks and jobs, enabling automated testing, deployment, and rollback of changes.