*Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries for Flagstar mobile applications. * Created Tables, Stored Procedures, and extracted data using T-SQL for Flagstar users whenever required. * Performed data analysis and design, and created and maintains large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDR *I have written shell script to trigger data Stage jobs. *In-depth experience in automating various PySpark, Hive, Bash and python applications using Airflow and Oozie*ETL pipelines in and out of data warehouse using combination of Python and Snowflakes Snow SQL Writing SQL queries against Snowflake. *Responsible for Design, Development, and testing of the database and Developed Stored Procedures, Views, and Triggers

* Utilized AWS services with focus on big data analytics, enterprise data warehouse and business intelligence solutions to ensure optimal architecture, scalability, flexibility.* Designed AWS architecture, Cloud migration, AWS EMR, DynamoDB, Redshift and event processing using lambda function.* Implemented usage of Amazon EMR for processing Big Data across a Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3)* Used AWS system manager to automate operational tasks across AWS resources.* Wrote Lambda function code and set Cloud watch Event as trigger with Cron job Expression.* Connected Redshift to Tableau for creating dynamic dashboard for analytics team.* Setup connection between S3 to AWS Sage Maker ML (Machine Learning platform) is used for predictive analytics and uploading inferenced data to redshift* Deployed the project on Amazon EMR with S3 connectivity for setting backup storage.* Conducted ETL Data Integration, Cleansing, and Transformations using AWS glue Spark script.* Wrote Python modules to extract data from the MySQL source database.

* Experience in migrating existing legacy applications into optimized data pipelines using Spark with Scala and Python, supporting testability and observability.* Experience in developing scalable real-time applications for ingesting clickstream data using Kafka Streams and Spark Streaming.* Developed optimized and tuned ETL operations in Hive and Spark scripts using techniques such as partitioning, bucketing, vectorization, serialization, configuring memory and number of executors.* Worked on Talend integrations to ingest data from multiple sources into Data Lake.* Developed an MVP on exporting data to Snowflake to understand usages and benefits for migration.* Experience in automating end to end Hadoop jobs using Oozie applications in an optimized way.* Developed various scripting functionalities using Shell Script and Python.* Analyzed the SQL scripts and designed it by using PySpark SQL for faster performance.* Developed APIs for quick real-time lookup on top of HBase tables for transactional data.* Built Jupyter notebooks using PySpark for extensive data analysis and exploration.* Implemented code coverage and integrations using Sonar for improving code testability. * Pushed application logs and data streams logs to Kibana server for monitoring and alerting purpose.* Developed spark applications in python (PySpark) on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables.* Exposure to Data Lake Implementation and developed Data pipelines and applied business logic utilizing Apache Spark.