Work closely with Business Analysts and Product Owner to understand the requirements. I have developed applications using spark to implement various aggregation and transformation functions of Spark RDD and Spark SQL. Used Joins in SPARK to make smaller datasets into large ones without shuffling data across nodes. Developed Spark Streaming jobs using Python to read messages from Kafka. Downloaded JSON files from AWS S3 buckets. Implemented ETL using AWS RedShift/Glue. Used Spark Streaming to receive real-time data from Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra Prototyped analysis and joining of customer data using Spark in Scala and processed it to HDFS Implemented Spark in EMR for processing Big Data across our One Lake in AWS System We consumed and processed data from DB2. Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources. Used Scala components to implement the credit line policy based on the conditions applied on spark data frames. Data sources are extracted, transformed, and loaded to generate CSV data files with Python programming and SQL queries. Used pandas UDF like building the array contains, distinct, flatten, map, sort, split, and overlaps for filtering the data Prototyped analysis and joining of customer data using Spark in Scala and processed it to HDFS Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connect to Tableau for generating interactive reports using Hive server2. Built NiFi flows for data ingestion purposes. Ingested data from Kafka, Microservices, and CSV files from edge nodes using NIFI flows. Implementing and orchestrating data pipelines using Oozie and Airflow. Building automated pipelines using Jenkins and Groovy scripts Using shell commands to push the environment and test files AWS using Jenkins automated pipelines

Designed, developed, and tested data pipelines utilizing Informatica Cloud to facilitate seamless migration from on-premises infrastructure to Azure cloud services. Ensured data quality by validating raw data and transforming it into structured formats using dbt (data build tool) models. Executed various tasks including data extraction from disparate sources, database connectivity, and SQL query execution to support data processing requirements. IMPLEMENTED custom macros, models, and tests to optimize data workflows and enhance efficiency. Created and managed internal and external tables within THE Snowflake data warehouse using dbt packages for centralized data storage. Loaded data into Snowflake warehouse using seed data and maintained data integrity throughout the process. Spearheaded the implementation of continuous integration (CI) jobs in dbt to automate and streamline data pipeline workflows. Authored SQL scripts to support warehouse administration tasks, ensuring smooth operation and management of the data infrastructure.