Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.• Optimize Databricks jobs for performance and scalability to handle big data workloads.• Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.• Implement best practices for data management, security, and governance within the Databricks environment. Collaborate with Data scientists to derive requirements for data pipeline implementationDesign and Deploy data pipelines using DLT, Jobs and AutoloaderHands on experience in performance tuning and optimization of pyspark jobsImplemented Databricks medallion architecture of enriching raw data, data cleansing and data governance principlesIntegrated reporting module with Kafka to make reporting dashboard near real time Performed platform management activities for Databricks like managing user permissions

Participate is designing a migration plan for moving 500 GB legacy data from on-premise DB2 to cloud using CDCImplemented scripts to load semi - structured and unstructured data into S3 bucket from on premise serverDesigned and implemented ELT pipelines for ongoing data feeds using Apache Airflow, pysparkImplemented Databricks medallion architecture for data cleansing and data governance principlesPerformed platform management activities for Databricks

Cloud Migration and modernization projectCreated ELT jobs for mainframe data feeds to load to Salesforce using pyspark, Hadoop and Hive stackCreate unit test cases for Scala and pyspark Jobs

Export large dataset from Oracle and DB2 databases into HDFSCreated python jobs to ingest dataImplemented transformations in pypark to load data from web server logs into HDFS for analysisCreated Hive tables and load data for analysis using Hive queriesCreated and worked on Sqoop jobs for incremental work loads

Responsibilities included attending workgroup meetings with Technical Leads, Project Architects and Project Manager and understand implementation scope and time linesParticipated in design walk through sessionImplemented data injection module using Spring Batch, to persist incoming data feeds from CSV format to Oracle Database tables.Implemented services layer using Spring MVC frameworkParticipated in code review sessionsResponsible for code elevations to higher environment following client specific changemanagement processSmoke Test the deployed application for standard functionality of the applicationPerformed the initial unit testing tests using JUnit during the development of code andfunctional testing of the business logic.Involved in the support and maintenance of the application during production phase