Data Engineer
Current• Optimize data performance and scalability.• Scheduling Jobs/Notebooks. / Using Workflows to schedule and deploy the notebooks.• Worked on different different file format like CSV, JSON.• Checking Data Quality, Accuracy, Completeness and Handling Null and Corrupt records.• Developed and optimized ETL pipelines using Azure Databricks for large-scale data processing.• Implemented data transformation and cleansing operations with Pyspark, Spark SQL and Python.• Provided… Show more • Optimize data performance and scalability.• Scheduling Jobs/Notebooks. / Using Workflows to schedule and deploy the notebooks.• Worked on different different file format like CSV, JSON.• Checking Data Quality, Accuracy, Completeness and Handling Null and Corrupt records.• Developed and optimized ETL pipelines using Azure Databricks for large-scale data processing.• Implemented data transformation and cleansing operations with Pyspark, Spark SQL and Python.• Provided technical support for data-related issues and queries.• Participated in training sessions for new Azure Databricks features and updates. Show less
