Data Engineer
• Accessing Data Lake in Databricks using Azure AD Service principal application.• Developed Pyspark code to read, transform and write data for Batch Processing.• Build ETL Script to implement Spark Structured Streaming that guarantees exactly-once stream processing.• Designing Spark Cluster for a Streaming ETL Process.• Autoscaling in spark clusters and Spot instances.• Understanding the use of inferSchema.• Designing Partitioning strategies in spark Using repartition() and coalesce() functions.• Involved in Spark SQL Optimizations.• Creates Notebooks with "Bucketing" Optimization technique to prevent shuffling and sorting of data during Compute heavy Operations.• Working with OPTIMIZE and ZORDER Commands to speed up queries by changing the layout of the data stored in the cloud storage.• Flip data from Rows to columns using Pivoting.• Building and maintaining data pipelines for data integration and processing.• Optimizing data processing performance through tuning and monitoring.• Collaborating with other teams to provide data for analytics and reporting.• Knowledge of data integration tools such as Azure Data Factory and Azure Databricks.• Ability to work with large datasets and perform data analysis.• Strong problem-solving and troubleshooting skills.
