Senior Data Engineer
CurrentGathered requirements for Analysis, Design, Development, testing, and implementation of business rules. Created data pipelines and transformed data to a structure that is relevant to the problem by selecting appropriate techniques. Performed ETL operations for Data cleansing, Filtering, Standardizing, Mapping, and Transforming of Extracted data from multiple sources such as Azure Data Lake, and on-prem SQL DB. Developed Spark applications using PySpark and Spark-SQL for data transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns. Worked on predicting the cluster size, monitoring, and troubleshooting the Spark data bricks cluster.