Data Engineer
Current- Demonstrated expertise in ETL processes utilizing Azure Data Factory, T-SQL, Spark SQL, U-SQL, and efficient data ingestion into Azure services.
- Proficient in managing Hadoop metadata and optimizing Spark clusters on Azure Databricks; adept at troubleshooting Azure customer issues.
- Skilled in data transfer from various sources to HDFS, leveraging Kafka, Zookeeper, and Apache Nifi for effective data movement.
- Conducted data migrations to Azure Data Factory and Lake, employing Kafka and Spark Streaming for real-time data processing and ingestion.
- Specialized in Snowflake Schema design, PySpark integrations, and OLAP cube development using Azure SQL Warehouse.
- Developed robust web applications using the Django Framework, focusing on the model-view-controller architecture.