Data Engineer
Current• Demonstrated expertise in ETL processes utilizing Azure Data Factory, T-SQL, Spark SQL, U-SQL, and efficient data ingestion into Azure services.• Proficient in managing Hadoop metadata and optimizing Spark clusters on Azure Databricks; adept at troubleshooting Azure customer issues.• Skilled in data transfer from various sources to HDFS, leveraging Kafka, Zookeeper, and Apache Nifi for effective data movement.• Conducted data migrations to Azure Data Factory and Lake, employing Kafka and Spark Streaming for real-time data processing and ingestion.• Specialized in Snowflake Schema design, PySpark integrations, and OLAP cube development using Azure SQL Warehouse.• Developed robust web applications using the Django Framework, focusing on the model-view-controller architecture.• Created Clusters using Azure Data Explorer (ADE) storing and processing the Batch and streaming data using ADF and querying the data using KQL (Kusto queries).• Crafted user-friendly website interfaces, creating views and templates with Python and Django's view controller and templating language.• Developed consumer-centric features and applications using a blend of Python, Django, Pyramid, Flask, Web2py, HTML, and other web technologies.• Gained substantial experience working with Cloudera, Horton Works, and Azure HDInsight platforms.• Implemented Flask-based web applications adhering to MVC architecture principles.• Utilized CSS Bootstrap for front-end development, enhancing the user interface of web applications.• Proficient in using Python collections for intricate manipulation and iteration over user-defined objects.• Enhanced Hadoop performance using Spark and automated test scripts development in Python.• Skilled in Azure cloud services, including Azure Storage, Azure Active Directory.