Data Engineer
CurrentDedicated IT professional with over 5 years of experience in DatabaseManagement, Data engineering Using Azure, Big Data Hadoop andApache Spark .Proficient in SQL, ETL processes, and data pipelinedevelopment. Seeking a challenging data engineering role to leveragemy expertise and drive data-driven success for the organization.Key Responsibilities:• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to ELT data from different sources like Azure SQL, Blob storage, APIs, Azure SQL Data warehouse• Developed Spark applications using Scala and Spark-SQL using Databricks• Used HDInsight, Jupyter notebooks, and Spark-Shell to develop, test, and Optimize Spark jobs• Performance tuning of Spark Applications for setting the right Batch Interval time, the correct level of Parallelism and memory tuning, and writing UDFs• Deployed and tested (CI/CD) our developed code using Visual Studio Team Services (VSTS).• Participated in creating HDInsight cluster and Databricks Storage Account with End-to-End policies• Used EventHub for Stream Data Ingestion using Avro into speed layer processing and Redshift• Participated in preparing Frameworks for developing Unit Tests cases and Integration testing• Developed Spark Jobs using Spark SQL queries, Data frames, Datasets APIsSkills: Azure Databricks • Event Hub • MS SQLSever • Data Engineering • Azure Data Lake • Big Data
