Lead Data Engineer
Current• Playing a lead role in the development of Confidential Data Lake and in building Confidential Data Cube on Microsoft Azure HDINSIGHT cluster.• Architect, Design, Develop, and Improve databases and ETL processes in scope of Application development.• Provide direction and guidance to other Big Data developers to solve problems, improve efficiency and process, and employ new technology.• Manager will be expected to perform hands on coding, performance optimization.• Work with business and technology partners to provide reporting capabilities for all our internal customers.• Responsible for managing data coming from different data sources and experience in working with Restful APIs.• Responsible for transporting and processing real-time stream data sourced from APIs for product management using Storm.• Developed scripts for extracting and processing data sourced from AZURE BLOB storage and analyzed using Hive data warehouse and Linux shell scripting.• Used Maven and J2EE for build and deploy the STORM programs. Developed STORM programming code Eclipse IDE tools.• performance tuning, monitoring the Hadoop cluster by gathering and analyzing the existing infrastructure using Cloudera manager and AMBARI UI.• Transporting, and processing real-time stream data using storm. working with NO-SQL databases such as Mongodb, Cosmosdb.• Writing Map Reduce jobs in Java.• Writing HQL queries in Hive Data warehouse.• Wrote Java scripts to copy or move data from local file system to HDFS Blob storage.• Experience in rendering and delivering reports in desired formats by using reporting tools such as PowerBI.• Involved in story-driven agile development methodology and actively participated in daily scrum meetings.