Cloud Data Engineer
Current• Analyze, design and build modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes. • Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards. • Outguessed the data from HDFS to Azure SQL data warehouse by building ETL pipelines using S worked on various methods including data fusion, machine learning and improved the accuracy of distinguished the right rules from potential rules. • Extensive Knowledge and hands-on experience implementing PaaS, IaaS, SaaS style delivery models inside the Enterprise (Data center) and in Public Clouds using like Azure, Google Cloud, and Kubernetes etc. • Extensively worked on making REST API (application program interface) calls to get the data as JSON response and parse it. • Experience in analyzing and writing SQL queries to extract the data in Json format through Rest API calls with API Keys, ADMIN Keys and Query Keys and load the data into Data warehouse. • Designed and developed Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda, Dynamo DB.• Creating lambda functions with Boto3 to deregister unused AMIs in all application regions to reduce the cost for EC2 resources.• Develop Databricks Python notebooks to Join, filter, pre-aggregate, and process the files stored in Azure data lake storage.