As an Azure Data Engineer, my role will encompass the design, development, and management of data integration and messaging solutions, modern data streaming tools. Responsible for seamlessly connecting and processing data across various systems and applications to enable efficient and real-time communication.Developed Azure pipelines using ADFv2 for diverse data sources, employing activities like Move&Transform, Copy, filter, and Databricks.Established Linked services, monitored ADF pipelines… Show more As an Azure Data Engineer, my role will encompass the design, development, and management of data integration and messaging solutions, modern data streaming tools. Responsible for seamlessly connecting and processing data across various systems and applications to enable efficient and real-time communication.Developed Azure pipelines using ADFv2 for diverse data sources, employing activities like Move&Transform, Copy, filter, and Databricks.Established Linked services, monitored ADF pipelines, and managed incidents based on severity.Implemented medium to large-scale BI solutions on Azure using various services, including AzureDataLake, DataFactory, DataLakeAnalytics, StreamAnalytics, AzureSQLDW, and HDInsight/Databricks.Executed ETL processes using AzureDataFactory, T-SQL, SparkSQL, and U-SQLAzureDataLakeAnalytics for data extraction, transformation, and loading.Enhanced conventional data warehouses based on STAR schema, datamodel updates, and facilitated Tableau data analytics and reporting.Conducted data ingestion and processing in AzureDatabricks for AzureServices like AzureDataLake, AzureStorage, AzureSQL, and AzureDW.Implemented software enhancements to transition legacy systems to Spark and Hadoop ecosystems on AzureCloud.Designed and developed batch processing and real-time solutions with ADF, Databricks clusters, and Stream Analytics.Leveraged Pyspark data frames in AzureDatabricks for data reading and SparkSQL context for transformation.Utilized Python for data manipulation, pipeline creation, and ETL prototyping in Snowflake, along with task scheduling in Airflow.Created ADF pipelines using LinkedServices/Datasets/Pipeline for data extraction, transformation, and loading from various sources.Developed data pipelines with Azure services like Data Factory to load data from Legacy SQL Server to Azure Database.Architected and implemented ETL and data movement solutions using AzureDataFactory, SSIS, and created/run SSIS Packages in ADFV2 Azure-SSIS IR. Show less

As an Data Engineer involves loading data into Spark data frames using Python from specific tables in a Cassandra database and performing aggregation and transformation operations based on UI requests. Responsibilities include building and architecting data pipelines using Azure stack components, developing Spark applications for data extraction and transformation, creating Kafka producers and consumers, managing Hadoop clusters, analyzing large datasets with Cloudera, and developing… Show more As an Data Engineer involves loading data into Spark data frames using Python from specific tables in a Cassandra database and performing aggregation and transformation operations based on UI requests. Responsibilities include building and architecting data pipelines using Azure stack components, developing Spark applications for data extraction and transformation, creating Kafka producers and consumers, managing Hadoop clusters, analyzing large datasets with Cloudera, and developing Spark/PySpark/Python notebooks for file organization in ADLS.The role encompasses CRUD operations on CQL in Cassandra, workflow development in Oozie for automation, PigScript development for change data capture, and creating POWERBI visualizations and dashboards from various sources. Additionally, responsibilities involve creating HBase tables for diverse data sets, using external loaders for data loading into Teradata, Oracle, and Database, building a Scala and Spark-based configurable framework for connecting common data sources, deploying applications with Jenkins, designing StarSchema and SnowflakeSchema for DataWarehouse, working with Tableau for customized reports, and writing Python and SQL/PL-SQL scripts for data extraction and testing purposes. The role requires a comprehensive skill set in data processing, analytics, ETL, and visualization across various technologies and platforms. Show less