•Data ingestion from diverse sources, such as Azure Data Lake Storage, Azure SQL Database, external databases, and streaming platforms.•Design and execute data engineering solutions utilizing Azure Databricks, with a focus on scalability, performance, and security enhancement.•Develop and implement ETL processes using Databricks, converting raw data into structured formats tailored for analytics and reporting purposes.•Evaluate or Transform data in Databricks by using different tools of Python, PySpark as well as SQL.•Creation of mount points, delta table from transformed the dataframe and load data into MySQL for creation of report in PowerBI.• Determine root cause and develop solutions for given data Using various tools in Azure Synapse.

•Discovered data acquisitions opportunities and involved in a complete pipeline for the system.•Developed robust and fault-tolerance ETL pipelines using HDFS, Spark, Hive and Oozie in On-Premises platform.•Data from client (various sources, 2 GB to 1 TB in size Daily/Weekly/Monthly etc.) -> To FTP -> Convert to Parquet format on Network Drive -> Store in HDFS -> Convert to structured data using Spark technology -> using Oozie to automate the process and scheduling tasks -> Different dashboards to display data with role-based security.•Get data from various sources and created dynamic reports in PowerBi using various tools.

• Developed and maintained databases, creating tables, writing SQL queries, and defined relationships between tables.• Developed SQL queries using T-SQL and SSIS to ingest data from various sources such as REST API, Database Views, SharePoint lists, FTP etc. to integrate with data warehouse.• Worked on database schema changes, bottlenecks analysis, server/database performance improvements, database maintenance and database recovery.• Maintained data dictionaries, data process flows and documentation of databases.

• Maintained data from client from FTP and convert into the parquet format to store in HDFS and make a structured data using Spark and for Scheduling using Triggers. • Designed and implementation of scalable data processing pipelines using Scoop, Spark and Hbase, resulting in an increase in data processing efficiency.• Scheduled jobs in Oozie to copy and transform the data on regular basis to reduce time and get only modified data by using Spark Queries.• Developed and managed database objects such as tables, stored procedures, functions and views