• Design data models for data warehouse with standards for consistency and integrity• Architect and develop data warehousing processes• Develop and maintain data dictionaries/metadata management processes for source data schema• Streamline data ingestion processes• Develop scalable, re-usable and metadata-driven solutions for data landing and transformation• Develop repeatable processes that accelerate delivery of user request fulfilment• Monitor data pipelines from sources to a big data environment• Lead the Design, development, and implementing of Lakehouse & data warehouses architecture in a cloud based data environment that integrates with an overall data architecture.• Work closely with business stakeholders to interpret their data needs and translate them into technical requirements.• Have a good understanding of data governance and develop & implement data governance policies and procedures.• Ensure data quality and integrity by implementing data testing and validation processes.• Deploying and debugging cloud data initiatives as needed in accordance with best practices throughout the development lifecycle.• Managing cloud data environments in accordance with company security guidelines.• Lead the design and build of reusable, repeatable solutions and components for future use.

• Working on Azure and GCP environment• Developing data pipelines for data ingestion, data management on big data clusters in spark and python using databricks• Work with data scientists on transform data into a format that can be easily used for report generation.• Python development using PySpark and databricks• Analyzed the SQL scripts and designed the solution to Developed a custom ETL pipeline by using Python• Write complex SQL queries on tables with hundreds of millions of records and ensure data integrity is maintained throughout the ETL lifecycle.• Setting up Tidal job to schedule the data pipeline jobs.• Analyze data quality and take actions to mitigate blocking quality issues

• Created Source to Target Mappings (STM) for the required tables by understanding the business requirements for the reports• Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.