Developed comprehensive data engineering projects, specializing in the design and implementation of end-to-end solutions across various Azure services. Managed data ingestion into Azure Data Lake, Azure Storage , Azure SQL, and Azure Data Warehouse, followed by efficient processing in Azure Databricks. Integrated Azure Data Factory and Azure Synapse Analytics with external services , enhancing functionality by incorporating machine learning models and connecting with third-party data providers .Deployed data processing algorithms and workflows within Azure HDInsight to improve overall performance and reduce processing time. Contributed to the development of informatica mappings , preparing design documents , technical design documents, and unit acceptance testing documents, Designed scalable and reliable data integration solutions using informatica power center to accommodate growing data volumes and meet business requirements.Developed and maintained data warehousing solutions using python, ensuring structured storage and easy access for analytical purposes Implemented CI/CD pipelines with jenkins to automate testing and deployment of python-based data engineering solutions, enhancing reliability and speed .Designed and implemented fault-tolerant Spark and Kafka architectures to guarantee data integrity and system reliability. Conducted regular code reviews to ensure adherence to best practices and efficiency of Spark code. Developed reusable spark scripts and functions for data processing in different pipelines and Kafka stream processors for data transformation , enrichment and cleasning .Dev

Implemented diverse data engineering projects, hearing the power of AWS services to transform, move, and manage large datasets. Utilized AWS EMR for seamless data transformation and movement, creating dynamic visualizations with Amazon QuickSight, and implementing serverless functions using AWS Lambda for event-driven data processing. Maintained comprehensive documentation for AWS Glue jobs and AWS Lambda functions.Designed and implemented complex data workflows using Apache Airflow, orchestrating end-to-end data processing tasks to ensure timely and accurate execution. Implemented data integration workflows using Informatica PowerCenter to move, transform, and load data from various sources into a data warehouse. Ensured data quality and integrity through the implementation of data validation and cleansing routines within the informatica environment.Maintained detailed documentation for snowflake data processes, ensuring clarity on data flows, transformations, and integrations for future reference and knowledge transfer. Implemented rigorous controls and encryption protocols in Snowflake to safeguard sensitive data and comply with data privacy regulations such as GDPR and HIPAA.Integrated Kafka with log management systems for centralized logging and auditing of data streams. Implemented advanced procedures like text analytics and processing using in-memory computing capabilities like Apache Spark written in Scala. Conducted Kafka security audits and implemented encryption mechanisms for data in transitI

Drove comprehensive data engineering projects , ensuring the security of sensitive data in Azure Data Lake and Azure Databricks while adhering to regulatory standards and data governance policies. Developed and implemented disaster recovery plans for Azure Data Factory and Azure SQL, incorporating regular backups to ensure data integrity and availability during unexpected events.Utilized advanced data partitioning and indezing techniques in Azure Synapse Analytics to enhance query performance and reduce latency. Employed metadata-driven approaches in Informatica to automate data integration processes, faciliating adaptability to changing data structures and minimizing development time.Maintained clear and concise documentation for Python scripts and codebases, fostering collaboration and ensuring code transparency. Developed Python scripts and applications with scalability in mind, enabling efficient handling of growing data volumes. Designed and deployed data table structures, reports, and queries in SQL Server, receiving and importing data from various formats.Utilized Hadoop custom scripting and automatic tools to streamline repetitive tasks, enhancing operational efficiency and reducing the likelihood of errors. I worked extensively with the Spark ecosystem using Spark SQL and Scala queries on different formats like text and CSV files .Implemented security measures at Power Bi Service by implementing authentication and authorization methods, ensuring the secure handling of sensitive information. Passionate about delivering robust and secure data solutions that align with business needs.Successfully migrated data pipelines from Azure to the Snowflake multi-cluster data warehouse, handling semi-structured data connected to analytical engines for reporting. I designed and developed Tableau visualizations, including dashboards with calculations, parameters, calculated fields , groups , sets and hierarchies .

Led and regulated disaster recovery plans for AWS S3 and AWS Lambda, ensuring data resilience in unforeseen incidents. Designed and developed ETL processes in AWS Glue to migrate data from external sources like AWS S3, parquet, and text files into AWS Redshift.Conducted periodic reviews of AWS SNS message formats and structures for efficiency and relevance. Designed and implemented custom configurations for specialized processing tasks within AWS EMR. Developed custom functions and scripts in R and Python to address specific data analysis challenges, enhancing code reusability and overall project efficacy.Transformed and cleaned raw data into usable formats using Python libraries such as Pandas and NumPy, ensuring data accuracy and quality. Tunes SQL queries to bring down runtime, worked on indexes, and devised SQL scripts to check and validate dataflow in various applications. Implemented cost optimization strategies, selecting appropriate NoSQL database configurations and services to balance performance requirements with cost-effectiveness. Conducted performance optimization tasks, including query optimization and index management, to enhance PostgreSQL database efficiency and response times.Performed data validation and quality assurance checks within Tableau to ensure the accuracy and reliability of visualizations. Utilized version control systems to manage Tableau workbooks and dashboards, enabling efficient collaboration, tracking changes, and ensuring version consistency. I worked on Power BI reports using multiple types of visualizations, including line charts, tables, matrix, KPIs, scatter plots , and box plots . Passionate about delivering data solutions that align with business needs and drive meaningful insights