Dinesh A Email and Phone Number

● Developed a framework using PySpark to ingest, validate, and efficiently process data, drawing on hands-on experience as a Data Engineer.● Configured CI/CD pipelines using GitLab CI/CD for automated build, test, and deployment of Java applications, ensuring rapid and reliable software delivery.● Designed and implemented a scalable ETL pipeline using PySpark on Azure Databricks, leveraging expertise in data modeling solutions and tools.● Migrate Matillion pipelines and Looker… Show more ● Developed a framework using PySpark to ingest, validate, and efficiently process data, drawing on hands-on experience as a Data Engineer.● Configured CI/CD pipelines using GitLab CI/CD for automated build, test, and deployment of Java applications, ensuring rapid and reliable software delivery.● Designed and implemented a scalable ETL pipeline using PySpark on Azure Databricks, leveraging expertise in data modeling solutions and tools.● Migrate Matillion pipelines and Looker reports from Amazon Redshift to Snowflake data warehouse.● Utilized Spark Scala scripts and User-Defined Functions (UDFs) to read data efficiently from Azure Blob storage in Azure Databricks, demonstrating in-depth knowledge of data modeling techniques.● Assisted clients in selecting, designing, and implementing new EHR and Practice Managment systems.● Designed and implemented Java-based ETL pipelines using technologies like Apache Kafka for real-time data streaming and Apache Flume for log aggregation.● Optimized Neo4j database performance by fine-tuning configuration settings, indexing strategies, and query execution plans, resulting in improved response times and scalability.● Implemented end-to-end data extraction, transformation, and loading (ETL) processes using Azure Data Factory (ADF) and Azure HDInsight, showcasing strong problem-solving skills in a fast-paced environment.● Created Linked services to land data from different sources into Azure Data Factory, applying experience with version control systems (e.g., Git) and agile software development methodologies. ● Developed custom Azure Synapse solutions using serverless SQL pools, enhancing scalability and reducing infrastructure costs. Show less

• Requirements gathering, design, developed, unit test and implement technical software solutions.• Developed framework using Scala and Spark to ingest data for analytics team.• Utilized Azure Synapse's integrated Apache Spark capabilities to perform complex data transformations and analysis on large datasets.• Create Scala/Spark jobs for data transformation and aggregation.• Produce unit tests for Spark transformations and helper methods.• Implemented a CI/CD pipeline with… Show more • Requirements gathering, design, developed, unit test and implement technical software solutions.• Developed framework using Scala and Spark to ingest data for analytics team.• Utilized Azure Synapse's integrated Apache Spark capabilities to perform complex data transformations and analysis on large datasets.• Create Scala/Spark jobs for data transformation and aggregation.• Produce unit tests for Spark transformations and helper methods.• Implemented a CI/CD pipeline with Docker, Jenkins and GitHub by virtualizing the servers using Docker for the Dev and Test environments by achieving needs through configuring automation using Containerization.• Monitored Data Engines to define data requirements and data Accusations from both relational and non-relational databases including Cassandra, HDFS.• Utilized Power Query in Power BI to Pivot and Un-pivot the data model for data cleansing and data massaging.• Implemented data governance policies and procedures within Azure Synapse, ensuring data integrity, compliance, and privacy standards are met.• Integrated Apache Airflow with AWS to monitor multi-stage ML workflows with the tasks running on Amazon SageMaker.• Experience in Kubernetes to deploy scale, load balance, and manage Docker containers with multiple names spaced versions using Helm charts.• Leveraged Snowflake's capabilities for semi-structured data handling to process JSON and Parquet files stored in S3 buckets. Show less