Dinesh Kumar Email & Phone Number

I am a AWS Data Engineer with nearly 3 years of working experience in building serverless ETL pipelines using AWS Services, Python and PySpark for cloud migration projects. My day to day work involves Developing, Testing and Deploying code through CICD Pipeline for Data Processing using a Batch AWS StepFunction Pipeline which includes Glue, Dynamo DB, Lambda, ECS and other commonly used services in Amazon Web Services. Key Contributions: • Developed AWS Glue jobs for data cleaning, transformation and migration across Landing, Raw, Enriched and Model layer on AWS Cloud. Integrated with CloudWatch service for continuous logging and monitoring purpose and with AWS DynamoDB service to read metadata and to record validation result on audit tables.• Built Step functions for orchestration of data pipeline. Used AWS EventBridge to create rules and automated the running of Daily loads as per schedule.• Developed automated Email notification system for Data pipeline through AWS Lambda with Snowflake integration using spark snowflake connector resulting in 40% turnaround time improvement.• Developed and Optimized AWS Glue job which loads processed data from AWS S3 to MySQL database using Python Multithreading approach and reduced job execution time by 60%.• Leveraged GitHub CI/CD to publish the codes for Lambda, Glue and ECS to higher environments. Used Terraform along with Git for deploying infrastructure of pipeline and DynamoDB configurations.• Developed Python and PySpark modules for performing file health checks and quality control on incoming data.