Aman Agrawal Email & Phone Number

Noida, Up, In

India

Building data pipelines, data migration, streaming pipelinesthrough AWS Glue & EMR on top of Apache Spark with AWSRedshift as target. Develop DBT scripts for business rule transformations running over AWS Fargate. Using AWS lambda for handling real-time events.Writing Docker script for containerization of python code forrunning dbt-core in on AWS using ECS Fargate. Build images using Sagemaker Notebooks and storing in AWS ECR. Deploy images from AWS ECR over Fargate.Implementing Data Quality using Deequ framework that runs over glue job and generate audit results into s3 that can be queriedusing athena. Building alert and monitoring framework for data pipelines usingairflow.

At my current organization I am tasked with the following responsibilities:1. Client Infrastructure Modernization to AWS.2. Provisioning all AWS resources using Cloudformation templates using Bitbucket and Bamboo data pipeline.3. Converting Cobol DB2 batch scripts business logics into data transformation rules in Pyspark running on AWS Glue ETL.4. Created one time data migration and validation scripts in Pyspark to migrate DB2 legacy tables to AWS Aurora Cluster.5. Orchestration of Glue Jobs using AWS Glue workflow, Triggers, AWS Step functions, Control-M etc.6. Configure FTP on EC2 instances.7. Load real-time streams into Aurora using AWS Kinesis and Kafka.8. Develop and maintain data pipelines to extract data from a variety of sources(DB2, Kafka etc.) and populate target database.9. Analyze Data using SQL.10. Setup splunk logging and create dashboards.11. Performing code and data analysis in local using Jupyter Notebook.12. Creating serverless functions for several utilities using AWS Lambda.13. Implementing data quality checks and perform unit test of pyspark scripts using python libraries like boto3, moto, mock, awsglue utils etc. running over a docker container in python venv and generating sonar coverage report.

1. Create a dynamic shopping cart template with HTML/CSS/TypeScript/Angular.2. Create API's for shopping cart application with Spring Boot MVC and enter table records in postgresql database running on docker container.3. Created pyspark script to fetch real-time stock index data from kafka server/broker and create a data pipeline to ETL into postgresql database.