Senior Data Engineer
CurrentClosely engaged with the consumer to gather functional requirements and convert them into technical requirements for implementation.● Leading and managing a team of Five, strategically assigning tasks to optimize their learning curve and productivity.● Implemented tailored task allocations based on individual strengths and development needs, ensuring each team member contributed effectively to overall goals.● Conducted regular check-ins and guided to foster a productive learning environment● Ingesting streaming and batch data to HDFS in Avro and Hbase file formats from multiple sources and loaded intoparquet files as per the consumer requirements using Spark Scala SQL and loaded to JP Morgan’s internalcloud/Enterprise Data platform/Cassandra.● Provided mentorship to the new joiners in the team, making them understand the end-to-end flow of the projectand aligning them with the overall company vision.● Working in the agile methodology with 15-day sprints.Migration from on-prem to AWS :● Implemented CI/CD process from code commit to AWS S3 bucket which includes, the AIM process and created alambda function to unzip the artifacts from AIM buckets to the project-level S3 buckets.● Created a script for generating bulk extracts from AWS S3 buckets to the internal cloud for consumers dealing withsecure data policy, using boto3 and Mercury APIs.● Created a Step function that sends an SQS message with upload details when the file is uploaded to the S3 bucket.● Onboarded 250 datasets to AWS, which included the creation of a raw zone process and the creation of tables inAthena using Glue and Terraform.● Created Airflow dags to schedule the AWS end-to-end pipeline.
