Divya T Email & Phone Number

Divya T is a Bigdata/Cloud Data Engineer at Medica.

• 

  Cloud Data Engineer

  Medica

  Feb 2023 - Present - Madison, Wi

  • Worked on the AWS Migration project and migrated Provider datasets to AWS S3 datalake.• Evaluated business requirements and prepared detailed technical specifications document that follow project guidelines required to develop data pipelines.• Performed analysis and worked collaboratively with business and consumption teams to migrate legacy datasets into AWS using PySpark. • Worked on GLUE pySpark Jobs to move data from trusted to curated layers • AWS Lambda, EventBridge and Step functions used for job poling and scheduling in AWS Datalake to kickstart the Glue CatLog external table Partitions to refresh daily partitioned tables.• Used Terraform for AWS Infrastructure provisioning like KMS keys, IAM roles, LAMBDA functions, Step Functions, Glue jobs and SNS notifications.• Experienced in performance tuning of Spark Applications and Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Glue job optimizations.• Created Control-M jobs to schedule data pipelines and setup dependencies to downstream Data Pipelines.• Used Bit Bucket as the code repository and frequently used Git commands to clone, push, pull code to name a few from the Bitbucket repository.• Create and update Jenkins jobs to develop pipelines to deploy the application in different environments like develop, QA and Production.• Used Jira as an agile tool to keep track of the stories that were worked on using the Agile methodology• Documented the new process and design documents in the Confluence pages.