In this project, I developed a configurable AWS Glue job responsible for reading data from the S3 refined bucket and subsequently creating tables in the Glue Data Catalog. Utilizing a crawler, we automated the process of table creation in the data catalog. Additionally, I crafted a stored procedure to facilitate the migration of data from the persistent schema (used for Glue catalog tables) to the data lake within Redshift. For connectivity with Redshift, I utilized pg8000. Alongside creating Glue jobs, I also played a role in configuring and managing Apache Airflow to schedule and orchestrate these tasks.In another project, I managed the migration of data from Oracle to the Aurora database. I translated the logic by reviewing all Informatica mappings and then implemented it using Python and PySpark within AWS Glue. Apache Airflow was employed to coordinate these Glue jobs. Ultimately, we replicated the data with the same logic from on-premises systems into the Aurora database.I also worked on a GDPR project where we processed requests to remove or update data as per GDPR compliance. Upon receiving a request in an Excel file, I created a pipeline that would trigger and automatically delete or update the relevant data from both the S3 bucket and Redshift.

During my tenure at Deloitte, I had the opportunity to work extensively with AWS services, including Glue, Lambda, and S3 buckets. My primary responsibility involved leveraging PySpark to efficiently transfer data from S3 buckets to DynamoDB tables. Additionally, I developed Python scripts to thoroughly test the codebase, employing tools such as MagicMock.I took a proactive role in optimizing various code segments utilized for data loading into both Redshift and DynamoDB by implementing PySpark. My contributions extended to resolving data type issues across all tables within Athena, making sure the data was correctly structured.I also made effective use of AWS Glue features, such as Job Bookmark, which proved invaluable for facilitating incremental data loading. My responsibilities encompassed processing both historical and incremental data sets, ensuring their seamless integration into the respective tables.

During my tenure at Cognizant, I had the opportunity to work extensively with various AWS services, including AWS Glue, Lambda, Step Functions, S3, DynamoDB, and EventBridge. In a solo capacity, I designed and implemented a comprehensive data pipeline that seamlessly handled data from its initial landing point to the raw storage bucket, and subsequently transformed it into a refined format, making it ready for analytical purposes.I had the privilege of creating both event-driven and scheduled data pipelines, showcasing my versatility in orchestrating data workflows. My data processing activities primarily leveraged Python and PySpark, ensuring efficient data transformation from the raw stage to the refined storage bucket.In addition to data processing, I managed the creation of data catalog tables, both with and without the use of a crawler, streamlining data discovery and accessibility. Notably, I also adeptly handled exceptions, particularly addressing throttling exceptions related to AWS Glue, using Step Functions to implement effective error-handling mechanisms.