Ramesh Betha Email & Phone Number

Bengaluru, Ka, In

Working in the Banking Domain in an ETL project in which we take the data (raw,prepared,transformed layer) which is stored in an S3 bucket and pass it through a step function, which is an orchestrated collection of lambda functions, the data is transformed into the necessary format and loaded into the destination location as Iceberg format and finally into Snowflakes.The state of the data (metadata) is stored in a Dynamo DB table. If the data is loaded correctly, it will be stored under the successful column. Based on the data in the table, cloud watch alarms are triggered, notifying the user about the success or failure of their process. The whole architecture is orchestrated by Airflow and coordinated by step functions.