Data Engineering Specialist
Current1. Healthfirst - I worked on Shell scripts and Python code development which we execute in EMR box for data processing. I have good experience on SQL queries, we used DBEaver for connecting to Postgre and SQL server databases for different environments. I worked on Lambda functions development in Python, Cloud watch alarms, S3, EMR, etc. AWS services in this project. Client is following Agile process, so I was part of all the Agile meetings like daily standup calls, PI planning meetings, etc.2. Financial Transformation - Migrated multiple Cloudera projects to Databricks. The existing projects were written in scala, shell and python language and deployed in Cloudera environment. We migrated all the code to Databricks and deployed or scheduled from Stonebranch. Also written Databricks Notebooks in Scala, Python and SQL. Performed unit testing on the comparison of data in both the environments, Cloudera & Databricks.