Platform Software Engineer
Current- Data Lineage product
- Developed a Spark lineage plugin to generate end-to-end data lineage for all Spark applications in Apache Atlas and Open Metadata.
- It replaces the discontinued Spark lineage feature from spark 2.4.Enhancement And Ownership Of Data Catalog
- Established automated deployment procedures for Apache Atlas, achieving a 90% automated CI/CD pipeline.
- Enhanced Apache Atlas for 99.999% availability and extended the back-end Java code to import hive metadata into atlas through a bridge.Introduced Gen 3 Data Catalog
- Conducted a successful POC and integrated Open Metadata into the data platform for data modernization.