Principal Informatics Data Engineer
CurrentLead engineering effort to overhaul internal small molecule data platform, streamlining data capture, sharing, standardization, and tracking of chemical compounds. • Architected reliable and scalable infrastructure by leveraging a range of AWS services, including Athena, Fargate, Batch, Lambda, S3, SNS, SQS, and Cloud Formation. • Utilized Airflow to manage ETL workflows to efficiently consolidate data across various internal and external sources, and to execute scientific pipelines. • Designed and deployed a PostgreSQL database to capture metadata and relational information from user uploads as well as ontological data related to internal chemistry databases. • Established a data lake and empowered queries across user-uploaded datasets via Athena and Glue. • Deployed a GraphQL API, ensuring seamless data retrieval for scientists and engineers.• Designed a Python library to simplify data exchange with the core chemoinformatics data platform, empowering scientists to share data and machine learning results.• Constructed a lightweight frontend using Typescript/Vue.js, facilitating search and retrieval of datasets.