Data Engineer Iii
CurrentIn a leading pharmaceutical company, I worked as a Data Engineer within the Artificial Intelligence and Analytics (AIA) department of the development branch. The AIA department is tasked with designing and implementing AI solutions, initially using secondary data, and later integrating these models into clinical trials once proven. Our goal is to create production-level, modern, cloud-based pipelines that are validated and GxP compliant, often pioneering first-in-class solutions with the collaboration of the FDA and international regulators.Key Responsibilities:• Data Management: Managed both structured (SDTM) and unstructured data (doctor's notes), performing transformations, filtering, imputations, and aggregations to prepare data for model ingestion.• Refactoring and Modernizing Efforts of Structured Data Pipeline: Transitioned the data processing pipeline to a Glue/Lambda/Step Functions job combo with a CI/CD pipeline, storing code and Terraform Infrastructure as Code in Azure DevOps, then to a Databricks Integration. ETL scripting involved Python, PySpark, and SQL languages. • Creation of Unstructured Data Pipeline: Key collaborator in creation of OCR/Textract pipeline, initially in AWS and later transitioned to Azure. Specific responsibilities include leading development of Terraform and CI/CD pipelines, pre-processing code design, and collaboration on processing/post-processing of unstructured data.• Validation and Quality Assurance: Created numerous validation scripts and functions to ensure data integrity and accuracy, prior to, during, and after processing. contributing to the successful completion of product POC.• Digitization Project: Developed a functional pipeline using AWS Textract for OCR, later transitioning to Azure. Created a comprehensive instruction manual on building a Terraform pipeline.• Documentation