Data Engineer
CurrentIn my role at Arup, I led a series of data engineering initiatives that significantly enhanced operational efficiency, reduced costs, and improved data accessibility and processing capabilities. Highlights of my contributions include:- Data Architecture Overhaul: I redesigned the data architecture to integrate file-based and NoSQL storage for managing weather data from over 700 stations. This strategic overhaul reduced monthly costs from $820 to $60 without compromising on query performance.- Advanced Data Processing: Leveraging PySpark on Databricks, I developed algorithms to clean, reformat, and repartition over 800 GB of weather data. This innovation enabled quick access to 20 years of minute-level data in under a minute, streamlining data analysis processes.- AI/ML Collaboration: In partnership with the National University of Singapore, I co-developed a Generative Adversarial Network (GAN) model using PyTorch. This model expedited the parameter space search by 70% and saved over 6,000 computation hours, highlighting our commitment to cutting-edge machine learning research and development.- Algorithm Optimization: I created and implemented a fan-in fan-out algorithm for parallel computation, enhancing the real-time processing of complex geometries and demonstrating the practical application of advanced computational techniques in urban planning.- Workflow Innovation: By identifying inefficiencies in the existing 3D modeling workflow, I directed the rollout of a new workflow that cut task completion times from one day to 10 minutes.- Operational Excellence: Collaborating closely with engineers, I understood their challenges and created algorithms and workflows that eliminated human error, saved significant labor hours, and streamlined their tasks.