Darren W. Email and Phone Number

I'm working on data engineering in the data platform team.

Backend data and API development with distributed, event-driven SOA on AWS systems. Primarily python development of libraries, messaging contracts, and Django-PostgreSQL data APIs. Services provided solutions to optimize trucking fleet networks and dispatching, with batch and near real-time services. Data systems included asynchronous workflows to ingest data and run optimization engines to return insights to customers.The distributed systems provided data workflows and AI-engine solutions to optimize trucking logistics. All systems were orchestrated using AWS-SQS for event-driven data flows and analytics, handled by AWS-ECS container solutions. The AI-engine results were output in AWS-S3 reports and transformed for customer presentations using AWS-RDS (PostgrSQL) and Django REST APIs (with React front-end UIs).I focused on application features that included back-end orchestration of AI engine workflows. I worked on new features (under feature flags), and improved the reliability and robustness of the applications. I refactored feature flag management with clean code for clarity and consistency. I added unit tests on significant behavior (PyTest, Faker). I collaborated on an integration test suite in the CI/CD. I added pre-commit hooks for code quality.I contributed to architecture docs and discussions to optimize data analytics and workflows. This included switching from CSV files on AWS-S3 (with slow, unreliable pandas transforms) to a more integrated analytics solution using DuckDB with parquet files. This evolved to include workflows with DBT and a Trino data lake solution. I recommended and introduced better data schema management and validations using Pydantic and Pandera.I assisted optimizing AWS services. I contributed to infrastructure as code with terraform on AWS. I recommended and implemented AWS-S3 Intelligent Tiering and ECS auto-scaling. I assisted with SOC2 compliance for infrastructure monitoring (with Vanta).

Lead architect and developer for AWS Serverless solutions for data APIs on several projects. Production solutions for a data API built on petabytes of GeoTIFF data on AWS s3, for global climate forecasts. Leveraged AWS Lambda and API Gateway for low-cost, scalable and fault tolerant architectures to build map-reduce solutions with 1000s of concurrent serverless workers; built with python and the FastAPI framework with data schemas and models in Pydantic with Postgres (with PostGIS on AWS-RDS). Also included event-driven architectures for data workflows and reports with AWS Lambda, AWS Batch, and Airflow. We also explored AWS Athena and Snowflake for query solutions on petabyte datasets (using Parquet on S3).Also worked with Geographic Information Systems (GIS) for raster and vector tiling. Scaled ETLs on AWS Batch (e.g. GeoTIFFs to MBtiles; rasterio, rio-mbtiles, GDAL, tippecanoe). Familiar with workflow engines such as Airflow, AWS (step functions), and Prefect (optimized tasks for cost effective, scalable, fault tolerant, decoupled, idempotent workflows). Contributed to Apache Airflow and created python asyncio workflows with AWS Batch (aio-aws).Also worked on several projects for near real-time weather systems for flood and fire risks. I lead a small backend engineering team, responsible for data ingest using event-driven workflows on S3/Lambda and Postgres (PostGIS). We built a REST API service for hazardous weather data and forecasts (tested with Postman for automated QA).General adoption of best practices, including extensive testing with pytest and Postman API test suites. Used gitlab for CI/CD systems, with private gitlab runners deployed using Helm on AWS EKS-K8s (dev-ops). Deployments coordinated with infrastructure as code including: Docker, serverless.com framework, AWS CloudFormation, AWS S3, AWS SQS, AWS Lambda, AWS Batch, AWS Fargate, AWS EKS, and Spinnaker.

Software engineer for various projects in the Digital Library Systems and Services team at the Stanford Library. Projects included: archives for the Stanford Digital Repository (SDR), academic publication harvesting and archives API for Stanford Profiles, and Linked-Data for Libraries (LD4L). The software stacks include a variety of microservices based on java and ruby frameworks for APIs and web-apps, backed by structured and unstructured document stores, semantic web systems (ontologies, linked-data, triple stores, etc.), and discovery systems (Solr).

Development of http://bioportal.bioontology.org, a semantic web system in the National Center for Biological Ontologies (http://www.bioontology.org/). The system is a java REST service and the interface is a ruby-on-rails web application. I've developed and maintained a Jenkins system for automated build and deployment, including svn, git and github triggers for various ant, maven, and shell script builds. I've also contributed a maven project with java programs for ontology mappings (matching terms between different ontologies). I also work on feature enhancements and debugging existing java and ruby-on-rails code.

Design and development for animation utilities in UCSF Chimera. The animation development facilitates easy creation of molecular scenes and transitions between keyframe scenes, including a storyboard UI. The project involves daily work in eclipse for development with python (pydev), subversion (subversive), and trac (mylyn).

Development of web interfaces to genetic and microscopy data related to aging in C. elegans. Scientific image processing and visualisation for microscopy (using cmake, c++, and ImageToolkit).

I was a post-doctoral fellow at the UCSF Dynamic Neuroimaging Laboratory (see http://dnl.ucsf.edu/). The work was supervised by Dr. Gregory Simpson, director of the Network for the Study of Brain Systems and Dynamics (http://dnl.ucsf.edu/NBSD/). We extended tools for advanced, non-invasive brain imaging to study human cognitive neuroscience. My research work focused on analysis of brain source activity, which was estimated from high resolution MEG and MRI imaging. We published novel findings on cortical activity involved in visual-spatial attention and working memory processes. My post-doctoral work continued into my role as an assistant specialist in brain imaging.

Investigation of cortical systems involved in visual attention, including models of cortical adaptation.

After completing a BSc(Hons), I worked as a research assistant in the Cognitive Neuroscience Laboratory at the Flinders University of South Australia. This was initially to complete work on publications arising from the honours thesis. The employment continued from 1992-2003, while I worked on a new research project, began a BA degree and later started a postgraduate thesis project. During these 10 years, especially after 1995, the laboratory made a transition from 16 channels of EEG only into a sophisticated brain imaging laboratory, incorporating 128 channel EEG (Neuroscan SYNAMP) and anatomical and functional MRI (Siemens Magnatom 1.5T). Furthermore, these acquisition methods comprised the basis for investigation of multimodal integration and brain source estimation.I also worked part-time as a teaching assistant in the departments of psychology, medicine, and statistics. These positions included classroom teaching, laboratory supervision, and assignment assessments in psychobiology and research methods (research design and statistics).