Yifan G. Email & Phone Number

San Mateo, California, United States

San Francisco Bay Area

* Embedded in the Feature Engineering team of a world-class fashion retail company: - Built multiple ETL pipelines with Airflow and Spark, for inventory and allocation data in US and EU regions. - Optimized various aspects of Feature Catalog, delivering reliable data to the DS team with ~100% SLA.

San Francisco Bay Area

* Embedded in the Data Science team of a world-class BioPharma company, as the only Data Engineer on the team: - Built high-quality marketing campaigns datasets using various data sources from vendors within a tight timeline. - Designed and implemented config-driven ETL pipelines using AWS SageMaker and PySpark on EMR.

San Francisco Bay Area

* Led the effort of building enterprise data catalog and data governance platforms and toolings* Led the effort of implementing data quality checks using Great Expectations and other homebrew toolings* Worked on building a self-servable config-driven data pipeline orchestrator "Aqueduct", which is capable of translating DSL configs to Kafka streaming pipelines, Spark jobs, for stream / batch transferring and transforming data from upstream systems to data warehouse and data lake* Worked on deploying and maintaining self-hosted Airflow, facilitated team on developing various DAGs for orchestrating reporting and ETL jobs with best practices

San Francisco Bay Area

* Implemented multiple data streaming pipelines using Kafka Connect (Debezium, S3/JDBC sinks, SMT) and custom built asynchronous Kafka streaming processors* Designed and implemented microservices for data ingestion from external bank partners' API and SFTP* Worked on optimizing Spark jobs using PySpark on EMR* Worked on optimizing and automating Nightly-ETL for batch loading data into Redshift data warehouse* Worked on various misc. tasks including migrating microservices to Kubernetes (AWS ROSA), migrating Kafka pipelines from Aiven to Confluent, implementing IaC with AWS CDK, optimizing Jenkins CI/CD, etc

Beijing, China

Researched and implemented a source code vulnerability analysis system* Implemented crawlers for OSS source code repositories and CVE vulnerabilities, used Postgres for artifact store* Implemented tokenization and parser based feature extractor, used Redis for indexing and searching