COMP 630: DATABASES

• Decoupled daily reports of 320 metrics into calculation and presentation layers for Tencent App Store using Spark SQL, improving performance by 30% and enhancing ETL independence. Developed Python scripts to validate data consistency and logical integrity against 36 regulations.• Developed Flink real-time tables for the advertising business, outputting to Kafka for downstream machine learning use, improving data timeliness from daily to minute-level.• Designed and implemented data warehouse for the eCPM(effective cost per mille) computation; built two Hive UDF in Java, referencing the Hive source code.

• Developed Spark process in Java, involving JNI and Protobuf, which completed loop-closure operations in SLAM used to build point cloud maps; modified and deployed Spark official image into containers of Kubernetes pods. Broke the bottleneck of the thread pool used in the original process; reached 6x better performance.• Investigated the Data Lakehouse architecture. Conducted comparative research on Parquet with ORC, and Iceberg with Hudi and Delta Lake; implemented the architecture with Iceberg, PostgreSQL, and MinIO; tested its performance with different scales of point cloud datasets.• Built and organized Docker images to comprise Argo pipelines based on the workflows of 3 main modules in the HD Map team; transformed 80% manual, local processes into automated, Kubernetes-based ones.• Located and eliminated errors in pipelines with the DevOps team. Reported the evaluation and solutions routinely.