Haolun Yu Email & Phone Number

Warsaw Metropolitan Area

Develop and maintain the order system for SHEIN global

Hangzhou, Zhejiang, China

ELT/ETL Pipeline Migration to Azure Databricks and SnowflakeSuccessfully migrated the ELT/ETL pipeline from a private cloud (Trino) to Microsoft Azure Databricks and Snowflake, enhancing the ingestion of complex AVRO data structures and reducing costs. Key achievements include:Leveraged Databricks and Snowflake's advanced capabilities to streamline the ingestion of intricate AVRO data structures, ensuring efficient data processing and analysis.Developed an AVRO message… Show more ELT/ETL Pipeline Migration to Azure Databricks and SnowflakeSuccessfully migrated the ELT/ETL pipeline from a private cloud (Trino) to Microsoft Azure Databricks and Snowflake, enhancing the ingestion of complex AVRO data structures and reducing costs. Key achievements include:Leveraged Databricks and Snowflake's advanced capabilities to streamline the ingestion of intricate AVRO data structures, ensuring efficient data processing and analysis.Developed an AVRO message send/receive function integrated with the AVRO Registry Center, enabling compressed data handling and significantly reducing bandwidth usage.This migration optimized the ELT/ETL pipeline by leveraging cloud-based technologies to handle complex data structures, reduce costs, and improve overall system performance and scalability.---------------------------------------------------------------------------------------------------------Optimized Hive storage with integrity check and customized statistics collectionSignificantly improved maintainability and reduced the burden on the Trino clusterDeveloped Quartz jobs to merge Hive files and verify the data integrityDeveloped RESTful APIs to dynamically configure statistical data collection, decoupling data writing from statistical processingThis optimization improved the system's stability running in the private cloud and accelerated query speed for end-users by reducing the burden on the big data platform. Show less

Hangzhou, Zhejiang, China

Microservice - IP Location QueryDeveloped a high-performance IP query service for Registry function to block requests from specific countries, handling over 300,000 new IP addresses monthly with a single instance.Designed and implemented a two-layered caching system to efficiently locate IP.Implemented a shared and updatable in-memory database (MMDB) across multiple instances for seamless IP location lookup.Infrastructure Development - Gateway, Registry Center… Show more Microservice - IP Location QueryDeveloped a high-performance IP query service for Registry function to block requests from specific countries, handling over 300,000 new IP addresses monthly with a single instance.Designed and implemented a two-layered caching system to efficiently locate IP.Implemented a shared and updatable in-memory database (MMDB) across multiple instances for seamless IP location lookup.Infrastructure Development - Gateway, Registry Center, MonitoringImplemented a robust infrastructure design for 'North-South' and 'East-West' traffic management and request tracking, handling over 1 million requests daily.Developed Lua scripts for OpenResty and maintained Consul Cluster/OpenResty in DEV/PROD environments, customizing the gateway with regional-based blacklists and specific redirection for internal users.Developed a service to auto-register/deregister Docker services to/from the Consul Cluster by monitoring docker.sock, enabling microservices interaction and registration as downstream services on the gateway.Deployed and maintained Cloud Host monitoring with Prometheus and Grafana, enabling automatic alerts for Cloud VM crashes. Show less

Master'S Degree, Information Engineer - Ai, Avg 74/100

Do not like current development of AI, read this please you may find the answer: Human-Level Intelligence or Animal-Like Abilities?.

Bachelor Of Technology - Btech, Software Engineering, Avg 80/100

Activities and Societies: First Prize of Hangzhou Dianzi University Freshman Cup Badminton Team Competition - 2015 Third Prize of Zhejiang.