Cheolho Choi Email & Phone Number

Tokyo, Jp

Tokyo, Japan

Tokyo, Japan

Engineering Director of LINE Data Platform.

Within 23 Wards, Tokyo, Japan

Have been working on projects for LINE's core data platform. - As a principal software engineer, I've designed the core architecture and hands-on implemented important components that are vital and affects to entire system characteristics. - Focused on structured data logging pipeline based on Protocol Buffers 3, Kafka, and Flink. It has been designed to be able to define and express various data schema requirements. It can store all data in multiple data sinks in a semi-realtime… Show more Have been working on projects for LINE's core data platform. - As a principal software engineer, I've designed the core architecture and hands-on implemented important components that are vital and affects to entire system characteristics. - Focused on structured data logging pipeline based on Protocol Buffers 3, Kafka, and Flink. It has been designed to be able to define and express various data schema requirements. It can store all data in multiple data sinks in a semi-realtime way, providing automatic Hive table mappings with orc formats. Hive tables are enabled dynamic schema, automatically updated on protobuf definition changes. - The system is capable of handling more than 10 millions of real-time event logs per second, hundreds of billions of records every day. - There are more than 1000 servers in a single Hadoop cluster with 100PB scale and the security integrated to the corporate SSO. It had built by HDP until last year, but recently we've changed to our in-house versions based on vanilla source codes. - The data platform also includes a couple of big Elasticsearch clusters and multiple Kibana sets to provide a real-time log search functionality over the incoming data stream. Some of the clusters are also security(authentications, access controls, audits) enabled, and clusters are applied hot-warm architecture to balance indexing throughput and disk usages for long-term index retentions. - Apache Flink is used for data pipeline connections between multiple, heterogeneous data systems - like Kafka, Elasticsearch, Hadoop, and HBase. - There are large-scale Kubernetes clusters in the data platform, mostly for stateless services and stream processing jobs - like Apache Flink jobs or other Kafka consumers. More components are planned to move into Kubernetes, including some core data processing engines like Presto and Spark to reduce YARN dependencies. Show less

Tokyo, Japan

Working on LINE messaging platform development regarding to various distributed systems for large scale data management/processing: - Building data access layer on top of NoSQL system like Redis and HBase - Handling service scalability and reliability issues - Building event-delivery and stream processing system - Large-scale data processing with Hadoop ecosystems - Kafka engineering for LINE messaging platform's core backend system

Search engine, Hadoop Mapreduce, Log data processing, Distributed filesystem, ...

Developed distributed web crawling system & search-engine services.

Developed network loadbalancing software & high-performance web-cache engine.