- As a Data Engineering Leader, I was responsible for managing, ensuring accountability, and building a team of Data Engineers specializing in the ETL domain, handling large-scale data analysis tasks, and leading a team of 4 members- Building, managing, optimizing, and monitoring Apache HBase and Apache Phoenix clusters as DataMarts for the Data Warehouse system, aiding in high-speed distributed storage and computation, ensuring high availability, and reducing query latency to under 3… Show more - As a Data Engineering Leader, I was responsible for managing, ensuring accountability, and building a team of Data Engineers specializing in the ETL domain, handling large-scale data analysis tasks, and leading a team of 4 members- Building, managing, optimizing, and monitoring Apache HBase and Apache Phoenix clusters as DataMarts for the Data Warehouse system, aiding in high-speed distributed storage and computation, ensuring high availability, and reducing query latency to under 3 seconds for large-scale data analytics tasks. This setup caters to over 100 data analysis reports for web and app NOC applications with more than 4000 active users.- Building automated ETL (Extract, Transform, Load) pipelines to analyze and process complex data streams from multiple sources into high-value foundational tables serves as an intermediary for constructing operational analysis reports for thousands of nationwide transportation hubs.- Constructing analysis reports and calculations to provide operational data for the NOC system with an update latency of under 5 minutes.- Research and apply Delta Lake for streaming ingestion and ETL for optimizing update latency of report Show less

- Designed, built, and developed data pipelines enabling data integration frommultiple sources, followed by data cleansing, transformation into standard formats,and ingestion into data storage systems (Data Lake)- Researched the architecture of big data technologies (Airflow, HBase, Druid, ...),constructed and tested these technologies in practical scenarios to apply them tovarious problem-solving requirements.- Constructed ETL (Extract, Transform, Load) processes to… Show more - Designed, built, and developed data pipelines enabling data integration frommultiple sources, followed by data cleansing, transformation into standard formats,and ingestion into data storage systems (Data Lake)- Researched the architecture of big data technologies (Airflow, HBase, Druid, ...),constructed and tested these technologies in practical scenarios to apply them tovarious problem-solving requirements.- Constructed ETL (Extract, Transform, Load) processes to facilitate datatransformation between large-scale data storage systems- Built and developed systems and tools to meet the needs of data exploration,analysis, and visualization (Metabase, Kibana, etc.).- Extracted data from the data lake, processed it, and generated daily, monthly, andquarterly data reports.- Collaborated and supported Data Analysts and Data Scientists in extracting andprocessing data on big data platforms.- As a mentor for several team members (intern or fresher), providing support intheir professional skills, system knowledge, and understanding of data. Show less