Data Engineer
CurrentProject name: Data governance platformTech stack: Hadoop, Spark, Flink, Hive, Springboot, MySQL, Linux Shell, AnsibleProject Introduction: Chinaunicom Cloud is a cloud service provider that offers customers a range of cloud services, including ECS, Redis, Rabbitmq, MongoDB, MySQL, and EMR, etc. Our product is a data governance platform that enables customers to order it directly through the China Unicom Cloud platform. It provides a range of functions, including data governance, data collection, data asset management, metadata management, data development, data security management, data quality management, and data API management. The front-end of the product is mainly developed using AngularJS, the back-end service is developed usingSpring Boot. For the Hadoop platform, Ansible playbook is used to achieve the automated deployment of multiple components, such as Yarn, HDFS, Hiveserver2, Hivemetastore, Zookeeper, Spark, Flink, etc. Personal responsibility: My primary duties involve developing back-end SpringBoot services, providing API interfaces, developing Ansible automation deployment scripts, and maintaining the normal operation of Hadoop cluster components. I also assist clients in investigating and troubleshooting issues.