Big Data Engineer
Working as a data engineer/data analyst to build the next generation of data platform for ANZ and provide advanced data service to business for analytics. Working with business and architect to design solution for real business use cases(e.g. Responsible Banking; Financial Crime Data Hub); develop and implement them to production. Responsibilities: - Ingesting files from various source systems(core system, payments, fraud etc.) into big data platform via in-housed ETL framework; -… Show more Working as a data engineer/data analyst to build the next generation of data platform for ANZ and provide advanced data service to business for analytics. Working with business and architect to design solution for real business use cases(e.g. Responsible Banking; Financial Crime Data Hub); develop and implement them to production. Responsibilities: - Ingesting files from various source systems(core system, payments, fraud etc.) into big data platform via in-housed ETL framework; - Sqooping historical data from Teradata/Oracle database into big data platform; - Developing transformation tool to convert data per businesses' requirements; - Integrating with downstream systems for data virtualization on Qlik etc; - End-to-end testing support; - Change management, implementation and warranty support; - Production support; - Developing monitor tools for production environment by using Splunk etc. Primary Skills: - Big data skills like Hadoop; HDFS; Yarn; Hive; HBase; Hue; Impala; Flume; Sqoop - Spark (Java API; Pyspark; Scala; Spark-Streaming); - Kafka; - Java; - Python; - Scala; - Cloudera; - Zookeeper; - Teradata; - OracleResponsibilities:- Ingesting files from various source systems(core system, payments, fraud etc.) to into big data platform via in-housed ETL framework;- Sqooping historical data from Teradata/Oracle database into big data platform;- Developing transformation tool to convert data per businesses' requirements on Hadoop platform;- Integrating with downstream systems for data virtualization on Qlik or other tools;- End-to-end testing support;- Change management, implementation and warranty support;- Production support;- Developing monitor tools for production environment by using Splunk etc.Primary Skills:- Big data skills like Hadoop; HDFS; Yarn; Hive; Hbase; Hue; Impala; Flume; Sqoop- Spark (Java API; Pyspark; Sparkshell; Spark-Streaming);- Kafka;- Java;- Python;- Scala;- Cloudera;- Zookeeper;- Teradata;- Oracle Show less