Big Data Engineer
Ankara, Turkey
-Data ingestion with Spark from HDFS, processing data as various XML files and transferring the data into Soap Service.-Migrating data from SQL Server into HPE Data Fabric (maprFS) as parquet files using Apache Spark, mounting those files as external Hive tables, denormalizing and modeling.-Management of Data Fabric cluster.-Databases used for querying data and detailed analysis such as MSSQL,Hive,Oracle.-Experience in working at Linux systems.-Experience in installation of… Show more -Data ingestion with Spark from HDFS, processing data as various XML files and transferring the data into Soap Service.-Migrating data from SQL Server into HPE Data Fabric (maprFS) as parquet files using Apache Spark, mounting those files as external Hive tables, denormalizing and modeling.-Management of Data Fabric cluster.-Databases used for querying data and detailed analysis such as MSSQL,Hive,Oracle.-Experience in working at Linux systems.-Experience in installation of Hadoop,Spark,Airflow at Linux based systems.-Creating ETL Workflows via Spark, Airflow.-Configuring Hive,Hue,Airflow to use MySQL as a backend db.-Experience in upserting(merge) the daily data via Delta Lake. Show less