Developing ORC Big Data project for Santander Global Tech. Creation of a data flow that automates the generation of a specific report. Obtaining data input from BDR databases through the creation of highly complex queries. Storing the raw data into partitioned Hive tables in ORC format in order to facilitate the analytics performance.Use of Apache Spark for processing and data analysis using PySpark and Spark SQL API's. Roles and responsibilities: • Responsible for creating Hive external tables, writing Hive queries and loading data in SUPRA Data Lake. • Basic knowledge of credit risk. • Involved in designing, development and testing Spark ELT. • Analyzed data using Hadoop components Hive and Impala Queries. • Designing and coding great quality experiences in an optimal timeframe. • Creation of data validation engine. • Managing deadlines. • Optimized data pipelines for real-time and batch processing on Databricks.

• Scala, Python and Spark SQL APIs. • RDDs and Dataframes processing and schema definition.• Loading, debugging and filtering of different datasets in different formats: csv, json, parquet, avro, etc. • Creation, insertion, storage and query analysis of tables with Hive, Impala and Pig. Transport of tables using Sqoop and Flume.• Installation and configuration of Hive, Hue, Sqoop, Zookeeper, Spark and Hbase.• Data flow processing through Spark Streaming.• Creation of topics, producers and consumer groups in Kafka to visualize the data flow.