● Responsible for building scalable distributed data solutions using Spark .● Ingested files from source servers into HDFS data lakes using Sqoop.● Developed Sqoop Jobs to ingest customer and product data into HDFS data lakes.● Experience in working with Spark applications like batch interval time, level of parallelism, memory tuning to improve the processing time and efficiency.● Worked on various hive optimizations like partitioning, bucketing, vectorization, indexing and using right type of hive joins like Bucket Map Join and SMB join.● Worked with various files format like CSV, JSON, ORC, AVRO and Parquet.● Optimized spark jobs using various optimization techniques like broadcasting, executor tuning, persisting etc.