Senior Data Engineer
- Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data
- Designed a custom Spark REPL application to handle similar datasets.
- Used Hadoop scripts for HDFS (Hadoop File System) data loading and manipulation
- Performed Hive test queries on local sample files and HDFS files
- Used AWS services like EC2 and S3 for small data sets.
- Developed the application on Eclipse IDE