We are developing an autonomous UAV piloting system designed for mission execution in communication-deprived environments

Project: ETL data processingTeam Size: Dev Team 28 Automation QA 2 Architects 3 DevOps 6 Admins 2Project Role: 3rd Line / Software Maintenance EngineerDatabase: MS SQL, Oracle, Hive, ImpalaTasks performed: Currently supporting 8 projects. Implement ETL subsystem using sqoop, hive, oozie and sometimes Spark. The system is deployed on Cloudera 5.8.2 cluster.

The project idea is to build system based on lambda-architecture with splitting processing in real-time and batch processing. Business case is an imaginative online gambling company which operates across several countries, produce huge amount of data and has huge amount of bets that must be analyzed in real-time. A reports must be accessible for fast drill-down analysis as well as access to data of the highest level of granularity.Participation: 1.Developed hive, drill and druid services 2.Created query for hive and drill for drill down menu 3.Developed socket for sending data from generator to spark streamingTeam: 8 membersTools: Intellij IDEA, Git ,HDP 2.4, VagrantTechnologies: Spark Streaming, Hive, Drill, Druid, Amazon S3, Spring