Independent Study
Author, Uski.Me
With a focus on the recreational rental property market, the project is a distributed multi-part system that fetches/processes/aggregates and logs accommodation data (calendar, description, rates) from disparate rent-by-owner sites across the web. The data in turn backs a property search engine and supports the analysis of various occupancy metrics.Technologies:An Apache Storm topology, connected to a Kestrel queue, is used to fetch, extract and log the raw property data from web… Show more With a focus on the recreational rental property market, the project is a distributed multi-part system that fetches/processes/aggregates and logs accommodation data (calendar, description, rates) from disparate rent-by-owner sites across the web. The data in turn backs a property search engine and supports the analysis of various occupancy metrics.Technologies:An Apache Storm topology, connected to a Kestrel queue, is used to fetch, extract and log the raw property data from web sources. The topology is defined, written and tested in Clojure. The logs are serialized in Avro. Once logged, various Hadoop jobs are run to consolidate the data into forms suitable for search and analysis. At the present time, the outputs are simply exported via Sqoop into a database. The Hadoop entities are written, defined and tested in Java. Show less