Toggle Main Menu Toggle Search

Open Access padlockePrints

A Platform for Analysing Stream and Historic Data with Efficient and Scalable Design Patterns

Lookup NU author(s): Rebecca Simmonds, Professor Paul WatsonORCiD, Professor Paolo MissierORCiD


Full text for this publication is not currently held within this repository. Alternative links are provided below where available.


Social media is an increasingly popular method for people to share information and interact with each other. Analysis of social media data has the potential to provide useful insights in a wide range of domains including social science, advertising and policing. Social media information is produced in real-time, and so analysis that can give insights into events as they occur can be particularly valuable. Similarly, analytics platforms providing low latency query responses can improve the user experience for ad-hoc data exploration on historic data sets. However, the rate at which new data is generated makes it a real challenge to design a system that can meet both of these challenges. This paper describes the deisgn and evaluation of such a system. Firstly, it describes how a meta-analysis of the types of questions that were being asked of Twitter data led to the identification of a small set of queries that could be used to answer the majority of them. Secondly, it describes the design of a scalable platform for answering these and other queries. The architecture is described: it is cloud-based, and combines both continuous query, and noSQL database technology. Evaluation results are presented which show that the system can scale to process queries on streaming data arriving at the rate of the full Twitter firehose. Experiments show that queries on large repositories of stored historic data can also be answered with low latency. Finally, we present the results of queries that combine both streaming and historic data.

Publication metadata

Author(s): Simmonds R, Watson P, Halliday J, Missier P

Publication type: Conference Proceedings (inc. Abstract)

Publication status: Published

Conference Name: 2014 IEEE World Congress on Services

Year of Conference: 2014

Pages: 174-182

Print publication date: 01/01/2014

Online publication date: 28/02/2014

Acceptance date: 27/06/2014

Publisher: IEEE


DOI: 10.1109/SERVICES.2014.40