28 06, 2015

Interactive low-latency motif discovery at scale

2016-12-28T00:21:14+00:00 Analytics, Motif Analysis, Time Series Analysis, Uncategorized|

Many algorithms have been developed in the academic world for Motif/time series analysis such as the popular dynamic time warping (DTW). However, most such algorithms have not been implemented to scale and as a result can not provide interactive low-latency response times needed for web applications when dealing with large data sets.

Parallel processing frameworks (such as map reduce) can improve the performance but require large (and expensive) clusters of servers to achieve significant improvements in response times. The expectation of users is often set by using OLAP tools or internet search engines where sub-second response times are the norm.

At Trendalyze Decisions, we have implemented our own patent-pending techniques that can accomplish low latency interactive response times when processing large number of time series for motif search, match and ranking.

This is illustrated below using data from NHS England comparing Trendalyze on a single node with Map Reduce on five nodes:

Scalability Chart