Finding the Signals: Why Unsupervised Machine Learning Techniques are Key for Anomaly Detection

The need to find anomalies

Businesses collect, record and analyze data about key performance indicators (KPIs) in the hopes of getting objective, quantifiable feedback on the success or failure of their actions and strategies. Each of these metrics has a normal baseline of behavior over time. Deviations (especially unexpected ones) from that baseline can signal opportunities to generate more revenue, or problems which are causing that business to lose revenue. These anomalies can be important signals which need to be acted on fast, and that’s why dashboards and alerts have long been a key component of basic business intelligence (BI). These tools allow the people who monitor them to react swiftly to any anomalies that may arise.

For some organizations, the number of these monitored metrics can number in the thousands or even millions. The problem is that there are only so many dashboards that a single person can monitor, and only so many alerts that a team can triage and respond to. Scaling out will not solve the problem: even if one person can effectively detect anomalies in real-time for 100 metrics for example, it would take 1,000 human analysts to handle 100,000 metrics. The scaling is even worse than that because of the communication overhead required: each of these analysts would then have to effectively communicate (in real-time) with each of the 999 analysts in order triage the important anomalies, group related anomalies together and analyze the root cause.

The search for automated solutions

It’s hardly surprising that many companies have been shifting toward automated anomaly detection. Offloading this task onto computers alleviates the nasty scaling problems noted above - if that technology is as good as flagging anomalies in time series data as the humans it replaced. See, millions of years of evolution have endowed humans with a highly sophisticated visual processing system. As a result, spotting anomalies in the graph of a time series is fairly easy, as we use our brain’s powerful pattern-finding ability to get a sense of the normal behavior of a given metric over time and then we visual scan that plot to find irregularities.

By extension, any automated anomaly detection method must also feature a similar level of sophistication and intelligence. This is where artificial intelligence (AI), specifically modern machine learning algorithms, steps in.

The advantage of unsupervised machine learning

There are essentially two main categories of anomaly detection methods: supervised and unsupervised. Supervised methods are perhaps the easiest to understand, because those are essentially training by example: the algorithm is fed lots of example items that are pre-categorized by humans before the algorithm is set loose to categorize new data.

An example of a supervised machine learning algorithm is one which is first trained on a massive set of images of cats and dogs. Each image in this training data has already been labeled as “cat” or “dog”. This algorithm could then scour through someone’s online picture gallery and automatically tag that person’s images according to which type of animal is present in the photo. This tag could be added to the meta-data of each image, making it possible for that user to search their images for ones which contain either cats or dogs (or both). This program could work quite well…until that algorithm encounters a picture containing only a horse.

Because supervised machine learning algorithms are only aware of the categories they’ve been trained on, they are unable to process items which belong to categories that weren’t included in that training data. Depending on how that algorithm was implemented in code, it may crash, skip that picture or label it with one of categories it’s actually familiar with. As inconvenient as a software crash may be, in some cases that would actually be better than a misclassification, as Google found out in the summer of 2015 when its Photos application labeled two African-American friends as “gorillas”.

Unsupervised machine learning algorithms, on the other hand, aren’t limited by a training data set. They learn by processing the “real” data. In the context of anomaly detection, this means these types of algorithms use the data in a time series to construct an internal model of what is normal for that particular metric, and anomalies are detected by submitting every data point to a statistical test. This means there’s a degree of probability built into unsupervised machine learning algorithms. The main advantage of unsupervised approaches is that they can detect types of anomalies that no one has seen before in addition to known types.

As real-time analytics software vendor Anodot explains, unsupervised anomaly detection techniques identify anomalies in much the same way we naturally flag odd events in everyday life. As we noted above, this human-like reasoning is a requirement for an ideal automated anomaly detection system.

Let’s continue with our animal example to see how an unsupervised machine learning method would work. First, it would read in a massive amount of animal pictures, construct its own animal categories along with the distinguishing characteristics for each of them, and then classify new images. This is exactly what happened when Google’s neural network used an unsupervised machine learning algorithm to learn what a cat is from YouTube videos. Advanced machine learning has the ability to spot the cats (and horses) in data on its own.

What’s waiting to be discovered in yours?

Is it a sudden drop in mobile app installs due to the release of a new version of a competing app? Or is it a spike in online sales among millennials due to an unusually effective online ad strategy?

Your business can’t afford to let these blips get lost in a sea of alerts on someone’s dashboard or let these signals to linger in some analyst’s queue while your competitors outmaneuver you. The sooner you find out your new ad strategy worked, the sooner you can increase your ad buys and increase the flow into your sales funnel, the better. Use unsupervised machine learning to give your BI some actual intelligence.