I’m so glad it’s December with holidays, warm socks, and hot chocolate (or any other warm drink). Online events are happening worldwide, so I hope you have a chance to enjoy them.
In this post, I want to talk about anomaly detection. I feel like this topic is unfairly left behind.
If you have a project: IoT, web, mobile, etc., you certainly need to have logs for it and monitor its behavior. I think it became a required set up by now, but if you don’t have it in your solution, I recommend reevaluating the need for monitoring and analytics. It can help you to receive valuable information about the health, stability, errors and bugs, users’ behavior, and other things, so you’re not sitting in the dark, trying to figure out what worked and what didn’t and why.
Getting data from your application is only the first step. Next you need to analyze it, and based on that analysis, decide your next steps. If you read the title of this blog post, you probably know that I will be talking about doing all that using machine learning. There’re lots and lots of ready to use tools available out there and that might be the perfect solution for some businesses. If you need a custom solution, but without going too deep into machine learning, there’re at least two options, I can recommend to check out.
Let’s start with a more straightforward tool – Anomaly Detector from Cognitive Services (https://docs.microsoft.com/en-us/azure/cognitive-services/anomaly-detector/). You probably know I much I like Microsoft Cognitive Services, so it’s not surprising I’m talking about one of them here. Anomaly Detector service is relatively new, it became generally available in the second half of 2019 (unfortunately, I don’t remember the exact date), and this year it got a “younger sibling” in preview – Metrics Advisor (https://docs.microsoft.com/en-us/azure/cognitive-services/metrics-advisor/). What can the anomaly detector do? In general, it can detect anomalies in time series. More specifically, it offers three main functions:
· detect anomalies in the entire dataset,
· detect the anomaly status of the latest data point and
· detect the change point in the dataset
Like other cognitive services, it can be used with different programming languages, targeted hardware and no experience in machine learning (it always helps to have at least a perspective, but it’s not required). You have the REST API option and a client library option, which is available for C#, JavaScript and Python. No matter what you choose, you need to create an Anomaly Detector resource on Azure and use the keys and endpoints from there. It is also available as a Docker container, which is useful in IoT solutions. The functions mentioned above can be used together or separately and it will depend on the type of application you’re building and the data you have. If the data comes in chunks, you need to analyze all data you have at once and it’s not critical to notify about an anomaly immediately, then you can detect anomalies in the entire dataset or detect a change point in the dataset. The change point will show you when a change in your dataset happened. For example, it might be the start of an increase or decrease in the number of sales. If you’re monitoring a real-time system and need to get notifications as soon as an anomaly happened, you need to include the latest data point’s anomaly status. This function will let you know if the latest data point was an anomaly or not. Here’s the portal where you can try Anomaly Detector and see how it works (although you still need key and endpoint from Azure): https://algoevaluation.azurewebsites.net/.
Metrics Advisor is currently in preview, and it is similar to the Anomaly Detector. The main difference is that Metrics Advisor can detect anomalies and help with root-cause analysis. Use Metrics Advisor to:
· Analyze multidimensional data from multiple data sources
· Identify and correlate anomalies
· Configure and fine-tune the anomaly detection model used on your data
· Diagnose anomalies and help with root cause analysis
To check the Metrics Advisor out, you need to create an Azure resource and then connect it to the portal: https://metricsadvisor.azurewebsites.net/. I won’t go too deep into this service because it’s still in preview and there’s an excellent overview on Channel 9: https://channel9.msdn.com/Shows/AI-Show/Introducing-Metrics-Advisor. The portal is very lovely, but I could see it was still a work in progress. I’m excited to see it becoming generally available.
Cognitive Services are excellent, but sometimes you might need a more custom solution. To create that type of solution, you need to know how machine learning works, some model types and how a model can be trained. The world of artificial intelligence is endless, so I recommend reading books and doing online courses, like Pluralsight(https://www.pluralsight.com/) and Microsoft Learn(https://docs.microsoft.com/en-us/learn/) to get up to speed. There’re so many different frameworks and tools that can help you to create an ML model, but if you are a .NET developer, I have a piece of great news for you: you can use ML.NET to create a custom model for anomaly detection.
ML.NET offers three main functions:
· Time series anomaly detection
· Spike detection
· Change point detection
Machine Learning always starts with data. Data for the ML.NET anomaly detection can be loaded from relational databases(SQL Server, Azure SQL, IBM DB2, etc.) and single and multiple files. The primary sample provided in the official documentation includes data in a csv file: https://docs.microsoft.com/en-us/dotnet/machine-learning/tutorials/sales-anomaly-detection. When you work with a data file or model zip file received from the Model Builder, you need to remember that those files should be copied to the output directory during a build.
File properties
To create a console project to see how it works, you’ll need to add a couple of lines of code. When I tried to follow the instruction, one important thing I missed was Microsoft.ML.TimeSeries library. It should be installed along with Microsoft.ML. Although it is mentioned in the instruction, but it’s easy to miss it there.
What can you do after the anomaly detection gets implemented? The answer, as always, “it depends.” It depends on what kind of anomaly you’re tracking and what you’re planning to do with this information. There’s a couple of popular options available:
- Send notifications to involved participants. That can be done by using different technologies, sending sms, automated calls or emails. Twilio offers sms and email services. Azure functions support Twilio’s SendGrid, so there’s not much setup has to be done to connect an Azure function to SendGrid (official documentation from Microsoft: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-sendgrid?tabs=csharp). Azure Logic Apps and SignalR can be also used for sending notifications. On mobile devices, it can be just a push notification, without additional communication.
- If you’re not working with solutions that have to be maintained only by humans, you can connect the anomaly detection to a machine learning system. That ML system can trigger some actions based on the anomaly type or metrics advisor root-case analysis. Anomalies can be eliminated automatically, or ML can learn from that experience and then act differently next time to prevent unexpected data.
- If the identified anomaly wasn’t in fact an anomaly, you can notify the service or stop monitoring if you expect a managed anomaly to happen.
I recommend starting with checking Anomaly Detector and Metrics Advisor portals, seeing how the samples are working and then evaluating if that is suitable for your project or you would prefer something more or less custom. Lots of companies are using out of the box solutions and they work just fine for standard analytics, like log analysis. Still, for IoT and real-time systems, I would seriously consider building an “in-house” solution if possible. A couple of months ago, we discussed the Anomaly Detector and the Metrics Advisor with Seth Juarez and Willem Meints at a Global AI event: https://globalai.live/october-sessions-getting-started/make-smarter-decisions-with-cognitive-services/. The whole event was a huge success and combined interesting topics covering different parts of AI.
I want to wish everyone happy holidays! As always, I’ll be glad to get your comments, questions, and concerns on Twitter. I’ll be happy to chat.
