API Management

Monitoring Azure API Management with OMS Log Analytics

Introduction

Out of the box, Azure API Management doesn’t come with tools to analyse the traffic that flows through it and to analyse events. Of course, the API Management Publisher portal gives you the Analytics page in which you see some statistical data, but it doesn’t give you the ability to look at the details of individual requests. You have to build or connect other tools to API Management to log and do analysis. This is where Log Analytics could come into play.

Log Analytics is part of the Operations Management Suite (OMS). It’s a powerful solution for log analysis and monitoring of services that live in Azure and on-premises. Log Analytics can give you additional insights about the requests that flow through your API Management instance, other than the statistical data in the Analytics section of the API Management publisher portal.

What’s also nice, it’s very affordable. There is even a free pricing tier that saves max 7 days of log data. So you can try it out without having to pay for it.

In this article I will show you how to use Log Analytics to analyse diagnostic logs for Azure API Management.

This article describes:

Setting up diagnostic logging for API Management

Lets start by going to the Azure Resource Manager portal. Then go to your API Management instance. In the left menu, scroll down and select Diagnostic logs. Click the Diagnostic settings link.

apim_diagnostic_logs1

In the Diagnostic settings blade, switch Status to ON. Then check the option ‘Send to Log Analytics’ and select your Workspace. (If you don’t have a Log Analytics workspace yet, you can create one there). Make sure the GatewayLogs checkbox is checked, under LOG. The METRIC option below doesn’t have to be checked, as we don’t need that data for the topic in this article.

apim_diagnostic_settings1

After you hit the save button, the diagnostic data will be sent to Log Analytics. Take into account that it may take up to 10 minutes before the first data becomes visible.

This is all you need to do . If you have multiple API Management instances, you need to repeat these steps for those instances. Of course you could automate this with Powershell or deploy your instances with ARM templates in which you configure the diagnostic settings.

Creating search queries in Log Analytics

Currently, while I’m writing this article, the OMS Solutions Gallery doesn’t contain a solution for Azure API Management. So in order to utilize the API Management diagnostic data we need to write our own search queries.

If you’re familiar with creating queries in Log Analytics, you will see that querying the API Management diagnostic data is a bit more challenging compared to other services that are configured in Log Analytics. To illustrate this, just type the name of your API Management instance in the search field, and filter the results based on Type ‘AzureDiagnostics’. You will see something similar as the image below.

APIM_Filtered_searchedresults

Much of the information we are interested in is contained within a single field called “properties_s“. That’s where the response codes, operations and other data is located. The problem is that all that stuff is stored in that single field. This makes it challenging to query, compared to how you would normally query . But with the use of regular expression symbols * and ? we should get the results that we need.

Example 1: Searching for HTTP 500 response codes

The query below checks for API Management events that report HTTP 500 response codes, which may indicate that something is wrong with your API on the back end.

Type=AzureDiagnostics DeploymentName_s="yourapimanager.azure-api.net" properties_s=*ResponseCode???500*

Let me explain what happens in this query.

Expression Description
Type=AzureDiagnostics Simply the type of logging that contains the API Management information we need
DeploymentName_s=”yourapimanager.azure-api.net” The fqdn of your API Management instance
properties_s= We look inside the properties_s field
* Is a wildcard that ignore all characters that come before it
? Means the character may be of any value. By using ??? I’m telling Log Analytics the three characters after ResponseCode may be any character. I know the actual response code is placed exactly 3 characters after the word ResponseCode and three characters before that may be ignored, because they probably contain spaces or punctuation characters.
500 The HTTP response code we’re actually looking for
* We end with the wildcard that ignores all characters that come after 500

Example 2: Searching for HTTP response codes in a certain range

This is almost the same query as example 1. But here I use a wildcard to search for HTTP return codes within a range. In this case the 400+ range. The query below will list events with response code 400, 401, 404, etc. Notice the 4* at the end which makes this happen.

Type=AzureDiagnostics DeploymentName_s="yourapimanager.azure-api.net" properties_s=*ResponseCode???4*

Example 3: List events from a specific IP address

Lets imagine you want to list all API requests that were made from a particular IP address. This is an easy one, because the caller IP address has its own property field: callerIPAddress. So to list the events from one address, use the query below.

Type=AzureDiagnostics DeploymentName_s="yourapimanager.azure-api.net" CallerIPAddress="11.22.33.44"

Example 4: Combining filters and conditions in a search query

If you need to further refine the results, simply add multiple conditions in the search query. In the query below I combined example 2 and 3 to search for response code 400+, a specific IP address and an additional condition to look for events that didn’t use caching.

Type=AzureDiagnostics DeploymentName_s="yourapimanager.azure-api.net" properties_s=*ResponseCode???4* CallerIPAddress="11.22.33.44" properties_s=*Cache????None*

 

Creating alerts

One of the features that makes Log Analytics a very useful monitoring tool is the ability to use a query and let it trigger an alert if a specific condition is met. An alert can be configured to send out a notification through email, the OMS mobile app, call a webhook (Slack, etc), and initiate an Azure Automation runbook.

To create an alert, first execute the search query. Then click the Alert button in the left top part of the window.

apim_oms_alert1

In the next window, configure your alert as you see fit. Give it a name, a severity, the time window,  alert frequency and of course, what the alert needs to trigger. So that could be an email, a webhook and a runbook.

apim_oms_alert2

More details about configuring alerts in Log Analytics can be found in the article “Understanding alerts in Log Analytics“.

Conclusion

As you can see, Log Analytics offers some very nice features to analyse the traffic that goes through Azure API Management. It also allows you to notify and automate things when certain events take place, using alerts and the ability to call webhooks and runbooks. And because Log Analytics is relatively cheap, it makes a very attractive solution.

Any comments or suggestions? I’d love to hear your thoughts in the comment section.