Elastic search alerts — Rich slack notifications

Elastic search alert slack notification

• The configuration in this blog post is for the open-distro version of elasticsearch.
• The blog post assumes you use elasticsearch for logs. However, the configuration examples are applicable to any data stored in the elasticsearch.

Introduction

During the configuration process of your monitoring/alerting, the Kibana will offer you with the default notification template. The default notification gives generic info that something has happened, however, you would probably need to include more information specific to your logging patterns to make alerting more useful.

Default elastic search action configuration

The above configuration produces the following notification:

Default elastic alert slack notification

Case study

If we want an alert configuration that watches our elasticsearch instance for WARN logs every hour, the above slack notification is better than not getting notified at all, but this can be further improved. Besides already present metadata, such as when did happened, monitor name, etc. , it would be useful to include those problematic logs in the notification body. That way, we can decide it’s worth further investigation or can it be ignored without opening the Kibana.

Which log properties to include highly depends on your logging practices, however, for this hypothetical one we will use the following: log message, service name, and the traceId (correlationId).

To do so, we need two things. The first one is to understand the context (ctx) data model, and have a basic Mustache (templating engine used by kibana) knowledge.

The alert context (ctx) has the following structure.

elasticsearch alert ctx structure

We have omitted details of monitor, trigger, and alert properties for the sake of brevity. If you are interested in those, you can read more here.

Now that we know how the ctx model looks like, we simply pick information we care about and apply slack specific formatting.

From mustache, we need the two basic features. First, to access the context value we can do it by simply putting the variable inside of a double brackets pair, for example:

{{ ctx.periodStart }}

The second feature we need is to loop over an array, which can is seen in the following example:

{{#ctx.results.0.hits.hits}} // Loop over hits
   {{_source.message}}
   {{_source.module_name}}
   {{_source.trace_id}}
{{/ctx.results.0.hits.hits}}

Now that we have all the information extracted, we can do slack specific formatting. Basically, we have all the slack formatting options at our disposal, like:

• *
• _
• ``

So we end out with something like this:

Information rich & formated slack notification template. Source code

You might be tempted to access array item with the square bracket notation as [0]. However, as this is a mustache template, the way we access array element is with the dot notation, as shown in the image above, results.0.hits.

The final notification in slack looks like this:

Information rich slack notification