Events Integration

Events: What, Why, How?

SPM can graph not only performance and custom metrics, but also events.  Such events may represent what is happening with a server or cluster, with an application (e.g., application or server restarts, deployments, alerts...), etc, as well as any sort of other event data that you want to correlate to metrics in SPM.  Events are graphed in timeseries charts and these charts can be shown next to all SPM metrics charts.  This makes it possible to easily correlate events and metrics.  In addition to showing events as timeseries charts, a detailed listing of events can be seen and, of course, events can have tags and priority, and can be searched and filtered.

Events are also exposed via a REST API that let's you post, retrieve, and search your events.  This REST API matches the Elasticsearch API, so you can use any Elasticsearch tool or client to post, get, and search events.

Event Fields

An event has the following set of fields, most of which are optional:

Adding Events

To post an event to your event stream use the following base endpoint:

http://event-receiver.sematext.com/APPLICATION_TOKEN/EVENT_TYPE

A single application token must be specified in the URL. Thus, to send multiple events associated with multiple applications, separate call to the API will need to be made for each application.  The EVENT_TYPE can have any value (e.g, alert, app_restart, server_restart, reboot, deployment...), but we suggest using a smaller number of distinct event types (1-10) to keep things manageable.

Example 1

Consider some SPM application whose token (your app tokens are at: https://apps.sematext.com/users-web/services.do) is 1111111-2222-3333-4444-555555555555.  To send a server_restart event call the Events API with token and event type:

http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/server_restart/

with POST content in JSON format like this:

{
  "timestamp" : "2014-02-17T15:29:04+0100",
  "message": "Application MyApp on MyHost04 restarted"
}

To post the above event with curl use: 

curl -XPOST "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/server_restart/" -d '
{
  "timestamp" : "2014-02-17T15:29:04+0100",
  "message" : "Application MyApp on MyHost04 restarted"
}
'

Example 2

Same SPM Solr application, but we want to post deployment event with more event properties populated. In this case the HTTP endpoint would be:

http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/deployment

with HTTP POST content:

{
  "timestamp" : "2014-02-17T15:58:04+0100",
  "message": "Solr 4.6.1 version deployed on prodhost06",
  "name" : "Solr 4.6.1 deployment",
  "tags" : ["solr", "4.6.1", "deployment", "upgrade"],
  "priority" : "High",
  "creator" : "John Smith"
}

or, again with curl:

curl -XPOST "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/deployment/" -d '
{
  "timestamp" : "2014-02-17T15:58:04+0100",
  "message" : "Solr 4.6.1 version deployed on prodhost06",
  "name" : "Solr 4.6.1 deployment",
  "tags" : ["solr", "4.6.1", "deployment", "upgrade"],
  "priority" : "High", "creator" : "John Smith"
}
'

Searching Events in SPM

SPM user interface lets you to show events and metrics from a specific time period. Additionally, the event chart has a search box where you can further narrow down events to only those that match the input query.

The query syntax is specified by Elasticsearch's query string query, as described here.

You can search on any event field you included in the event when posting it.

Searching Events Programmatically

SPM exposes the Events Search HTTP API - as Elasticsearch search API - so events can be searched and retrieved programmatically/remotely, via HTTP, using curl or any other Elasticsearch client.  The API endpoint is:

http://event-receiver.sematext.com/APPLICATION_TOKEN

Alternatively, you can also use the same endpoint which was used when adding events, where event type is specified, in which case the matching events will be limited to the type specified in the URI:

http://event-receiver.sematext.com/APPLICATION_TOKEN/EVENT_TYPE

The simplest way to run a query is using URI search, like this:

$ curl -XGET "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/_search?q=creator:john"

More query options are available when using request body search, e.g.:

curl -XGET "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/_search" -d '
  "query" : {
    "query_string" : {
      "query" : "MyHost04",
      "default_field" : "message"
    }
  } 
'

This example shows how to use one of the simpler query types - query_string. To see which other query types are available, please check Elasticsearch docs.

Security: Post Events via HTTPS

You can use HTTPS instead of HTTP for all calls, in which case the endpoint becomes:

https://event-receiver.sematext.com/APPLICATION_TOKEN

Note: when using curl, you may experience "SSL certificate problem" errors. The reason is that curl doesn't bundle any CA certs any more, for more info see this. Regardless of curl errors, HTTPS communication should be functional.