Events Integration

Events: What, Why, How?

SPM can graph not only performance and custom metrics, but also events.  Such events may represent what is happening with a server or cluster, with an application (e.g., application or server restarts, deployments, alerts...), etc, as well as any sort of other event data that you want to correlate to metrics in SPM.  Events are graphed in timeseries charts and these charts can be shown next to all SPM metrics charts.  This makes it possible to easily correlate events and metrics.  In addition to showing events as timeseries charts, a detailed listing of events can be seen and, of course, events can have tags and priority, and can be searched and filtered.

Events are also exposed via a REST API that let's you post, retrieve, and search your events.  This REST API matches the Elasticsearch API, so you can use any Elasticsearch tool or client to post, get, and search events.

Event Fields

An event has the following set of fields, some of which are optional:

Adding Events

To post an event to your event stream use the following base endpoint:

http://event-receiver.sematext.com/APPLICATION_TOKEN/EVENT_TYPE

A single application token must be specified in the URL. Thus, to send multiple events associated with multiple applications, separate call to the API will need to be made for each application.  The EVENT_TYPE can have any value (e.g, alert, app_restart, server_restart, reboot, deployment...), but we suggest having a smaller number of distinct event types (1-10) to keep things manageable.

Example 1:

Consider some SPM application whose token (your app tokens are at: https://apps.sematext.com/users-web/services.do) is 1111111-2222-3333-4444-555555555555.  To send a server_restart type of event we'd call the Events API with token and event type to:

http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/server_restart/

with post content in JSON format like this:

{
  "timestamp" : "2014-02-17T15:29:04+0100",
  "message": "Application MyApp on MyHost04 restarted"
}

The easiest way to quickly send an event is using curl, so the same event would be sent with the following command:

curl -XPOST "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/server_restart/" -d '{"timestamp" : "2014-02-17T15:29:04+0100", "message" : "Application MyApp on MyHost04 restarted"}'

Example 2:

Same SPM Solr application, but we want to send deployment event which should have more event properties populated. HTTP post would in that case be sent to:

http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/deployment

with content like:

{
  "timestamp" : "2014-02-17T15:58:04+0100",
  "message": "Solr 4.6.1 version deployed on prodhost06",
  "name" : "Solr 4.6.1 deployment",
  "tags" : ["solr", "4.6.1", "deployment", "upgrade"],
  "priority" : "High",
  "creator" : "John Smith"
}

or again with curl:

curl -XPOST "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/deployment/" -d '{"timestamp" : "2014-02-17T15:58:04+0100", "message" : "Solr 4.6.1 version deployed on prodhost06", "name" : "Solr 4.6.1 deployment", "tags" : ["solr", "4.6.1", "deployment", "upgrade"], "priority" : "High", "creator" : "John Smith"}'

Searching events in SPM

SPM user interface allows to define time range for which events (and SPM metric charts) are displayed. Additionally, event chart has search box where you can specify conditions for queries displayed in the chart:

Query syntax is specified by Elasticsearch's query string query, as described here.

You can do search on any field which was present in JSON content of your events when you were adding them.

Searching events outside of SPM

If you wish, you can also search your events using Elasticsearch search API. The base endpoint for searching is:

http://event-receiver.sematext.com/APPLICATION_TOKEN

however, you can also use the same endpoint which was used when adding events (in which case you are fetching only events of one specific type):

http://event-receiver.sematext.com/APPLICATION_TOKEN/EVENT_TYPE

Simplest way to run a query is using URI search, like this:

curl -XGET "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/_search?q=creator:john"

More query options are available when using request body search, e.g.:

curl -XGET "http://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/_search" -d '
  "query" : {
    "query_string" : {
      "query" : "MyHost04",
      "default_field" : "message"
    }
  } 
'

This example showed how to use one of the simplest query types - query_string. To see which other query types are available, please check Elasticsearch docs.

Security

You can use https instead of http for all calls, in which case endpoint becomes:

https://event-receiver.sematext.com/APPLICATION_TOKEN

Note: when using curl, you may experience "SSL certificate problem" errors. The reason is that curl doesn't bundle any CA certs any more, for more info see this. Regardless of curl errors, https communication should be functional.