Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Getting Started

...

A: You can reach us at spm-support@sematext.com or tweet @sematext.

Is there an On Premises version of SPM that I can run on my own servers

A: Yes there is.  Please see SPM On Premises Edition.

Is there a limit to how many servers I can monitor with SPM

A: Each plan has a limit, but it is pretty high and if you need to monitor more servers just let us know.

Can I use SPM to monitor any application

A: SPM can be used to monitor any UNIX server and any Java application in addition to monitoring specific applications.

Can I run the complete SPM on my own servers

...

Which Applications can SPM monitor

A: SPM can monitor Hadoop, HBase, Kafka, Spark, Storm, Cassandra, Elasticsearch, Solr/SolrCloud, MySQL, MariaDB, Redis, Memcached, Apache, HAProxy, ZooKeeper, AWS/CloudWatch, SenseiDB, JVM, NodeJS, Express, etc.  You can also feed it Custom Metrics.

Does SPM Integrate with ChatOps like Slack and HipChat?  How about PagerDuty or custom WebHooks

A: Yes, see Integrations.

Can hostnames in SPM UI be obfuscated or customized

A: Yes, you can obfuscate or alias hostnames starting with SPM client version 1.17.7. This feature allows you to:

  • never send your real hostnames over the Internet
  • use custom hostname in SPM UI in case original hostname is too cryptic (e.g. have SPM shows "my-solr-host1" instead of "ip-12-123-321-123")

To achieve this, after SPM client is installed, open /opt/spm/properties/spm-sender.properties file and just add desired hostname to the value of property spm_sender_hostname_alias, e.g. spm_sender_hostname_alias=web1.production.

After that, restart SPM monitor with:

sudo service spm-monitor restart

and restart any Java process which was using javaagent/in-process version of SPM monitor.

Few notes to keep in mind:

  • you can do this change at any point, even after SPM client was running on your machine for the past few months or years - you just have to upgrade SPM client to version 1.17.7 or greater
  • old data will still be seen in SPM under the old hostname, while new data (after hostname change) will be displayed under the new hostname
  • if you are installing SPM client on some machine for the first time, and you want to be 100% sure its original hostname never leaves your network, define your hostname alias in spm-sender.properties file immediately after you complete "1. Package installation" step and before you begin with "2. Client configuration setup" step (Installation instructions can be accessed from https://apps.sematext.com/ui/monitoring, click Actions->Install Monitor on app you are installing)

Can I reduce the amount of logs generated by SPM monitor

A: Yes, starting with 1.20.6 version of SPM client you can make SPM monitor reduce the number of messages it logs every minute (otherwise it logs detailed info about its monitoring lifecycle). To do that, change properties files of your SPM applications. You can find those files with:

Code Block
ls /opt/spm/spm-monitor/conf/spm-monitor-config-*.properties

You can adjust one or more of them, depending on which application's monitor log output you want to reduce. At the bottom of those files add the following line:

Code Block
SPM_MONITOR_LOGGING_LEVEL=reduced

After you have made the changes, restart your SPM monitor – if you are using standalone variant, run: sudo service spm-monitor restart , otherwise restart your application/java process.

How frequently does SPM monitor collects metrics and can I adjust that interval

SPM monitor by default collects metrics every 10 seconds. To reduce this frequency to 30 seconds, for example, simply add the following line to your SPM monitor properties files located in /opt/spm/spm-monitor/conf :

Code Block
SPM_MONITOR_COLLECT_INTERVAL=30000

The value is expressed in milliseconds. If you are adjusting it, we recommend setting it to 30000. With bigger values it is possible some 1-minute intervals would be displayed without the data in the UI.

After you make this change, make sure SPM monitor is restarted (if using in-process monitor, restart the monitored services; if using standalone monitor, just run sudo service spm-monitor restart).

How much memory is standalone SPM monitor using and can it be adjusted

By default, each standalone SPM monitor process is started with "-Xmx384m" setting. This means that its JVM heap will use 384 MB at most. In many cases SPM monitor doesn't actually need or use that much memory. If you want to be absolutely sure about it, simply lower this number in /opt/spm/spm-monitor/bin/spm-monitor-starter.sh, with the following variable (around line 63):

Code Block
JAVA_OPTIONS="$JAVA_OPTIONS -Xmx384m -Xms128m -Xss256k"

You can change "-Xmx" part to "-Xmx192m" for example and see how it goes (check if SPM monitor is still working over the next few hours/days and is still not using much CPU).

We don't recommend changing this setting if you are monitoring Elasticsearch or Solr clusters with a high number of indices or shards. In those cases SPM monitor has to parse and gather large amounts of metrics data returned by Elasticsearch or Solr.  You can contact us in chat or via email (spm-support@sematext.com) if you'd like to get more info and help around making this adjustment.

Can I use SPM to monitor multiple applications running on the same server / VM

A: Yes. There are really 2 different scenarios here:

1) If each of those applications should be monitored under a different SPM application (e.g., you could have Solr running on your server along with some Java app and you want to monitor both - Solr would be monitored with SPM application of Solr or SolrCloud type, while the Java app would be monitored with SPM application of JVM type), just complete all installation steps (which are accessible from https://apps.sematext.com/ui/monitoring, click Actions->Install Monitor for app you are installing) for each of them separately.

2) If you want them monitored under the same SPM application (e.g., you have 3 Solr instances running on a server), you must use different JVM name for each of them. To do this, "1. Package installation" step should be run only once on this machine, while "2. Client configuration setup" step should be run once for each of the 3 Solr instances (installation instructions are accessible from https://apps.sematext.com/ui/monitoring, click Actions->Install Monitor for app you are installing). When running script /opt/spm/bin/spm-client-setup-conf.sh in step "2. Client configuration setup", you should add jvmname parameter (and value) at the end of parameter list, like this:

Code Block
sudo bash /opt/spm/bin/spm-client-setup-conf.sh 11111111-1111-1111-1111-111111111111 solr javaagent jvmname:solr1

In this example, we are setting up things for 3 separate Solr processes, monitored under JVM names: solr1, solr2 and solr3 (you choose any names), so this is adjusted command for solr1 instance.

In the remaining sub-steps of "2. Client configuration setup", replace word "default" with the jvmname value you just used.

"2. Client configuration setup" step will have to be repeated N times, once for each monitored application (in our example 3 times with 3 different jvmname values).

Note: By using this kind of setup, you will be able to see JVM stats of all 3 processes separately (JVM name filter is used to do the filtering). When it comes to other, non-JVM stats, they will be aggregated into single value (for instance, request rate chart will show sum of request rate value of these 3 Solr instances). If you want to see even non-JVM stats separately, you will have to create 3 separate SPM Solr applications (one for each Solr instance running on this machine).

Can I use SPM to monitor my service which runs on Windows or Mac OS X

A: SPM client installers currently exist only for Linux, however there is still a way to monitor your service. If you are OK with installing SPM client on separate linux box, and pointing it (more about that further below) to your service (which should be monitored) running on Windows or Mac, you can use SPM to monitor all non-OS related metrics. That means you will not be able to see CPU or system Memory metrics, but metrics specific to your process will be displayed (for example, in case of Solr, you will see all search, index, cache metrics along with all JVM metrics like pool memory, GC stats, JVM threads etc; as another example, in case of MySQL you will see all metrics related to connections, users, queries, handlers, commands, MyISAM, InnoDB, MySQL traffic, etc) .

When monitoring Solr, Elasticsearch, HBase, Hadoop and other Java-based services, you will have an option to choose between using In-process (javaagent) or Standalone monitor. The workaround described here requires the use of standalone monitor variant. Here's what you'd need to do to see your metrics in SPM:

  1. Install spm-client on any Linux box (you can use this box for anything else, it is needed here just to run a process which collects metrics and sends them to Sematext). Installation instructions are accessible from https://apps.sematext.com/ui/monitoring, just click Actions->Install Monitor for app are installing. In step 1 choose the "Other" tab to create a minimal installation.  This will not install modules needed for monitoring OS metrics - if those modules were installed, they would collect OS metrics of your Linux box, which is probably something you don't need. If you decide to follow instructions from some other tab, keep in mind OS metrics displayed in SPM UI will not be OS metrics of your Windows/Mac machine.
  2. In step 2, if you are given a choice between In-process and Standalone monitor, choose Standalone monitor. It will use remote JMX to collect metrics from your Windows/Mac machine. Just follow instructions given on Standalone tab.  The only difference you will want to make is in -Dspm.remote.jmx.url parameter used in monitor properties.  You will know where to adjust it if you follow standard instructions for Standalone monitor. While this parameter will usually have a value like -Dspm.remote.jmx.url=localhost:3000, in this case you will have to replace localhost with the address of your Windows/Mac machine(s) by which it can be reached from your helper Linux box.
  3. If you are monitoring something that doesn't offer a choice between In-process and Standalone monitor, installation instructions will explain where you can define your own address. Instructions typically use localhost, but you can instead use something like 10.1.2.3 or my-win-solr-server.mycompany.com to point to the machine that hosts the service you intend to monitor.

In case you want to monitor multiple machines belonging to the same cluster in this way, you can still use SPM installed on a single Linux helper box. Just do this:

  • Create a separate SPM application (https://apps.sematext.com/ui/integrations) for each of the machines which should be monitored
  • For each of those SPM applications (and your monitored machines), go through installation process presented after the app was created (installation instructions are accessible from https://apps.sematext.com/ui/monitoring, just click Actions->Install Monitor for app your are installing). Note: Since each SPM application uses its own token, they all have slightly different installation commands. Besides different token, you will also use different addresses of Windows/Mac machines that host the monitored service.

Can SPM monitor metrics via http-remoting-jmx protocol (e.g. WildFly, JBoss, etc.)

Yes. The following steps are needed:

In /opt/spm/spm-monitor/conf/spm-monitor-config-YOUR_TOKEN-default.rt look for a line similar to this (the file name might be a bit different, depending on app type you have chosen):

Code Block
SPM_MONITOR_JAR="/opt/spm/spm-monitor/lib/spm-monitor-jvm.jar"

Replace this line with something like the following (e.g. in case of WildFly adjust the path depending on where your WildFly is installed, like /opt/wildfly/bin/client/jboss-cli-client.jar):

Code Block
SPM_MONITOR_JAR="/opt/spm/spm-monitor/lib/spm-monitor-jvm.jar:/path/to/your/jboss-cli-client.jar"

Change the value of SPM_MONITOR_JMX_PARAMS in /opt/spm/spm-monitor/conf/spm-monitor-config-YOUR_TOKEN-default.properties. Of course, you can append to that additional JMX parameters, for example with password file location etc:

Code Block
SPM_MONITOR_JMX_PARAMS="-Dspm.remote.jmx.url=service:jmx:http-remoting-jmx://localhost:9990"

...

At this point the metrics will start appearing in charts. If they don't, run the diagnostics script (sudo bash /opt/spm/bin/spm-client-diagnostics.sh) to get fresh output of errors. If you see errors similar to:

Code Block
Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:

java.io.FileNotFoundException: /opt/wildfly/standalone/tmp/auth/local8363680782928117445.challenge (Permission denied)

javax.security.sasl.SaslException: DIGEST-MD5: Cannot perform callback to acquire realm, authentication ID or password [Caused by javax.security.auth.callback.UnsupportedCallbackException
it means there is permissions issue on WildFly dirs. There are two ways to get around this:
  1. Run SPM monitor with the same user that is running WildFly. To do that, add an entry like this to the end of /opt/spm/spm-monitor/conf/spm-monitor-config-YOUR_TOKEN-default.properties:

    Code Block
    SPM_MONITOR_USER="wildfly"

    After that restart spm-monitor with: sudo service spm-monitor restart

  2. Change permissions for the problematic directory, adjusting the path to match your environment:

    Code Block
    chmod 777 /opt/wildfly/standalone/tmp/auth 

    This approach is not encouraged because of the obvious security problem, so use such approach only while testing, or if other options are not possible. As usual, restart spm-monitor after this change.

When should I run Standalone and when Embedded SPM monitor

A: Standalone SPM monitor runs as a separate process, while Embedded monitor runs embedded in the Java/JVM process.  Thus, if you are monitoring a non-Java application, Standalone monitor is the only option.  Standalone monitor is a bit more complex to set up when one uses it to monitor Java applications because it typically requires one to enable out-of-process JMX access, as described on Standalone SPM monitor page.  With Embedded monitor this is not needed, but one needs to add the SPM agent to the Java command-line and restart the process of the monitored application.  When running Standalone monitor one can update the SPM monitor without restarting the Java process being monitored, while a restart is needed when Embedded SPM monitor is being used.  To be able to trace transactions or database operations you need to use the Embedded SPM monitor.

Can I use SPM for (business) transaction tracing

A: Yes, see Transaction Tracing.

Can I move SPM client to a different directory

A: Yes. Starting with version 1.22.25, SPM client provides two scripts that can be used to move SPM client files/directories to another location:

1) Soft move script - Moves all SPM files/directories to a new location, but symlinks /opt/spm to the new location. Use this script if you are OK with having /opt/spm symlinked. This script is recommended for most situations since it keeps your SPM client installation completely in line with standard setup (all standard SPM client commands and arguments are still valid).

It accepts 1 parameter: new directory where SPM client should be moved to (if such directory doesn't exist, it will be created)

Code Block
sudo bash /opt/spm/bin/move-spm-home-dir-soft.sh /mnt/some_dir

And that is it.

2) Full move script - Moves all SPM files/directories to a new location without leaving any SPM artifacts in /opt directory. Please note that using this script will make management of SPM client a bit harder (for example, there will be some required actions before running SPM client upgrade; also, if you are using in-process (javaagent) type of monitor you will need to adjust some paths in java arguments - more about that below).

It accepts 2 parameters: the current location (by default /opt/spm), and the new location (if such directory doesn't exist, it will be created). It can be run as:

Code Block
sudo bash /opt/spm/bin/move-spm-home-dir.sh /opt/spm /mnt/some_other_dir/spm

If you are using in-process (javaagent) versions of SPM monitor to monitor Java-based applications, please adjust agent path(s) you use when starting application(s) you monitor with the SPM agent and make sure to use new location where old location was used before. Also add the following argument to the monitored application's Java process (you can add it right before -javaagent part; adjust the value to match directory you used):

Code Block
-Dspm.home=/mnt/some_other_dir/spm
and restart your application/server.

Note: When doing SPM client update on systems where SPM client was moved to different directory using "2) Full move script", you will first have to move SPM back to original /opt/spm directory:

Code Block
sudo bash /opt/spm/bin/move-spm-home-dir.sh /mnt/some_other_dir/spm /opt/spm

and then proceed with regular update. After update is done, you can again move SPM client to location which suits you.

Is there an HTTP API I could use

A: Yes, see SPM API Reference.

I have multiple SPM applications installed on my machine, can I uninstall just one of them

A: Yes, you can use the following command for that (it accepts only one parameter, token of SPM application you want to uninstall):

Code Block
sudo bash /opt/spm/bin/spm-remove-application.sh 11111111-1111-1111-1111-111111111111

Can I disable SPM agent without uninstalling the SPM client

A: Yes, just find its .properties file in /opt/spm/spm-monitor/conf and add to it:

Code Block
SPM_MONITOR_ENABLED=false

After that restart the monitor to apply the change. In case of standalone agent, run:

Code Block
sudo service spm-monitor restart

And in case of in-process agent, just restart the service that has this monitor's javaagent parameter.

How can I share my apps with other users

A: There are two options: Account Sharing or App Sharing. With Account Sharing, you invite others to your whole account, so they get access to all your applications, dashboards, notification hooks, alert rules, etc.,.  They can also create new apps under your account and invite other users. Depending on the role you assign to invitees, they may be able to administer your apps, dashboards, users or even billing info (change app plan, assign credit card, etc). Account Sharing is very convenient because as soon as a new app (SPM or Logsene or ...) is added to your account or new dashboard is created, all users added to your account get access to it.  Of course, the level of access depends on the role you initially assigned to each person.

...

When would I want to add someone as BILLING_ADMIN

A: When you don't have a credit card that should be used for charging, but some other person has it, you should invite this person and give them the BILLING_ADMIN role. Similarly, if you created an account and defined a credit card, but now want somebody else to take care of all billing related activity (assigning plans and credit cards to various apps), you'd give them the BILLING_ADMIN role.

What might a typical use of roles for an organization with many employees look like

A: Typically you might have one person create an account via https://apps.sematext.com/ui/registration. This account might be considered a "parent" account for your whole organization.
Since the person who created the account would be its OWNER, this person is typically (but not necessarily) a team leader or manager, or somebody whose responsibility is to oversee servers/operations, typically in production. This person might then choose to share his/her whole Account with every other person from the team/organization to allow others to easily access all apps created under that account.

Some of the invited users might be given the ADMIN role, which gives them read and write (and invite) rights. Other invitees might get the USER role, which gives mostly just read rights (plus ability to create/edit their own dashboards/alerts/subscriptions which are available to everyone under the shared account). In some cases, account OWNER will not be able to handle billing related info and will want to invite 1 or more BILLING_ADMINs who will be able to define/edit/delete credit cards and choose plans to be used for apps under your account.

Agent Setup Automation

Is there an Ansible Playbook for the SPM client

A: Yes, see the Install and Configure playbooks, with examples.

Is there a Puppet Module for the SPM client

A: Yes, see the Install and Configure module, with examples.

Is there a Chef Recipe for the SPM client

A: Yes, see SPM client Chef Recipe example.

General Troubleshooting

Can I enable debugging in the SPM Agent

A: Yes. Simply add or edit the SPM_MONITOR_LOGGING_LEVEL property in any of the /opt/spm/spm-monitor/conf/spm-monitor\*.properties files and restart the agent (or the process the agent is attached to).  Available levels are: FATAL, ERROR, WARN, INFO, DEBUG, TRACE

Can I install SPM client on servers that are behind a proxy

A: Yes.  If you are installing the RPM, add this to /etc/yum.conf:

Code Block
proxy=http or https://proxy-host-here:port
proxy_username=optional_proxy_username
proxy_password=optional_proxy_password

If you are using apt-get, set the http_proxy environmental variable:

Code Block
export http_proxy=http://username:password@yourproxyaddress:proxyport

Can SPM client send data out from servers that are behind a proxy

A: Yes, just provide the proxy info in the /opt/spm/properties/spm-sender.properties file:
Code Block
spm_sender_proxy_host=
spm_sender_proxy_port=
spm_sender_proxy_user_name=
spm_sender_proxy_password=

When installing SPM client, I see "The certificate of `pub-repo.sematext.com' is not trusted" or similar error.  How can I avoid it

A: There can be multiple reasons for this, most likely:

  • system time on your machine is not correct and adjusting it should fix the problem
  • you are installing SPM client as root user - in some cases that can cause this error and installing SPM client as a different user may help
  • you are missing the ca-certificates package.  Use one of the following commands to install it:
Code Block
sudo apt-get install ca-certificates
sudo yum install ca-certificates

If none of the above eliminates the problem for you try adding a flag to avoid certificate checking. In case the command with wget is failing in your case, add "--no-check-certificate" as wget argument. If command with curl is failing, add "-k" flag.

How do I create the diagnostics package

A: If you are having issues with SPM, you can create diagnostics package on affected machines where SPM client was installed by running:

Code Block
sudo bash /opt/spm/bin/spm-client-diagnostics.sh

The resulting package will contain all relevant info needed for our investigation. You can send it, along with short description of your problem, to spm-support@sematext.com or contact us in chat.

I see only my system metrics (e.g. CPU, Memory, Network, Disk...), but where is the rest of my data

A: Make sure you have followed all steps listed on installation instructions page. Package installation steps should be done first, followed by Client configuration setup. If you have done that and you still don't see application metrics, run sudo bash /opt/spm/bin/spm-client-diagnostics.sh to generate diagnostics package and send it to spm-support@sematext.com with description of your problem.

I DO NOT see any system metrics (e.g. CPU, Memory, Network, Disk), what could be the problem

A: Make sure you have followed all steps listed on installation instructions page. It is possible you missed Client configuration setup step. If you have done that and you still don't see application metrics, run sudo bash /opt/spm/bin/spm-client-diagnostics.sh to generate diagnostics package and send it to spm-support@sematext.com with description of your problem.

I am using SPM Solr or SPM ES. My Request Rate and Latency charts are empty

A: If other Solr/ES charts are showing the data, it is most likely that there were no requests sent to your Solr/ES in the time range you are looking at. Try sending some queries and see if request rate/latency charts will show them. If they don't, please send us an email to spm-support@sematext.com or contact us in chat.

I am not seeing any data in SPM charts, how do I check if network connectivity is OK

A: SPM agents send the data to SPM backend (hosted on sematext.com servers) so it is important that servers where you install SPM agent can access the internet. Things to check to ensure network connectivity is ok:

1. Try connecting to spm-receiver.sematext.com with the following command:

Code Block
nc -zv -w 20 spm-receiver.sematext.com 443

The output should show something like: Connection to spm-receiver.sematext.com 443 port [tcp/https] succeeded!

In case you see some other result:

  • if your server requires proxy to access the internet, you can define its settings in /opt/spm/properties/spm-sender.properties. After that restart SPM agent.
  • if firewall is used to protect your server, it may be blocking outbound traffic from it. SPM agent sends the data over port 443, so please ensure with your network admins that port 443 is open for outbound traffic
  • check your DNS (see below)

2. Check if your DNS has correct entries for SPM Receiver:

Code Block
nslookup spm-receiver.sematext.com

The output of this command should look like this, although the IP addresses and names may be somewhat different, as they change periodically:

Code Block
Server:        127.0.1.1
Address:    127.0.1.1#53

Non-authoritative answer:
spm-receiver.sematext.com    canonical name = SPM-Prod-Receiver-LB-402293491.us-east-1.elb.amazonaws.com.
Name:    SPM-Prod-Receiver-LB-402293491.us-east-1.elb.amazonaws.com
Address: 50.16.206.179
Name:    SPM-Prod-Receiver-LB-402293491.us-east-1.elb.amazonaws.com
Address: 107.20.222.136

If you see different output, you may want to check with your network admins if everything with your DNS is ok or may need adjustment to be able to reach spm-receiver.sematext.com.

I registered for SPM more than 5 minutes ago and I don't see ANY of my data / I am getting errors when starting SPM Monitor or when starting my service which uses javaagent version of SPM monitor, what should I check

...

sudo service spm-monitor restart

and restart any other javaagent (in-process) based SPM Monitors by restarting your server which is being monitored.

...

Code Block
languagebash
sudo vim /etc/security/limits.conf
spmmon     -    nofile    32000

sudo vim /etc/pam.d/su
session    required   pam_limits.so

Restart SPM Monitor after the above changes.

...

My server stopped sending metrics, so why do I still see it under Hosts Filter

A: Filters have 1 day granularity, which means that a server will be listed under Hosts filter until 24 hours since it last sent data have passed.  For example, if a server stopped sending data at 1 PM and if at 8 PM you are looking at the last 6 hours of data (for a period from 2 PM until 8 PM) you will not see data from this server on the graph, but you will still see this server listed under the Hosts filter until 1 PM on the following day.  After 1 PM on the following day this server should disappear from the Hosts filter.

I rebooted my server and now I don't see any data in my graphs. What should I check

A: Here are a few things to check and do:
  1. Make sure SPM Monitor is running: sudo service spm-monitor restart
  2. Make sure disk is not full: df -h
  3. Make sure maximal open files limit was not reached: see "I registered for SPM more than 5 minutes ago and I don't see any of my data, what should I check?"

How come Disk Space Usage report shows more free disk space than df command

A: SPM reports both free and reserved disk space as free, while df does not include reserved disk space by default.

I changed my server's hostname and now I don't see new data in my graphs. What should I do

A: Simply restart SPM Monitor:
Code Block
languagebash
sudo service spm-monitor restart

and restart any process which is using javaagent/in-process version of SPM monitor.

Can I specify which Java runtime the SPM client should use

A: Yes, you can edit the /opt/spm/properties/java.properties file where you can specify the location of Java you want the SPM Monitor to use.

Can SPM client use HTTP instead of HTTPS to send metrics from my servers

A: Yes. Since version 1.22.0, SPM client by default uses HTTPS to send metrics data to Sematext servers. If you prefer to use HTTP instead (for example, if you are running SPM on premises or if you don't need metric data to be encrypted when being sent to SPM over the Internet), you can adjust that in /opt/spm/properties/spm-sender.properties by changing protocol to http in property:

Code Block
spm_sender_receiver_url=https://spm-receiver.sematext.com/receiver/v1

Solr Monitoring

How do I enable JMX in Solr

A: Add or uncomment the <jmx/> directive in solrconfig.xml and restart Solr.  See http://wiki.apache.org/solr/SolrJmx for more info.

I am using SPM for Solr and I don't see any data on Solr and JVM reports, what is the problem

A: You should probably enable JMX in your Solr. Add or uncomment the <jmx /> directive in solrconfig.xml and restart Solr.  See http://wiki.apache.org/solr/SolrJmx for more info.

I am using SPM for Solr and I don't see any data only in Solr Components or Errors reports, what should I do

A: Most likely you are using standalone variant of Solr monitor. In that case, SPM monitor can't collect metrics which are available only when running in-process. If so, switch to in-process (javaagent) version of SPM monitor.

Elasticsearch Monitoring

Why doesn't the number of documents I see in SPM match the number of documents in my Elasticsearch index

A: SPM collects index stats from primary shards only.  To see the total number of documents in an index, select all shards in that index and choose "sum".  The list of shards and the "sum" function can be found in the "Shard filter" in the Index Stats report.

Can SPM collect metrics even when Elasticsearch HTTP API is disabled

A: Each SPM agent collects Elasticsearch metrics only from the local node by accessing the Stats API via HTTP.  To allow only local access add the following to elasticsearch.yml. Don't forget to restart each ES node to whose elasticsearch.yml you add this.
Code Block
http.host: "127.0.0.1"

Can I point SPM monitor to a non-localhost Elasticsearch node

A: Yes.  Adjust /opt/spm/spm-monitor/conf/spm-monitor-config-TOKEN_HERE-default.properties and change the SPM_MONITOR_ES_NODE_HOSTPORT property from the default localhost:9200 value to use an alternative hostname:port.  After that restart SPM monitor (if you are running a standalone version) or Elasticsearch process(es) with embedded SPM monitor. 

My Elasticsearch is protected by basic HTTP authentication, can I use SPM

A: Yes. You just need to adjust /opt/spm/spm-monitor/conf/spm-monitor-config-TOKEN_HERE-default.properties file by adding the following two properties (replace values with your real username and password):
Code Block
SPM_MONITOR_ES_NODE_BASICAUTH_USERNAME=yourUsername
SPM_MONITOR_ES_NODE_BASICAUTH_PASSWORD=yourPassword

Restart your SPM monitor after this change (either with sudo service spm-monitor restart in case of standalone monitor or by restarting Elasticsearch node if you are using in-process javaagent).

 

I am using SPM for Elasticsearch monitor and I don't see Index (and/or Refresh/Flush/Merge) stats, why is that

A: SPM for Elasticsearch monitor collects Index stats only from primary shards, so it is possible that you installed SPM monitor on some Elasticsearch node which hosts only replicas. The same is also true for Refresh/Flush and Merge stats. Also note that SPM Elasticsearch monitor should be installed on all your Elasticsearch nodes to get the complete picture of your cluster in SPM Reports UI.

HBase Monitoring

How do I enable JMX in HBase

A: Please see HBase Metrics page for instructions.

Do I need to add a separate SPM Application for each HBase server/node I want to monitor

A: No, one Application is enough. Think of an SPM "Application" as a "HBase Cluster". Thus, to monitor N HBase servers that belong to the same cluster you would create just a single SPM Application and use its Token in SPM configuration file on all HBase servers that are a part of this cluster.

Why don't some HBase metrics graphs have any data

A: There could be 2 possible reasons:

  1. Some metrics are for RegionServers (HBase slaves), some for HBase Master. Thus, if you select the Master node in the UI, graphs that contain Slave-specific metrics will be blank and vice-versa.
  2. Different versions of HBase provide different metrics. Thus, if you have an older version of HBase, it may not be providing all metrics that SPM collects and graphs.

Which versions of HBase does SPM support

A: SPM has been tested with HBase 0.90, 0.92, 0.94, and 0.98, but will work for newer versions, including all CDH versions.

Kafka Monitoring

Why am I not seeing all Kafka metrics if I'm running a 0.8.x version of Kafka that is pre-0.8.2

A: Kafka had great metrics in 0.7x versions and then in pre-0.8.2 the metrics had a lot of issues.  In short, the MBeans Kafka exposed via JMX were named in a way that made them very difficult/impossible to parse systematically.  We worked with the Kafka developers to fix that in Kafka 0.8.2 (see https://issues.apache.org/jira/browse/KAFKA-1481).  To make your Kafka Producers, Consumers, and Brokers properly monitorable, upgrade to 0.8.2.

How can I see metrics for Kafka Producers and Brokers and Consumers

A: To see metrics for Kafka Producers, Brokers, and Consumers make sure to run SPM client on all of them.  Each of them has its own metrics.  Kafka Brokers do not expose Producers' or Consumers' metrics, so ensure you have SPM client on all three Kafka tiers.

Can I see metrics for non-Scala/JVM Kafka Producers and Consumers

A: No, because none of them seem to expose metrics the way Kafka's own Producer and Consumer implementations expose metrics.

AWS Monitoring

Which IAM permissions are needed to fetch Amazon EC2, EBS and ELB metrics

When you create an AWS app, you need to provide the access key and secret for a user that can fetch metrics for EC2, EBS and/or ELB, depending on which of those you select to be monitored. We recommend creating a separate IAM user for this, with the minimum permissions:
Code Block
languagetext
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeAddresses",
                "ec2:DescribeVolumes",
                "elasticloadbalancing:DescribeLoadBalancers",
                "cloudwatch:GetMetricStatistics"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
The Describe* permissions are needed to identify the resources which need to be monitored (Instances and Addresses for EC2, Volumes for EBS and LoadBalancers for ELB), while GetMetricStatistics will allow SPM to fetch the actual metrics from CloudWatch.

Alerts

Can I send alerts to HipChat, Slack, Nagios, or other WebHooks

A: SPM has integrations for Slack, HipChat, PagerDuty, VictorOps, OpsGenie, BigPanda, general WebHooks, email, and more.  See Integrations

What are Threshold-based Alerts

A: Threshold-based alerts are classic alerts where you choose a metric and a numeric value (a threshold) that, when reached or crossed, triggers an alert.

What is Anomaly Detection

A: Anomaly Detection uses a number of statistical algorithms to determine if a metric has become anomalous, meaning that its recent values have become significantly different from the "base" – the normal/typical values of that metric.  An example of an anomaly is a sudden spike or a sudden dip in a metric value.  SPM can trigger alerts when such anomalies are detected.

What are Heartbeat Alerts

A: Heartbeat Alerts are extremely useful for keeping an eye on your applications.  You create a single Heartbeat Alert for each SPM App.  From then on, SPM will watch all instances of your application that are a part of that SPM App. Should SPM not hear from any one of your application's instances for more than N minutes, it will alert you.  N can be as low as 2 minutes and you choose it yourself when you create a Heartbeat Alert.  

Upgrading

How do I upgrade the SPM client

A: If you have previously installed the SPM client package (RPM, Deb, etc.), simply upgrade via apt-get (Debian, Ubuntu, etc.), or yum (RedHat, CentOS, etc.), or zypper (SuSE).

Debian/Ubuntu:

Code Block
languagebash
wget -O - https://pub-repo.sematext.com/ubuntu/sematext.gpg.key | sudo apt-key add -  # NOTE: this will update the sematext gpg key
sudo apt-get update && sudo apt-get install spm-client  # NOTE: this does not update the whole server, just spm-client

RedHat/CentOS/...

Code Block
sudo wget https://pub-repo.sematext.com/centos/sematext.repo -O /etc/yum.repos.d/sematext.repo  # NOTE: this will update sematext repo file
sudo yum clean all && sudo yum update spm-client        # NOTE: this does not update the whole server, just spm-client

SuSE:

Code Block
sudo zypper up spm-client

After that is done, also do:

  • if you are using SPM monitor in in-process/javaagent mode - restart monitored server (restart your Solr, Elasticsearch, Hadoop node, HBase node... Exceptions: In case of Memcached, Apache and plain Nginx - no need to restart anything; in case of Redis only standalone SPM monitor exists so check below how to restart it)

OR

if you are using standalone SPM monitor, restart it with:
Code Block
sudo service spm-monitor restart

If you used bash script based installer, then just run the installation command again ("curl -ko installer.sh ...." or "wget --no-check-certificate -O installer.sh ..."). There is no need to go through "Client configuration setup" steps since the existing configuration will be preserved.

After that is done, next step is the same as with apt-get/yum/zipper upgrade - you should restart monitored server (if using in-process/javaagent monitor) or standalone SPM monitor.

Note: In case of Memcached, Apache and plain Nginx - after completing upgrade steps described above, you must also run commands described in Step 2 - Client Configuration Setup (which is accessible from https://apps.sematext.com/ui/monitoring, click Actions->Install Monitor for app you have installed)

Note: If you moved SPM client from /opt/spm to different directory, you will first have to move SPM back to original /opt/spm directory:

Code Block
sudo bash /opt/spm/bin/move-spm-home-dir.sh /mnt/some_other_dir/spm /opt/spm

and then proceed with regular update. After update is done, you can again move SPM client to location which suits you.

Uninstalling

How do I uninstall the SPM client

A: On servers where you want to uninstall the client do the following:

  1. remove spm-client, for instance: sudo apt-get purge spm-client   OR   sudo yum remove spm-client
  2. after that, ensure there are no old logs, configs, etc. by running the following command: sudo rm -R /opt/spm
  3. if you used in-process (javaagent) version of monitor, remove "-javaagent" definition from startup parameters of process which was monitored

Note: in case you used installer described on "Other" tab (found on https://apps.sematext.com/ui/monitoring, click Actions->Install Monitor for app your are installing), instead of commands from step 1 run: sudo bash /opt/spm/bin/spm-client-uninstall.sh . After that proceed with steps 2 and 3 described above.

Security

What information are SPM agents sending

A: SPM agents are sending metrics and data used for filtering those metrics, such as hostnames (which can be obfuscated).  To see what exactly is being sent you can use tcpdump or a similar tool to sniff the traffic.  SPM agents ship data via HTTPS, but can also ship it via HTTP.  Java-based agents let you change the protocol in appropriate files under /opt/spm/spm-sender/conf/ directory, while Node.js-based agents let you change it via the SPM_RECEIVER_URL environmental variable.

Billing

How exactly is usage metered for billing purposes

A: Usage is metered hourly per-server-per-app. For example:

If you send metrics from a server A for SPM app Foo between 01:00 and 02:00 that's $0.035 for the Standard plan.
If you send metrics for a different SPM app Bar, even if from the same server A, that's another $0.035.
If you are not sending metric from a server A for SPM app Foo between 02:00 and 03:00 then you pay $0 for that hour.
If you send metrics from a server for that app X 24/7 that ends up being ~ $25/month.  If you do that from 2 servers it ends up being 2 x ~ $25/mo.

How do you bill for monitoring of Docker containers.

A: Docker monitoring is based on the base price and per-container price.  The base price includes monitoring of a Docker host and free monitoring of up to N containers. Per-container price is applied only if you run more than N containers per host.  The number of containers per host is averaged for the whole account.  The base price and the number of containers included in it depends on the plan.  Note that monitoring of Docker host and containers is independent of monitoring of applications you run in those containers.  Containerized applications monitored by SPM are metered as separate hosts. In other words, whether the monitored application is running in a container or in a VM or directly on a server or in a public cloud instance is the same as far as metering and billing is concerned. For plans and price details see https://sematext.com/spm/#plans-and-pricing.

Which credit cards are accepted

A: We accept all major credit cards - Visa, MasterCard, American Express, JCB, Discover, and Diners Club.  We can also invoice you if you want to pay via ACH or bank wires.

Can I be invoiced instead of paying with a credit card

A: Yes, we offer a pre-payment option through invoicing.  When you select this option simply email us and let us know how much you want to pre-pay.  If you want that credit to cover some period of time you'll want to consider how many apps/servers/containers you need to monitor, whether they are running 24/7, and which plan(s) you'd like.  We'll invoice you and, once we receive your payment, we'll enter this amount as credit towards your Sematext account.  This amount will then be burned down, based on your usage, on a monthly basis.  We'll notify you before your balance gets too low, so that we can repeat the invoicing process before your credit runs out.

How often will I get billed

A: We bill on a monthly basis and send an email to notify you of the amount.

Can the billing email be sent to our Accounts Payable/Accounting instead of me

A: Yes, when you select your plan and payment method you will also be able to enter an alternative billing email address.  Also, search this FAQ for information about BILLING_ADMIN role.

Do I have to commit or can I stop using SPM at any time

A: There is no commitment and no contract. When/if you want to stop using SPM you simply stop sending us your metrics and/or switch to the Free plan.

Can I get invoices

A: Yes, just log in and look under Account > Billing >  Invoices.  If you don't see invoices there then you likely need to switch to a different account.  To do that look for a pull-down menu at the top-right of the UI.