Versions Compared

Old Version 174

changes.mady.by.user Otis Gospodnetić

Saved on

compared with

New Version 175

changes.mady.by.user Otis Gospodnetić

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

...


Code Block
spm_sender_proxy_host=
spm_sender_proxy_port=
spm_sender_proxy_user_name=
spm_sender_proxy_password=


When installing SPM client, I see "The certificate of `pub-repo.sematext.com' is not trusted" or similar error.  How can I avoid it

A: There can be multiple reasons for this, most likely:

  • system time on your machine is not correct and adjusting it should fix the problem
  • you are installing SPM client as root user - in some cases that can cause this error and installing SPM client as a different user may help

  • you are missing the ca-certificates package.  Use one of the following commands to install it:

Code Block
sudo apt-get install ca-certificates
sudo yum install ca-certificates

If none of the above eliminates the problem for you try adding a flag to avoid certificate checking. In case the command with wget is failing in your case, add "--no-check-certificate" as wget argument. If command with curl is failing, add "-k" flag.

How do I create the diagnostics package

A: In case you are having issues with SPM, you can create diagnostics package on affected machines where SPM client was installed by running:

Code Block
sudo bash /opt/spm/bin/spm-client-diagnostics.sh

The resulting package will contain all relevant info needed for our investigation. You can send it, along with short description of your problem, to spm-support@sematext.com or contact us in chat.

  • I see only my system metrics (e.g. CPU, Memory, Network, Disk...), but where is the rest of my data

A: Make sure you have followed all steps listed on installation instructions page. Package installation steps should be done first, followed by Client configuration setup. If you have done that and you still don't see application metrics, run sudo bash /opt/spm/bin/spm-client-diagnostics.sh to generate diagnostics package and send it to spm-support@sematext.com with description of your problem.

  • I DO NOT see any system metrics (e.g. CPU, Memory, Network, Disk), what could be the problem

A: Make sure you have followed all steps listed on installation instructions page. It is possible you missed Client configuration setup step. If you have done that and you still don't see application metrics, run sudo bash /opt/spm/bin/spm-client-diagnostics.sh to generate diagnostics package and send it to spm-support@sematext.com with description of your problem.

  • I am using SPM Solr or SPM ES. My Request Rate and Latency charts are empty

A: If other Solr/ES charts are showing the data, it is most likely that there were no requests sent to your Solr/ES in the time range you are looking at. Try sending some queries and see if request rate/latency charts will show them. If they don't, please send us an email to spm-support@sematext.com or contact us in chat.

  • I registered for SPM more than 5 minutes ago and I don't see ANY of my data / I am getting errors when starting SPM Monitor or when starting my service which uses javaagent version of SPM monitor, what should I check

A: Here are a few things to check and do:
  • Log into your monitored servers and make sure there are running SPM monitor processes (there should be more than one of them)
  • Check if system time is correct. If not, you should adjust the time, restart the SPM monitor with:
    sudo service spm-monitor restart

    and restart any other javaagent (in-process) based SPM Monitors by restarting your server which is being monitored.

  • Make sure disks are not full.

  • Make sure user spmmon can have more than 1024 files open:

    Code Block
    languagebash
    sudo vim /etc/security/limits.conf
spmmon     -    nofile    32000

sudo vim /etc/pam.d/su
session    required   pam_limits.so

    Restart SPM Monitor after the above changes.

  • Check if hostname of your server is defined in /etc/hosts
  • If you are starting your Jetty (or some other server) with command like "java ... -jar start.jar ..." and using inprocess (javaagent) version of monitor, make sure -D and -javaagent definitions occur before "-jar start.jar" part in your command
  • If none of the suggestions helped, run sudo bash /opt/spm/bin/spm-client-diagnostics.sh to generate diagnostics package and send it to spm-support@sematext.com

  • My server stopped sending metrics, so why do I still see it under Hosts Filter

A: Filters have 1 day granularity, which means that a server will be listed under Hosts filter until 24 hours since it last sent data have passed.  For example, if a server stopped sending data at 1 PM and if at 8 PM you are looking at the last 6 hours of data (for a period from 2 PM until 8 PM) you will not see data from this server on the graph, but you will still see this server listed under the Hosts filter until 1 PM on the following day.  After 1 PM on the following day this server should disappear from the Hosts filter.

  • I rebooted my server and now I don't see any data in my graphs. What should I check

A: Here are a few things to check and do:


  • How come Disk Space Usage report shows more free disk space than df command

A: SPM reports both free and reserved disk space as free, while df does not include reserved disk space by default.


  • I changed my server's hostname and now I don't see new data in my graphs. What should I do

A: Simply restart SPM Monitor:


Code Block
languagebash
sudo service spm-monitor restart

and restart any process which is using javaagent/in-process version of SPM monitor.


  • Can I specify which Java runtime the SPM client should use

A: Yes, you can edit the /opt/spm/properties/java.properties file where you can specify the location of Java you want the SPM Monitor to use.

  • Can SPM client use HTTP instead of HTTPS to send metrics from my servers

A: Yes. Since version 1.22.0, SPM client by default uses HTTPS to send metrics data to Sematext servers. If you prefer to use HTTP instead (for example, if you are running SPM on premises or if you don't need metric data to be encrypted when being sent to SPM over the Internet), you can adjust that in /opt/spm/properties/spm-sender.properties by changing protocol to http in property:

Code Block
spm_sender_receiver_url=https://spm-receiver.sematext.com/receiver/v1


  • Solr Monitoring

  • How do I enable JMX in Solr

A: Add or uncomment the <jmx/> directive in solrconfig.xml and restart Solr.  See http://wiki.apache.org/solr/SolrJmx for more info.

  • I am using SPM for Solr and I don't see any data on Solr and JVM reports, what is the problem

A: You should probably enable JMX in your Solr. Add or uncomment the <jmx /> directive in solrconfig.xml and restart Solr.  See http://wiki.apache.org/solr/SolrJmx for more info.

  • I am using SPM for Solr and I don't see any data only in Solr Components or Errors reports, what should I do

A: Most likely you are using standalone variant of Solr monitor. In that case, SPM monitor can't collect metrics which are available only when running in-process. If so, switch to in-process (javaagent) version of SPM monitor.

  • Elasticsearch Monitoring

  • Why doesn't the number of documents I see in SPM match the number of documents in my Elasticsearch index

A: SPM collects index stats from primary shards only.  To see the total number of documents in an index, select all shards in that index and choose "sum".  The list of shards and the "sum" function can be found in the "Shard filter" in the Index Stats report.

  • Can SPM collect metrics even when Elasticsearch HTTP API is disabled

A: Each SPM agent collects Elasticsearch metrics only from the local node by accessing the Stats API via HTTP.  To allow only local access add the following to elasticsearch.yml. Don't forget to restart each ES node to whose elasticsearch.yml you add this.
Code Block
http.host: "127.0.0.1"

  • Can I point SPM monitor to a non-localhost Elasticsearch node

A: Yes.  Adjust /opt/spm/spm-monitor/conf/spm-monitor-config-TOKEN_HERE-default.properties and change the SPM_MONITOR_ES_NODE_HOSTPORT property from the default localhost:9200 value to use an alternative hostname:port.  After that restart SPM monitor (if you are running a standalone version) or Elasticsearch process(es) with embedded SPM monitor. 

  • My Elasticsearch is protected by basic HTTP authentication, can I use SPM

A: Yes. You just need to adjust /opt/spm/spm-monitor/conf/spm-monitor-config-TOKEN_HERE-default.properties file by adding the following two properties (replace values with your real username and password):


Code Block
SPM_MONITOR_ES_NODE_BASICAUTH_USERNAME=yourUsername
SPM_MONITOR_ES_NODE_BASICAUTH_PASSWORD=yourPassword

Restart your SPM monitor after this change (either with sudo service spm-monitor restart in case of standalone monitor or by restarting Elasticsearch node if you are using in-process javaagent).

 

  • I am using SPM for Elasticsearch monitor and I don't see Index (and/or Refresh/Flush/Merge) stats, why is that

A: SPM for Elasticsearch monitor collects Index stats only from primary shards, so it is possible that you installed SPM monitor on some Elasticsearch node which hosts only replicas. The same is also true for Refresh/Flush and Merge stats. Also note that SPM Elasticsearch monitor should be installed on all your Elasticsearch nodes to get the complete picture of your cluster in SPM Reports UI.

  • HBase Monitoring

  • How do I enable JMX in HBase

A: Please see HBase Metrics page for instructions.

  • Do I need to add a separate SPM Application for each HBase server/node I want to monitor

A: No, one Application is enough. Think of an SPM "Application" as a "HBase Cluster". Thus, to monitor N HBase servers that belong to the same cluster you would create just a single SPM Application and use its Token in SPM configuration file on all HBase servers that are a part of this cluster.

  • Why don't some HBase metrics graphs have any data

A: There could be 2 possible reasons:

  • Some metrics are for RegionServers (HBase slaves), some for HBase Master. Thus, if you select the Master node in the UI, graphs that contain Slave-specific metrics will be blank and vice-versa.
  • Different versions of HBase provide different metrics. Thus, if you have an older version of HBase, it may not be providing all metrics that SPM collects and graphs.

  • Which versions of HBase does SPM support

A: SPM has been tested with HBase 0.90, 0.92, 0.94, and 0.98, but will work for newer versions, including all CDH versions.

  • Kafka Monitoring

  • Why am I not seeing all Kafka metrics if I'm running a 0.8.x version of Kafka that is pre-0.8.2

A: Kafka had great metrics in 0.7x versions and then in pre-0.8.2 the metrics had a lot of issues.  In short, the MBeans Kafka exposed via JMX were named in a way that made them very difficult/impossible to parse systematically.  We worked with the Kafka developers to fix that in Kafka 0.8.2 (see https://issues.apache.org/jira/browse/KAFKA-1481).  To make your Kafka Producers, Consumers, and Brokers properly monitorable, upgrade to 0.8.2.

  • How can I see metrics for Kafka Producers and Brokers and Consumers

A: To see metrics for Kafka Producers, Brokers, and Consumers make sure to run SPM client on all of them.  Each of them has its own metrics.  Kafka Brokers do not expose Producers' or Consumers' metrics, so ensure you have SPM client on all three Kafka tiers.

  • Can I see metrics for non-Scala/JVM Kafka Producers and Consumers

A: No, because none of them seem to expose metrics the way Kafka's own Producer and Consumer implementations expose metrics.
 

  • Alerts

  • Can I send alerts to HipChat, Slack, Nagios, or other WebHooks

A: SPM let's one configure arbitrary Webhooks to call with Alert event information when Alert events are triggered. The details for HipChat integration are described here. 

  • What are Threshold-based Alerts

A: Threshold-based alerts are classic alerts where you choose a metric and a numeric value (a threshold) that, when reached or crossed, triggers an alert.

  • What is Anomaly Detection

A: Anomaly Detection uses a number of statistical algorithms to determine if a metric has become anomalous, meaning that its recent values have become significantly different from the "base" – the normal/typical values of that metric.  An example of an anomaly is a sudden spike or a sudden dip in a metric value.  SPM can trigger alerts when such anomalies are detected.

  • What are Heartbeat Alerts

A: Heartbeat Alerts are extremely useful for keeping an eye on your applications.  You create a single Heartbeat Alert for each SPM App.  From then on, SPM will watch all instances of your application that are a part of that SPM App. Should SPM not hear from any one of your application's instances for more than N minutes, it will alert you.  N can be as low as 2 minutes and you choose it yourself when you create a Heartbeat Alert.  


  • Upgrading

  • How do I upgrade the SPM client

A: If you have previously installed the SPM client package (RPM, Deb, etc.) as described on https://apps.sematext.com/spm-reports/client.do simply upgrade via apt-get (Debian, Ubuntu, etc.), or yum (RedHat, CentOS, etc.), or zypper (SuSE).

Debian/Ubuntu:

Code Block
languagebash
wget -O - https://pub-repo.sematext.com/ubuntu/sematext.gpg.key | sudo apt-key add -  # NOTE: this will update the sematext gpg key
sudo apt-get update && sudo apt-get install spm-client  # NOTE: this does not update the whole server, just spm-client


RedHat/CentOS/...

Code Block
sudo wget https://pub-repo.sematext.com/centos/sematext.repo -O /etc/yum.repos.d/sematext.repo  # NOTE: this will update sematext repo file
sudo yum clean all && sudo yum update spm-client        # NOTE: this does not update the whole server, just spm-client


SuSE:

Code Block
sudo zypper up spm-client



After that is done, also do:

  • if you are using SPM monitor in in-process/javaagent mode - restart monitored server (restart your Solr, Elasticsearch, Hadoop node, HBase node... Exceptions: In case of Memcached, Apache and plain Nginx - no need to restart anything; in case of Redis only standalone SPM monitor exists so check below how to restart it)

OR

  • if you are using standalone SPM monitor, restart it with:

Code Block
sudo service spm-monitor restart


If you used bash script based installer, then just run the installation command again ("curl -ko installer.sh ...." or "wget --no-check-certificate -O installer.sh ..."). There is no need to go through "Client configuration setup" steps from https://apps.sematext.com/spm-reports/client.do since the existing configuration will be preserved.

After that is done, next step is the same as with apt-get/yum/zipper upgrade - you should restart monitored server (if using in-process/javaagent monitor) or standalone SPM monitor.


Note: In case of Memcached, Apache and plain Nginx - after completing upgrade steps described above, you must also run commands described in Step 2 - Client Configuration Setup from https://apps.sematext.com/spm-reports/client.do


Note: If you moved SPM client from /opt/spm to different directory, you will first have to move SPM back to original /opt/spm directory:

Code Block
sudo bash /opt/spm/bin/move-spm-home-dir.sh /mnt/some_other_dir/spm /opt/spm

and then proceed with regular update. After update is done, you can again move SPM client to location which suits you.

  • Uninstalling

  • How do I uninstall the SPM client

A: On servers where you want to uninstall the client do the following:

  • remove spm-client, for instance: sudo apt-get purge spm-client   OR   sudo yum remove spm-client
  • after that, ensure there are no old logs, configs, etc. by running the following command: sudo rm -R /opt/spm
  • if you used in-process (javaagent) version of monitor, remove "-javaagent" definition from startup parameters of process which was monitored


Note: in case you used installer described on "Other" tab on http://apps.sematext.com/spm-reports/client.do , instead of commands from step 1 run: sudo bash /opt/spm/bin/spm-client-uninstall.sh . After that proceed with steps 2 and 3 described above.

  • Security

  • What information are SPM agents sending

A: SPM agents are sending metrics and data used for filtering those metrics, such as hostnames (which can be obfuscated).  To see what exactly is being sent you can use tcpdump or a similar tool to sniff the traffic.  SPM agents ship data via HTTPS, but can also ship it via HTTP.  Java-based agents let you change the protocol in appropriate files under /opt/spm/spm-sender/conf/ directory, while Node.js-based agents let you change it via the SPM_RECEIVER_URL environmental variable.

  • Billing

  • How exactly is usage metered for billing purposes

A: Usage is metered hourly per-server-per-app. For example:

If you send metrics from a server A for SPM app Foo between 01:00 and 02:00 that's $0.035 for the Standard plan.
If you send metrics for a different SPM app Bar, even if from the same server A, that's another $0.035.
If you are not sending metric from a server A for SPM app Foo between 02:00 and 03:00 then you pay $0 for that hour.
If you send metrics from a server for that app X 24/7 that ends up being ~ $25/month.  If you do that from 2 servers it ends up being 2 x ~ $25/mo.

  • How do you bill for Docker containers.  If I run 10 containers on 1 server is that billed like 10 servers or like 1 server

A: This would be billed like just 1 server.

  • Which credit cards are accepted

A: We accept all major credit cards - Visa, MasterCard, American Express, JCB, Discover, and Diners Club.  We can also invoice you if you want to pay via ACH or bank wires.

  • Can I be invoiced instead of paying with a credit card

A: Yes, we offer a pre-payment option through invoicing.  When you select this option simply email us and let us know how much you want to pre-pay.  If you want that credit to cover some period of time you'll want to consider how many apps/servers/containers you need to monitor, whether they are running 24/7, and which plan(s) you'd like.  We'll invoice you and, once we receive your payment, we'll enter this amount as credit towards your Sematext account.  This amount will then be burned down, based on your usage, on a monthly basis.  We'll notify you before your balance gets too low, so that we can repeat the invoicing process before your credit runs out.

  • How often will I get billed

A: We bill on a monthly basis and send an email to notify you of the amount.

  • Can the billing email be sent to our Accounts Payable/Accounting instead of me

A: Yes, when you select your plan and payment method you will also be able to enter an alternative billing email address.  Also, search this FAQ for information about BILLING_ADMIN role.

  • Do I have to commit or can I stop using SPM at any time

A: There is no commitment and no contract. When/if you want to stop using SPM you simply stop sending us your metrics and/or switch to the Free plan.

  • Can I get invoices

A: Yes, just log in and look under Account > Billing >  Invoices.  If you don't see invoices there then you likely need to switch to a different account.  To do that look for a pull-down menu at the top-right of the UI.