Help Docs ~ Server Monitoring

The Scoutd agent supports the StatsD protocol, enabling you to easily view custom metrics within Scout.

This eliminates the need to rollout an additional monitoring stack specifically for StatsD (statsd daemon, graphite, alerting, etc) as you can view those metrics on PSM dashboards and create triggers for alerting inside PSM.

This two-minute video shows how easy it is to get started with StatsD & Scout:

Installing ScoutD

You'll need to install the Scoutd agent in order to collect and report StatsD metrics to PSM. We recommend using our interactive install script to install Scoutd.

Upgrading an existing version of ScoutD

If ScoutD is already installed, you'll want to install the latest version. Versions 0.5.4+ support StatsD.

Ubuntu/Debian:

sudo apt-get update && sudo apt-get install scoutd

Centos/Redhat/Fedora/etc:

sudo yum install scoutd

Enabling and Configuring StatsD

The built-in StatsD server is enabled by default in scoutd versions >= 0.5.10. Refer to Upgrading an existing version of ScoutD to upgrade to the latest scoutd version.

Changing the UDP Address or Port

To change the address or port for the built-in StatsD server:

  • Add to your /etc/scout/scoutd.yml:
statsd:
  addr: 127.0.0.1:8125
  • Restart scoutd:
sudo scoutctl restart

Disabling the StatsD server

To disable the built-in StatsD server:

  • Edit your /etc/scout/scoutd.yml:
statsd:
  enabled: false
  • Restart scoutd:
sudo scoutctl restart

Performance Impact

The performance impact of scoutd is dependent on the number of statsd metrics processed per second. Scoutd's built-in statsd server is equally as performant as the statsd node.js server.

Although scoutd is limited to a single CPU, it can handle over 100k metrics per second on a 2.7GHz CPU.

Setup:

  • System: VMWare virtual machine, Ubuntu 14.04
  • CPU: Utilizing a single core on Intel Dual Core i7 2.7GHz
  • RAM: 2GB
  • 1000 unique statsd metrics
  • One statsd counter metric per UDP packet
  • Each metric name 34 characters long

Results:

  • Total scoutd RAM usage: 23MB
  • 95% CPU usage
  • ~100k statsd events processed per second

Supported Metric Types

Counters

Counters are used to measure the frequency of an event per minute, like page views or failed login attempts. You may specify the amount to increment the counter. Counters reset to 0 every minute.

statsd.increment("logins.failed")
statsd.increment("jobs_processed", 20)

Gauges

Gauges are used to measure things at a specific time, like the amount of memory used or number of users logged in. Once a gauge is set, its value does not change until you set it again.

statsd.gauge("users.online", 512)

Timers

Timers are used to measure the duration of a task, like database calls or render times.

statsd.time("video_uploads.render") do
  # process video
end

When a timer is used, 8 total metrics are reported:

  • [METRIC].mean
    • mean of all values
  • [METRIC].max
    • maximum of all values
  • [METRIC].min
    • minimum of all values
  • [METRIC].sum
    • sum of all values
  • [METRIC].count
    • count of all values reported over the one-minute reporting interval (see Counters above).
  • [METRIC].mean_95
    • mean of values below the 95th percentile
  • [METRIC].sum_95
    • sum of values below the 95th percentile
  • [METRIC].upper_95
    • the value at the 95th percentile

Migrating to PSM's StatsD support from standalone StatsD

We suggest removing hostname identifiers from metric names (ex: app1.response.count => response.count) as Scout automatically tracks the same metric name separately across servers.

Removing the hostname facilities easier dashboard + trigger creation.

Metrics across Servers

First, remove hostname identifiers from metric names (ex: app1.response.count => response.count). Depending on your use case, two options are available:

  1. If a trigger is not needed, to view the sum of a metric from multiple servers on your dashboard, set the chart display to Stacked.

  2. Choose a scoutd server to be a central aggregator and set scoutd to listen on an address reachable by your other servers. Define a second StatsD endpoint within your code that sends metrics to this central aggregator. For example:

STATSD = Statsd.new 'localhost', 8125
STATSD_CENTRAL = Statsd.new 'central-aggregator-hostname', 8125

def generate_metric
  # do something
  STATSD.increment("metric")
  STATSD_CENTRAL.increment("metric")
end

Rate Limits

The following rate limits are enforced for StatsD metrics:

  • 500 metrics per-server
  • 5,000 metrics per-account

Note that using a StatsD timer results in 8 distinct metrics.

Known Issues

  • Realtime is not yet supported for custom metrics

StatsD