Collecting Web Server Metrics

- 6 mins

Metrics are incredibly important to collect and understand. Not only will they be there to support you when “shit hits the fan” but also when you want to improve the resiliency of your service. They also play an important role in capacity planning, trending, continuous intergration/continuous deployment and performance tuning. Etsy talks more about the importance of collecting metrics here, measure anything, measure everything.

In this post we are going to examine how to collect metrics for two popular web servers, Nginx and Apache.

Let’s get started…

Nginx

Installation

The ‘HttpStubStatusModule’ is not compiled into Nginx by default. To check if it is, run

[[email protected] ~]# /usr/sbin/nginx -V
nginx version: nginx/1.2.5
configure arguments:  **--with-http_stub_status_module**

Here we can see the version of Nginx I’m using in AWS from the ‘amzn-updates’ repository comes with it compiled in. If you didn’t see the output above and are compiling from source, ensure the following argument is set --with-http_stub_status_module

More information about the module can be found here.

Configuration

The configuration is simple for the Nginx status module. Add the code below to a server block, replacing “YOUR_IP” with the IP address of your computer. This will only allow connections via localhost and your computer to the status page. Then reload nginx /etc/init.d/nginx reload

server {
    server_name www.drewl.org;
    ...

    location /nginx_status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        allow YOUR_IP;
        deny all;
    }
}

Results

Either with curl or a web browser go to ‘http://YOUR_SITE.com/nginx_status’

curl http://YOUR_SITE.com/nginx_status

Active connections: 59
server accepts handled requests
 11939 11939 17784
Reading: 0 Writing: 1 Waiting: 58

The output can be interpreted as…

Apache

Installation

The mod_status module is usually shipped by default when you install Apache via the package provider of your OS’s distribution but lets check…

[[email protected] ~]# httpd -V
Server version: Apache/2.2.15 (Unix)
Server built:   Aug 13 2013 17:29:28

[[email protected] ~]# httpd -M | grep status
[Wed Dec 18 00:01:27 2013] [warn] module wsgi_module is already loaded, skipping
 status_module (shared)

If mod_status is not available, when you compile it from source ensure the --enable-status argument is present.

Assuming the module is available let’s get started. Before copying and pasting this, review your httpd.conf, as it might already have these options enabled. If not, let’s add the Location directive below.

Replace “YOUR_IP” with the IP address of your computer. Once completed restart apache but before restarting it’s always a good idea to test the config (/etc/init.d/httpd configtest). This Location directive can be moved to a Virtualhost instead of the global httpd.conf if desired for security reasons.

vi /etc/httpd/conf/httpd.conf

<IfModule !status_module>
    LoadModule status_module modules/mod_status.so
</IfModule>

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
    Allow from YOUR_IP
</Location>

Results

With a web browser go to “http://YOUR_SITE.com/server-status?refresh=5”. This will refresh the status page every 5 seconds.

apache status

This shows…

With curl add ‘?auto’ to the end of the URI and this will produce a machine friendly view of the information we saw before.

curl http://YOUR_SITE/server-status?auto

BusyWorkers: 1
IdleWorkers: 74
Scoreboard: W________________________..RRR....

ExtendedStatus

The ExtendedStatus directive shows even more information in regards to the requests being processed. But be cautious, according to the docs the web server can slow down, how much I don’t know yet.

In additional to what we saw above we also get…

Automatically with a curl

curl http://YOUR_SITE/server-status?auto

Total Accesses: 28
Total kBytes: 23
CPULoad: .166667
Uptime: 18
ReqPerSec: 1.55556
BytesPerSec: 1308.44
BytesPerReq: 841.143
BusyWorkers: 2
IdleWorkers: 73
Scoreboard: W________________________..RRR....

Manually via a web browser

apache extended status

This can be helpful if you need to trace down user requests that are causing Apache to use a lot of CPU resources.

Srv    Child Server number - generation
PID    OS process ID
Acc    Number of accesses this connection / this child / this slot
M      Mode of operation
CPU    CPU usage, number of seconds
SS     Seconds since beginning of most recent request
Req    Milliseconds required to process most recent request
Conn   Kilobytes transferred this connection
Child  Megabytes transferred this child
Slot   Total megabytes transferred this slot

Scoreboard

The scoreboard shown above is what’s used by programs such as Monit to gather metrics to determine the health of Apache.

The scoreboard can be decoded as…

Things to look for

Collecting

Now that we have both web servers configured to query for metrics its time to start collecting.

There are a couple of tools to use at our disposable.

I’m going to be using a tool called Riemann for collecting the metrics. I’ll leave the setup, configuration and Graphite integration for another post as this one is already lengthy.

Assuming you have Riemann setup along with Riemann Tools we are ready to start collecting the metrics.

Nginx

Let’s start collecting the Nginx metrics

riemann-nginx-status --host 162.243.251.204 -u http://127.0.0.1/nginx_status --event-host drewl.org


apache extended status

Apache

and the Apache metrics

riemann-apache-status --host 162.243.251.204 -u http://127.0.0.1/server-status --event-host graphite.org


apache extended status

and that’s all folk’s, hopefully this will prove useful to help you start collecting and measuring all the things!

More Areas to Explore



Thoughts or comments? Please share below…


comments powered by Disqus
rss facebook twitter github youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora