Day 6 - Setting up monitoring

Published by Michael Banzon on Sat Feb 18, 2017

In this part we’ll set up monitoring of the currency conversion service we’ve made and the server it is running on. We will use Datadog - you can read a previous (more generic) article on setting up monitoring with Datadog.

The previous article described how to setup monitoring to check if your server is running and to keep an eye on resources and resource consumption.

In the following we’ll assume that something similar is up and running and is not needed.

What monitoring can we add?

Now that we have a concrete server we can actually start to monitor something about it.

Since it is writted in Go we can use expvar package that’ll give is the ability to export some variables from inside Go and give an interface to access them from Datadog.

Datadog previously wrote a blog post about it - it basically covers everything we/you need to know.

We’ll add the following metrics:

  • currency_hits a counter for the hits to the /currencies endpoint - both the GET and POST are counted as one
  • conversion_hits a counter for the hits to the /convert endpoint
  • webhook_hits a counter for the hits to (succesfully) register a webhook
  • webhook_triggers a counter for the times webhooks are called by the server

Starting

We start by going to the /etc/dd-agent/conf.d/ and renaming go_expvar.yaml.example to just go_expvar.yaml.

Then we’ll open edit the file to contain only the following:

init_config:

instances:
  - expvar_url: http://localhost/debug/vars
    metrics:
      - path: currency_hits
      - path: convert_hits
      - path: webhook_hits
      - path: webhook_triggers

That’s it!

Restart the agent:

service datadog-agent restart

And you’re running!

Changes in the code

Check out the new version of the server in the repository. It adds some expvar variables in to code and performs Add(1)’s to them on the requests.

Setting it up in Datadog

When we go to our dashboard in Datadog we can add new metrics from expvar.

It’ll look like this:

expvar in Datadog

Now Datadog let’s us set up dashboard elements for the four metrics we added in the code along with the standard metrics from expvar like memory allocated eg.

What’s next?

Hopefully this gives you some ideas on how to use tools like Datadog to keep an eye on your servers and services. Next we’ll need to put this currency conversion tool into the hands of some users.

Continue to day seven…