How to monitor Redis (or anything else running on your Linux machine) and get a notification when needed?

With Trafikito you can monitor the output of any command and execute any API call when needed. A default notification template is used to trigger Trafikito endpoint which will send you an email but also you can make an HTTP call.

That sounds interesting, isn’t it? Think about it for a minute. Every tool you use databases, servers, all kind of software usually come with command line tools that can provide statistical information. Starting from the Linux itself with uptime, free -m, df, top…. ending with things like Redis.

What are you going to learn?

After this article, you will be able to monitor the output of any command available on your Linux machine. As an example, I will use a sample Redis instance and will monitor the memory usage.

I will show you how you could have a chart showing memory usage of your Redis instance for free.

You will also learn how to set up notifications to get an email when Redis memory usage is in an error range. Also, you will be able to do the same thing for anything you want to monitor on your Linux machine.

Chart showing Redis memory usage over time

Most useful Redis monitoring command — INFO

INFO command provides a lot of information and statistics about your Redis instance. As Redis documentation describes, these sections of information are available:

• server: General information about the Redis server
• clients: Client connections section
• memory: Memory consumption related information
• persistence: RDB and AOF related information
• stats: General statistics
• replication: Master/replica replication information
• cpu: CPU consumption statistics
• commandstats: Redis command statistics
• cluster: Redis Cluster section
• keyspace: Database related statistics

For example, to get memory usage, you would use redis-cli info memory. Then, you would get something like this:

Sample output of redis-cli info memory command

Let’s monitor some Redis memory stats

redis-cli info memory returns a lot of output. While Trafikito usage plan is counting the amount of bytes you are sending to Trafikito, let’s reduce the output just to fragments we actually need.

I want to monitor used_memory and used_memory_rss. I’ll use awk which is almost always available for any Linux environment to filter the output.

sudo redis-cli info memory | awk '/used_memory(_rss)?:/'

The output of just what we need

Make new command available for Trafikito

This is 4 easy steps process:

1. Add new command at your server to available_commands.sh file
2. Wait a minute for the next data cycle sent to Trafikito from your server
3. Add new data source with the new command
4. Wait again a minute to get sample output for the command with the next data cycle

Adding a new command to the available_commands.sh file

Open /opt/trafikito/available_commands.sh for a quick edit

sudo nano /opt/trafikito/available_commands.sh

Scroll all the way down and add a new line:

trafikito_redis_memory="sudo redis-cli info memory | awk '/used_memory(_rss)?:/'"

It should look like this now:

Added a new command to the list of all available commands

Save the file and close. With nano, it would be ctrl+x then enter “y” and click Enter/Return.

Add a new data source with the new command

Trafikito

On servers view open your server

Open the server

Click “Settings”

Add new data source

Make sure that the Data source is selected and select a new command from the list and click Add new data source to save.

If the command is not available — wait a minute and refresh the page. This list is refreshed after the new data comes from your server to the Trafikito dashboard.

Adding new data source

Open data sources

Find your new data source and click to expand.

View the last output

Parse the output and set variables

The default delimiter to split output to rows is a new line symbol \n. So, you can leave the delimiter to rows field empty.

The default delimiter to split rows to columns is [ ,]+ which means that to split by one or more spaces or commas. However, Redis return values without any spaces but it is separated by a colon :

Update the delimiter to split rows to columns and replace it with a colon for both rows and you will notice now that you have a beautiful table of cells. On cell B1 you have the current used_memory value and on cell B2 it’s used_memory_rss

Rows split to columns by color symbol

Add variables to track both values

1. Click Add Variable twice to add 2 variables. I called them redis_used_memory and redis_used_memory_rss.

2. Enter some meaningful names. It will be visible in settings while editing the charts or the other dashboard elements.

3. Select the source cell

4. Select the display type. Though these are bytes and it would be nice to have it displayed in KB, MB, GB, TB select the Base type.

Each variable has some options. The options depend on the display type. All numeric values may have some ranges defined as warn and error.

Click Options next to the variable to change the warn and error ranges. This is the default view for both variables:

Default warn and error ranges for new variables

You can see that the input label is already showing the preview in the selected display type.

For both redis_used_memory and redis_used_memory_rss let’s set the warn range from 10–50MB and the error range anything above 50MB. These are small values for production servers but in this case, this is an unused database and it’s just to show how things work. You should set these values based on your case.

Sample warn and error ranges. You can add as many ranges as needed. Set the values based on your case.

Save data source

Save data source

Add a new chart to the dashboard which would display these two values over time

Close the settings and enter edit mode on your dashboard

Edit mode switch

Add a new dashboard box by clicking the plus button. It will appear on the bottom of the dashboard. Scroll down to find it. You can drag it to the top if needed. You can resize all the boxes, drag around and change it’s content.

Add a new box to the dashboard

I dragged the new box just below the arc charts and resized it to it’s full width. Click the pencil icon on this empty box and view the chart settings.

Dragged the new box just below the arc charts and resized it to it’s full width

Add a couple of series to the chart

Select your new variables redis_used_memory and redis_used_memory_rss. Also, you can update the color or name displayed on the chart.

Adding new series

Click Save to save and close this dialog.

Turn edit mode off

Your beautiful chart with the memory usage of your Redis instance

Now you have your new beautiful chart which displays the memory usage of your Redis instance over time.

You can hover over it to see the details and do all other actions that are available for any time-series chart. Explore the setting options for adventure.

Chart showing Redis memory usage over time

How about notifications?

You can think about Trafikito as software with input and output. Inputs are data sources which you already configured. Outputs are visuals on the dashboard and notifications.

Each notification has when cases defining when to trigger the notification and API call configuration which is made of HTTP method, URL to call, headers and body.

What is an API call?

API stands for Application Programming Interface. Most of the software is based on APIs. As a user, you have a graphical interface e.g. to use Facebook. As a developer, you can make many actions that can be done with a graphical interface automatically by using APIs without any user interaction.

API calls may do many things, some samples:

1. Post a message to Facebook, Slack, Twitter…
2. Restart a server (ask your hosting provider if they have API documentation)
3. Send SMS. E.g. with Twillio.com
4. Send an email (default action at Trafikito). If you want to use this, just click “Click here to use email template” at notification settings.
5. Turn on the light in your living room
6. ???

A good example of what API calls can do are products as IFTTT and Zapier

Default notifications

After a fresh installation, you will find 4 notifications.

1. Average load. Uses load_1 variable from a data source with uptime command. load_1 maps to average load during the last minute.
2. Disk space available. Uses disk_space_primary_available. Trafikito tries to guess which disk is primary by disk space. Highest is set as primary. Data source which has disk_space_primary_available uses df -P command.
3. RAM available. Uses free -m command to get RAM usage information. Some Linux distributions do not provide RAM available numbers so it falls back to RAM free.
4. Uptime. Different type of notification which checks if your server is sending data to Trafikito. Trafikito has no way to call your server. The only connection with Trafikito is outgoing from your server.

Overview of default notifications.

Click the round yellow button on the bottom right-hand corner with the plus sign to add a new notification.

Notification at Trafikito.com settings

Find your new notification and click it to view the settings.

Click on the notification to view all the settings

New notification settings overview

These are when cases that were added for all the possible range changes.

Updated the body to include all kinds of useful information.

Redis used memory: {{redis_used_memory}}
Redis used memory rss: {{redis_used_memory_rss}}
Average load: {{load_1}}, {{load_5}}, {{load_15}}

At Trafikito you can use variables in all places where you can set values. E.g. here in the notification view — you can use variables in URL, Header or body.

In the charts, you can use variables inside settings as “maximum value for Y-axis” or title of the chart.

Updated action body to include useful information

Save notification and click Test. You will get the notification email. You can click the Test button several times to test it again.

Test results

You can always click the Show logs button to see the last 15 calls executed during the last 7 days. Each log has a full payload and result. It is useful while debugging.

After a couple seconds you should see similar an email in your inbox:

Notification email after testing

We had an exciting journey already!

1. Added a new command to /opt/trafikito/available_commands.sh
2. Added a new data source at Trafikito dashboard
3. Parsed an output of redis-cli info memory command and created two variables: redis_used_memory and redis_used_memory_rss
4. Set the warn range to 10–50MB and the error to 50MB+ for both variables
5. Added a new notification which sends emails and includes useful information inside the body of the message.

Now let’s run a test. I will use redis-benchmark command to create an artificial load for some time. This will run SET and GET commands many times against the server with ~0.5Kb of data.

redis-benchmark -d 500000 -t set,get -n 9999999

Creating some load on Redis

After a few minutes, I got an email from Trafikito with all the details.

1. The title is a great summary. The variable name, current range, how long it is in this range, what is the server name and what is the current value of the variable at the moment.
2. The message itself clearly shows that there is an issue. It shows all the details from the title just in a more beautiful form plus any custom body with current values.
3. The message has a couple of buttons to jump straight to the dashboard or notification settings to view more details or edit the settings.

The start of a new beginning

Thank you for following along. Tell me what you think in the comments section and give it a try by yourself for free at Trafikito.com

This is an exciting journey for me and, hopefully, for you too.

https://trafikito.com/