| Title: Using Netdata on NixOS and connecting to Netdata cloud
Author: Solène
Date: 16 September 2022
Tags: nixos monitoring netdata cloud
Description: In this article, you will learn about Netdata monitoring
tool, how to use it on NixOS and how to connect it to the free Netdata
privacy friendly cloud solution
# Introduction
I'm still playing with monitoring programs, and I've been remembered
about Netdata. What an improvement over the last 8 years!
This tutorial explains how to get Netdata installed on NixOS, and how
to register your node in Netdata cloud.
|
| Netdata GitHub project page |
| Netdata live demo |
|
# What's Netdata?
This program is a simple service to run on a computer, it will
automatically gather a ton of metrics and make them easily available
over the local TCP port 19999. You just need to run Netdata and
nothing else, and you will have every metrics you can imagine from your
computer, and some explanations for each of them!
That's pretty cool because Netdata is very efficient, it draws nearly
no CPU while gathering a few thousands metrics every few seconds, and
is memory efficient and can be constrained to a dozen of megabytes.
While you can export its metrics to something like graphite or
Prometheus, you lose the nice display which is absolutely a blast
compare to Grafana (in my opinion).
Update: as pointed out by a reader (thanks!), it's possible to connect
Netdata instances to only one used for viewing metrics. I'll
investigate this soon.
|
| Netdata documentation about streaming. |
|
Netdata also added some machine learning anomaly detection, it's simple
and doesn't use many resources or require a GPU, it only builds
statistical models to be able to report if some metrics have an unusual
trend. It takes some time to gather enough data, and after a few days
it's starting to work.
# Installing Netdata on NixOS
As usual, it's simple, add this to your NixOS configuration and
reconfigure the system.
```
services.netdata = {
enable = true;
config = {
global = {
# uncomment to reduce memory to 32 MB
#"page cache size" = 32;
# update interval
"update every" = 15;
};
ml = {
# enable machine learning
"enabled" = "yes";
};
};
};
```
You should have Netdata dashboard available on http://localhost:19999 .
## Streaming mode
Here is a simple configuration on NixOS to connect a headless node
without persistency to send all on a main Netdata server storing data
but also displaying them.
You need to generate an UUID with uuidgen, replace UUID in the text
with the result. It can be per system or shared by multiple Netdata
instances.
My networks are 10.42.42.0/24 and 10.43.43.0/24, I'll allow everything
matching 10.* on the receiver, I don't open port 19999 on a public
interface.
### Senders
```
services.netdata.enable = true;
services.netdata.config = {
global = {
"default memory mode" = "none"; # can be used to disable local data storage
};
};
services.netdata.configDir = {
"stream.conf" = pkgs.writeText "stream.conf" ''
[stream]
enabled = yes
destination = 10.42.42.42:19999
api key = UUID
[UUID]
enabled = yes
'';
};
```
### Receiver
```
networking.firewall.allowedTCPPorts = [19999];
services.netdata.enable = true;
services.netdata.configDir = {
"stream.conf" = pkgs.writeText "stream.conf" ''
[UUID]
enabled = yes
default history = 3600
default memory mode = dbengine
health enabled by default = auto
allow from = 10.*
'';
};
```
# Netdata cloud
Netdata company started a "cloud" offer that is free, but they plan to
keep it free but also propose more services for paying subscribers.
The free plan is just a convenience to see metrics from multiple nodes
at the same place, they don't store any metrics apart metadata (server
name, OS version, kernel, etc..), when you look at your metrics, they
just relay from your server to your web browser without storing the
data.
The free cloud plan offers a correlating feature, but I still didn't
have the opportunity to try it, and also email alerting when an alarm
is triggered.
|
| Netdata cloud website |
| Netdata cloud data privacy information |
|
## Adding a node
The official way to connect a Netdata agent to the Netdata cloud is to
use a script downloaded on the internet and run it with some parameter.
|
| Connecting a Linux agent |
|
I strongly dislike this method as I'm not a huge fan of downloading
script to run as root that are not provided by my system.
When you want to add a new node, you will be given a long command line
and a token, keep that token somewhere. NixOS Netdata package offers a
script named `netdata-claim.sh` (which seems to be part of Netdata
source code) that will generate a pair of RSA keys, and look for the
token in a file.
|
| Netdata data page: Add a node |
|
Once you got the token, we will claim it to associate it to a node:
1. create /var/lib/netdata/cloud.d/token and write the token in it
2. run nix-shell -p netdata --run "netdata-claim.sh" as root
3. your node should be registered in Netdata cloud
# Conclusion
Netdata is really a wonderful tool, ideally I'd like it to replace all
the Grafana + storage + agent stack, but it doesn't provide persistent
centralized storage compatible with its dashboard. I'm going to
experiment with their Netdata cloud service, I'm not sure if it would
add value for me, and while they have a very correct data privacy
policy, I prefer to self-host everything. |