This is a text-only version of the following page on https://raymii.org:
---
Title : Icinga2 / Nagios / Net::SNMP change the default timeout of 60 seconds
Author : Remy van Elst
Date : 16-05-2018
URL : https://raymii.org/s/tutorials/Icinga2_Nagios_Net_SNMP_raise_default_timeout.html
Format : Markdown/HTML
---
Recently a rather large amount of new infrastructure was added to one of my
monitoring instances. Using SNMP exclusively, but not the fastest network or
infrastructure. The SNMP checks in the Icinga2 instance started giving timeouts,
which look like false positives and give unclean logs. Raising the SNMP timeout
for the checks above 60 seconds was not that easy since the 60 second timeout is
hardcoded in the underlying library (NET::SNMP). This article shows you how to
raise that timeout on an Ubuntu 16.04 system.
<p class="ad"> <b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:</b><br><br> <a href="https://leafnode.nl">I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!</a><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days. </a><br><br> </p>
In the webinterface of Icinga2 the logs are unclean, making it harder to find
actual errors:
![][2]
### Timeout in the Manubulon SNMP checks
The plugins used for the SNMP monitoring are [from Manubulon][3], Icinga2 has
them integrated, and using Ansible I generate the configuration for the servers.
An example apply rule could look like this:
apply Service "snmp-win-service-Microsoft-Exchange-IMAP4" {
import "generic-service"
check_command = "snmp-service"
vars.snmp_perf = "true"
vars.snmp_v2 = "true"
vars.snmp_service_name = "Microsoft Exchange IMAP4"
vars.snmp_community = "example"
vars.snmp_timeout = "60"
vars.snmp_service_count = "1"
assign where match("example", host.name)
}
All the variables can be found in:
/usr/share/icinga2/include/command-plugins-manubulon.conf
Raising this timeout higher than 60 gives an error for the check. Let's dive on
the commandline and check what goes wrong:
/usr/bin/perl -w /usr/lib/nagios/plugins/check_snmp_process.pl -H example -C example -t 88 -n "Microsoft Exchange IMAP4"
Timeout must be >1 and <60 !
Looking through the perl code, `$o_timeout` is checked to not be greater then 60
seconds. Let's change that:
sed -i 's/$o_timeout > 60/$o_timeout > 900/g' /usr/lib/nagios/plugins/check_snmp_process.pl
Should be fixed right?
/usr/bin/perl -w /usr/lib/nagios/plugins/check_snmp_process.pl -H example -C example -t 88 -n "Microsoft Exchange IMAP4"
ERROR: The timeout value 88 is out of range (1..60).
And that error message is nowhere to be found in the plugin.
### Net::SNMP perl code
[This post][4] (the site is behind a login now, so I had to archive the google cache, neat), had more information where this error came from, hardcoded in the SNMP library used by the Icinga2 checks.
I was unable to find out why this limit was choosen and not made configurable.
[This][5] was the closest, but that does not give a reason. Now, 60 seconds is a
long timeout and I'd also rather not raise it, but this case is an exception.
Change it in library:
sed -i 's/sub TIMEOUT_MAXIMUM { 60/sub TIMEOUT_MAXIMUM { 900/g' /usr/share/perl5/Net/SNMP/Transport.pm
Which does allows for a larger timeout:
/usr/bin/perl -w /usr/lib/nagios/plugins/check_snmp_process.pl -H example -C example -t 88 -n "Microsoft Exchange IMAP4"
1 services active (matching "Microsoft Exchange IMAP4") : OK
Not the most pretty fix, but a suitable workaround. Do note that my checks are
not done every minute, but every 15 minutes or every hour. Otherwise it could
clog up and cause a waterfall of load everywhere in this, lets call it vintage,
infrastructure.
Also note that when you do system upgrades, these changes will be overwritten.
You might want to `chattr +i` some files.
[1]: https://www.digitalocean.com/?refcode=7435ae6b8212
[2]: https://raymii.org/s/inc/img/icinga_timeout.png
[3]: http://nagios.manubulon.com/
[4]: http://web.archive.org/web/20180516054845/https://webcache.googleusercontent.com/search?q=cache:QbXBdQJ8bzoJ:https://support.nagios.com/forum/viewtopic.php%3Ft%3D40836%26p%3D200360+&cd=2&hl=nl&ct=clnk&gl=nl&client=ubuntu
[5]: http://web.archive.org/web/20180516064030/http://search.cpan.org/%7Edtown/Net-SNMP-v6.0.1/lib/Net/SNMP.pm
---
License:
All the text on this website is free as in freedom unless stated otherwise.
This means you can use it in any way you want, you can copy it, change it
the way you like and republish it, as long as you release the (modified)
content under the same license to give others the same freedoms you've got
and place my name and a link to this site with the article as source.
This site uses Google Analytics for statistics and Google Adwords for
advertisements. You are tracked and Google knows everything about you.
Use an adblocker like ublock-origin if you don't want it.
All the code on this website is licensed under the GNU GPL v3 license
unless already licensed under a license which does not allows this form
of licensing or if another license is stated on that page / in that software:
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Just to be clear, the information on this website is for meant for educational
purposes and you use it at your own risk. I do not take responsibility if you
screw something up. Use common sense, do not 'rm -rf /' as root for example.
If you have any questions then do not hesitate to contact me.
See https://raymii.org/s/static/About.html for details.