NDOUTILS v 1.4b7 performance issue


#1

Hi,

I am running Nagios 3.0.6 (debian lenny 32bit xen domu) with NDOUTILS v 1.4b7.
Nagios checks ~ 1200 services/120 hosts (about 30 of service checks are passive).
There is no latency (only a few seconds) if ndoutils are not used, if I use ndoutils I get about 5-10min latency, the wait is constantly 50-100%, and obviously load average goes up as well.
Mysql is doing ~ 10qps.
What it looks like nagios is “pausing” until ndo finishes each query.
Is there any way to optimise it?
I have tried to use either tcpsocket/unixsocket, each have no effect on performance.
I could not find the doc on how to make ndo use file instead of socket.
It looks like ndo is a major hit on performance. I didn’t expect that such low volume of data to cause such load.

So my options are either optimise ndo and mysql or look for another plugin that will log data to mysql.
Is there any other plugin that allow nagios to log to mysql?

Thank you very much.


#2

Ok, here what I have done to fix the issue.
By default mysql comes with really conservative settings, so I did some optimisations on it:
key_buffer = 64M
max_allowed_packet = 32M
thread_stack = 512K
thread_cache_size = 32
key_buffer_size = 32M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 1M
thread_cache = 24
query_cache_size = 100M
tmp_table_size = 32M
thread_concurrency = 2
innodb_buffer_pool_size = 128M
innodb_additional_mem_pool_size = 16M
innodb_log_buffer_size = 8M
innodb_lock_wait_timeout = 50
innodb_log_file_size = 150M
innodb_log_files_in_group = 3

I also limited what is being logged into the database (only applicable for my application that does not need ~80% of the stuff):
event_broker_options=276480 in nagios.cfg
data_processing_options=276480 in ndomod.cfg

This list is useful to determine what number to use:

#define NDOMOD_PROCESS_PROCESS_DATA 1
#define NDOMOD_PROCESS_TIMED_EVENT_DATA 2
#define NDOMOD_PROCESS_LOG_DATA 4
#define NDOMOD_PROCESS_SYSTEM_COMMAND_DATA 8
#define NDOMOD_PROCESS_EVENT_HANDLER_DATA 16
#define NDOMOD_PROCESS_NOTIFICATION_DATA 32
#define NDOMOD_PROCESS_SERVICE_CHECK_DATA 64
#define NDOMOD_PROCESS_HOST_CHECK_DATA 128
#define NDOMOD_PROCESS_COMMENT_DATA 256
#define NDOMOD_PROCESS_DOWNTIME_DATA 512
#define NDOMOD_PROCESS_FLAPPING_DATA 1024
#define NDOMOD_PROCESS_PROGRAM_STATUS_DATA 2048
#define NDOMOD_PROCESS_HOST_STATUS_DATA 4096
#define NDOMOD_PROCESS_SERVICE_STATUS_DATA 8192
#define NDOMOD_PROCESS_ADAPTIVE_PROGRAM_DATA 16384
#define NDOMOD_PROCESS_ADAPTIVE_HOST_DATA 32768
#define NDOMOD_PROCESS_ADAPTIVE_SERVICE_DATA 65536
#define NDOMOD_PROCESS_EXTERNAL_COMMAND_DATA 131072
#define NDOMOD_PROCESS_OBJECT_CONFIG_DATA 262144
#define NDOMOD_PROCESS_MAIN_CONFIG_DATA 524288
#define NDOMOD_PROCESS_AGGREGATED_STATUS_DATA 1048576
#define NDOMOD_PROCESS_RETENTION_DATA 2097152
#define NDOMOD_PROCESS_ACKNOWLEDGEMENT_DATA 4194304
#define NDOMOD_PROCESS_STATECHANGE_DATA 8388608
#define NDOMOD_PROCESS_CONTACT_STATUS_DATA 16777216
#define NDOMOD_PROCESS_ADAPTIVE_CONTACT_DATA 33554432