Nagios stops checking after start

Nagios
1

I know nothing of Linux and very little about Nagios, I inherited the system for a tech who left.

Checks of hosts will be performed after a reboot of the nagios services approx 4 times, then it stops checking until the next restart

Also the services do not appear to have been checked for any of the systems for the last 3 days.

Running 3.1.0 and with NagiosQL installed also 3.0.3

HELP!

Mark

running /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg states no serious problems were detected

2

first thing, stop nagios, check there are no leftover nagios processes (“ps -ef | grep nagios”) if there are kill the leftover processes. then start nagios.

See what happens :slight_smile:

3

login as: root[root@idc-nagios01 ~]# service nagios stop
Stopping nagios: /etc/init.d/nagios: line 67: kill: (11911) - No such process
done.
[root@idc-nagios01 ~]# ps -ef |grep nagios
avahi 2149 1 0 Nov19 ? 00:00:00 avahi-daemon: running [idc-nagios01.local]
root 24355 24310 0 06:31 pts/0 00:00:00 grep nagios

4

idc-nagios01.local is the name of your server?

try starting nagios again and see what happens.

Thinking of it… Keep in mind that Hosts usually DO NOT get checked as long as services on the host respond, when the servoces fail nagios will check if the host is alive.

5

After I restarted, made no difference, did the same thing.

Not sure if these will help ?

/usr/local/nagios/bin/nagios -sv /usr/local/nagios/etc/nagios.cfg

Processing object config file ‘/etc/nagiosql/hosts/hou-esx01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/escondido.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-rsa01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-filer01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sbcitrix01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-esx03.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-lumtron01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-ex01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/lan-mgmt01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/cafrn1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/agourahills.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/cairv1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/proxy.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-blkberry01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-esx02.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-om01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-prodapp01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/kop-c3750g-sw1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/comsrv01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/domdocsd.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/walac1sw1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/wan-admin.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sdexchnotes.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idcfiler01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/newportbeach.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-dhcp01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-citrix01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/casdg1is6.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/fre-ups01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-esx10.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/nbo-lumtron01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/mlsec.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/wan-apps.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/smo-dc01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/drivernet.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/wasea1s1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/asinta.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-esx01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/tolvmdbjkmb71.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/optiman.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-exchnode01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/irv-filer01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/lacey_3750G.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/mibir1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/irsql.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/mdlan1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/coden1sw1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/cabkl1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-owa01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/vmware3.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/cairv1s1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sdprint01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-tacacs01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/vmware1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/lan-dc01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/seattle01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/dal-dc01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-trend01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-3750.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-netpro01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sea-dc01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/printserver.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-notes03.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/backup.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/casdg1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-intgdev01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-faxadv01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/gaatlms1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sfofiler01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sf-ups01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-esx01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/sbo-ups02.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-hypapp02.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/nbo1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/mpls-7206.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-raster01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/casdg1is3.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/atl-print01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/cafre1sw2.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/phx-3560-1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/smo-ups01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/ctwtr1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-trmlice01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-unity01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-bes01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/irv-ups01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-dc01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/orptl1s1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/ldcitrix1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-citrix03.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/wtr-vm01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/chn-fax01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/den-file01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/dal-fs01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-esx02.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-data01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/chn-dc01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/mibirs1ms1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-sql01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/seattle_router.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-citrix01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/hou-ctx01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/vachn1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-kltcitrix01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/ahg_sprint_mpls.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/escitrix02.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/casac1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/irv01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-sharepoint01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/oktul1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/calao2r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/cabkl1sw1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/nynyc1r1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-exch01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/idc-smbr01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/phx-3560-2.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/daiswin2.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/casmcsis1.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/san-wsus01.cfg’…
Processing object config file ‘/etc/nagiosql/hosts/casfo1r1.cfg’…
Processing object config file ‘/etc/nagiosql/contactgroups.cfg’…
Processing object config file ‘/etc/nagiosql/servicegroups.cfg’…
Processing object config directory ‘/etc/nagiosql/services’…
Processing object config file ‘/etc/nagiosql/services/fan_status.cfg’…
Processing object config file ‘/etc/nagiosql/services/cisco_cpu_load.cfg’…
Processing object config file ‘/etc/nagiosql/services/backup_exec_service.cfg’…
Processing object config file ‘/etc/nagiosql/services/router_loopback0.cfg’…
Processing object config file ‘/etc/nagiosql/services/volt_status.cfg’…
Processing object config file ‘/etc/nagiosql/services/power_status.cfg’…
Processing object config file ‘/etc/nagiosql/services/interface_check_serial_0_0_0_0.cfg’…
Processing object config file ‘/etc/nagiosql/services/interface_check_t1_0_1_0.cfg’…
Processing object config file ‘/etc/nagiosql/services/temp_status.cfg’…
Processing object config file ‘/etc/nagiosql/services/router_multilink.cfg’…
Processing object config file ‘/etc/nagiosql/services/interface_check_t1_0_0_0.cfg’…
Processing object config file ‘/etc/nagiosql/services/interface_check_serial_0_1_0_23.cfg’…
Processing object config file ‘/etc/nagiosql/services/all_services.cfg’…
Processing object config file ‘/etc/nagiosql/services/interface_check_t1_0_0_1.cfg’…
Processing object config file ‘/etc/nagiosql/services/dns_status.cfg’…
Processing object config file ‘/etc/nagiosql/services/bgp_check.cfg’…
Processing object config file ‘/etc/nagiosql/serviceextinfo.cfg’…
Processing object config file ‘/etc/nagiosql/hosttemplates.cfg’…
Processing object config file ‘/etc/nagiosql/servicedependencies.cfg’…
Timing information on object configuration processing is listed
below. You can use this information to see if precaching your
object configuration would be useful.
Object Config Source: Config files (uncached)
OBJECT CONFIG PROCESSING TIMES (* = Potential for precache savings with -u option)

Read: 0.320385 sec
Resolve: 0.000934 sec *
Recomb Contactgroups: 0.000056 sec *
Recomb Hostgroups: 0.003438 sec *
Dup Services: 0.002610 sec *
Recomb Servicegroups: 0.030042 sec *
Duplicate: 0.000026 sec *
Inherit: 0.000419 sec *
Recomb Contacts: 0.000001 sec *
Sort: 0.000001 sec *
Register: 0.006337 sec
Free: 0.000858 sec
============
TOTAL: 0.365114 sec * = 0.037534 sec (10.28%) estimated savings

Read object config files okay…
Running pre-flight check on configuration data…
Checking services…
Checked 529 services.
Checking hosts…
Checked 321 hosts.
Checking host groups…
Checked 40 host groups.
Checking service groups…
Checked 7 service groups.
Checking contacts…
Checked 23 contacts.
Checking contact groups…
Checked 4 contact groups.
Checking service escalations…
Checked 0 service escalations.
Checking service dependencies…
Checked 0 service dependencies.
Checking host escalations…
Checked 0 host escalations.
Checking host dependencies…
Checked 0 host dependencies.
Checking commands…
Checked 73 commands.
Checking time periods…
Checked 5 time periods.
Checking for circular paths between hosts…
Checking for circular host and service dependencies…
Checking global event handlers…
Checking obsessive compulsive processor commands…
Checking misc settings…
Total Warnings: 88
Total Errors: 0
Timing information on configuration verification is listed below.
CONFIG VERIFICATION TIMES (* = Potential for speedup with -x option)

Object Relationships: 0.004586 sec
Circular Paths: 0.004768 sec *
Misc: 0.000539 sec
============
TOTAL: 0.009893 sec * = 0.004768 sec (48.2%) estimated savings

Things look okay - No serious problems were detected during the pre-flight check
You have new mail in /var/spool/mail/root
[root@idc-nagios01 etc]#

Files under /usr/local/nagios/etc/

1 backup5.tar.gz list.txt objects
2 backup.uc.tar.gz location.cfg pnp
backup2.tar.gz cgi.cfg nagios.cfg resource.cfg
backup3.tar.gz create_location.sh nagios.cfg.bak
backup4.tar.gz htpasswd.users nagiosgraph

6

Its named idc-nagios01 on our systems.

I have restarted it and reboot the actual server several times.

It checks the hosts 4-5 times, then stops checking and hasnt check the services on anything for the last 4 days

7

If no service checks are executed check if active_checks_enabled is set to 1 in the hosts template you are using…
anything in the nagios.log file when it stopped checking?
Are the service checks being rescheduled?
could it be some template is using a timeperiod which doesn’t make sense?

8

Nothing in nagios.log file.