For months, I’ve been happily changing my nagios service configs without incident. Suddenly, the last change I made and restarted, nagios sometimes uses the new configuration and sometimes the old, it flipping back and forth between the two. What I’ve discovered is that, periodically, the status.dat file shows the old service definition, and then sometimes the new definition. I know that status.dat is periodically written by the nagios process and so to try to rememdy the problem, I’ve reconfig’ed nagios not to use retention data at all and blown away retention.dat, status.dat and objects.cache before restarting nagios. But, it STILL keeps happening.
So, my question is: Where else could nagios be getting the old config from?
Odds are that you have 2 nagios processes running. Stop nagios, killall -9 nagios (a couple of times until it tells you hou have no processes to kill), start nagios. Generally I found this happened when I used reload rather than restart.
nevermind! gonna sound stupid, but whatever you do, DO NOT let two nagios processes run at the same time. When I first made the changes and used the service call to restart nagios, there were some issues with it going down. I killed it (or so I thought), and so every subsequent restart to try to determine the cause of what I was seeing was only killing the new process, leaving the old to run. Killing the old one, THEN restarting nagios resolved this. Whoops!
