I work for a small wired and wireless network infrastructure company and I use Nagios heavily to monitor client deployments. Recently one of our developers requested that Nagios data be stored in a database so that a reporting project could be undertaken. In the end, without permission or documentation he and another colleague carried out the implementation. I had already warned them that this was only going to be authorised if all potential issues were first explored…
Now I am getting false alerts in Nagios, which means that I and my support colleagues cannot rely on what was a bullet proof solution.
Having had discussions with the Development team here, it has been suggested that this may be to do with a clash between nagios and the ndoutils/ndo2db plugin, possibly relating to lock files (as Nagios attempts to perform checks and the database tries to carry out an update). Apparently our current configuration maintains the original text based logs and the DB, which one Developer claims is a potential problem…(Taken with a pinch of salt)
I don’t see how writing into a DB the results of a check could get other checks to fail… I’d expect to loose some data, not to get wrong results from checks. And i wouldn’t expect lock up problems until you start running LOTS of concurrent checks… mainly because you shouldn’t get concurrent updates on the same data.
Can you try to disable ndo2db and see if the errors go away?
However, I am told that there is a known issue/best practise which says, that you shouldn’t write to the nagios text files as well as to a database, that you should write logging/historical data solely to one or the other because it can create inconsistent states. I need to find out where that information came from, because it sounds like bull. The difficulty I have is that Nagios was fine before the Database was integrated into our system, so our developer seems to have caused us an issue.
It simply means that from now on if anybody wants a change made, they’ll have to go through me.
I’m hoping that somebody out there has seen something similar or has a suggestion of possible checks to ensure that the DB is not responsible in some way.
I had planned to shutdown ndo2db as my next check, I guess that would be decisive.