Notification timeout oddities


#1

Hi All,

I’ve been running nagios for about 2-3months fine (thanks for the help with initial problems btw)

I’ve just changed my configuration to incorporate notification via SMS. Everything works fine from command line however, I get the following error from the nagios log:

[01-04-2006 15:04:56] HOST NOTIFICATION: dbnx_onCall;hostsrv2;DOWN;host-notify-by-text;CRITICAL - Plugin timed out after 10 seconds

thats fair enough how my nagios.cfg file reads:
notification_timeout=120

is there another timeout setting that I’ve missed?
failing that does anyone have any idea as to why this is occuring?

cheers in advance,
Andrew


#2

Ah…notification timeouts and plugin timeouts are two different things that you seem to be confusing. By default, Nagios labels a service CRITICAL if it times out for 10 seconds. I don’t know if there’s a way to change the defaults so that Nagios doesn’t label it as such until after a certain period of time.

…]

But from the looks of it, Nagios is running just as it should.

Edited Wed Jan 04 2006, 06:03PM ]


#3

Sorry could be me not being clear enough. the plugin timing out isn’t a service check, it is using smsclient to notify engineers about problems.

the notification_timeout (according to comments in nagios.cfg) is in seconds.

I’m still at a loss as to were I’m going wrong though,

although I have tried running the script as a background process (&), just need to wait for a service to complain that it’s busted ;). I’ll let you know what happens,

cheers,
Andrew


#4

Is it bad that it’s using SMS to notifiy engineers about problems? I guess now I’m trying to figure out what you want Nagios to do for you.


#5

Basically we are a service provider, the idea is for nagios to tell us about a problem before the phone starts going beserk with confused customers.

This works fine during office hours with email, however as we operate a 24/7 policy I would rather be notified, awake and fixing a problem before the MD rings at 4 in the morning asking why customer X has just woke him up. :wink:

I’m still playing about with it, the Nagios log says the notification has been sent but there is still nothing coming out of the other end.

Incidently running as a background process didn’t help, although it did stop nagios from killing it. I’m now wonderng if it could be a problem with permissions although the bin is chown nagios:www-data.

Thanks for the help so far, any other ideas would be greatly appreciated,
Andrew


#6

OH! You’re saying that notifications aren’t being sent to your cell phone, is that it?


#7

thats the one, sent many from the commandline no problem, just nagios thats the stumbling block now.

definition in misccommands.fg is:

‘host-notify-by-text’ command definition

define command{
command_name host-notify-by-text
command_line /etc/nagios/sms_client $CONTACTPAGER$ “Host: $HOSTALIAS is $HOSTSTATE$” &
}

sms_client syntax is:
sms_client


#8

I’m not familiar with using sms_client, and since I’m seeing you’re from the UK, I don’t know how this will work (I don’t know how cell carriers operate outside the US).

I’ve just been e-mailing notifications to my cell phone. Carriers in the US associate your phone number with a unique e-mail address containing your phone number, hosted on the carrier’s domain. For example, a Cingular cell phone’s e-mail address would be [email protected]. I can send messages to that address from any email service that I’ve tried, and I receive the e-mails just as if they’re SMS messages. I don’t know if that’s how things work with carriers in the UK, though.

Anywhos, I’ve just been modifying the default “notify-by-email” command and included my mobile e-mail there, and I’ve been getting notifications whenever there’s an issue. Using the Unix mail command works fine, but not when that printf string is piped to it. So here’s what my command looks like:

Notify My cell - service problem
define command{
command_name notify-by-mobile
command_line /bin/mail -s “Nagios-02: $SERVICEDESC$ on $HOSTALIAS$ is $SERVICESTATE$!” $CONTACTEMAIL$
}

I just had to make sure to define another contact whose e-mail was my mobile e-mail. You might decide this is more trouble than it’s worth, and it might not work in the UK, but I thought I’d try anyways 'cause I don’t know much about sms_client.

Weird that you’re getting that timeout error in Nagios and not from command-line, though…


#9

I’ll give that a shot, but I’m not too sure how SMS works in UK either tbh. I’m running sms_client as I inherited it from my predecessor. Been shifting to nagios from the old mon setup we had.

I’ll let you know how it works, but I’m off home no as it’s finally the end of the day, have a good one.


#10

Well, let me know how it goes. It might be quicker to contact Customer Service for your carrier or access their website. That’s how I found out all my info pertaining to mobile e-mail addresses.


#11

[quote=“waity85”], I get the following error from the nagios log:

[01-04-2006 15:04:56] HOST NOTIFICATION: dbnx_onCall;hostsrv2;DOWN;host-notify-by-text;CRITICAL - Plugin timed out after 10 seconds[/quote]

You can run it by hand, but did you try it as user nagios?
su - nagios
now run whatever command line you have your host-notify-by-text configured as.
Like perhaps:
/usr/bin/printf “%b” “***** Nagios ***\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $DATETIME$\n\nAdditional Info:\n\n$OUTPUT$" | /bin/mail -s " $NOTIFICATIONTYPE$ alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **” [email protected]

From your description, the command will fail, since nagios is unable to execute your command defintion, when it’s suppose to.


#12

My apologises for not coming back sooner.

First off SMS notification is now working, so thanks for the help. It turned out to be a permissions error with the sms_client’s modules log file.

In response to the last remaining suggestions made:

Son of Thunder: I went to the Customer service department for Vodafone (my provider). The UK doesn’t provide any service similar to that of the US in this regard. The best we’ve got is a Vodafone email address that will be checked periodically by the phone handset and notification given that there are unread messages in the mailbox.
Not the best solution as I don’t fancy being woken up everytime the world and his dog try selling my viagra (or whatever the latest spam craze is ;))

N.B. if there is anyone from the telecoms industry reading this, why don’t we have a similar set-up to the US? could be an opportunity there…

Jakkedup: It still ran when called of the commandline as the nagios user, that was one of the things that was getting me stumped. The user was getting access to the log file from commandline but not when it was spawned from nagios.

On a seperate note, if there is anyone in the UK having similar problems with SMS notification, give me a shout and I’ll try to help all I can.

Again, thanks for all the help and apologises for taking so long to get back to you,

Andrew