How to upgrade remote hosts plugins

Hey all,

im planning a nagios instalation over 150 servers, but im wondering that if i need to add a simples plugin after all servers are installed, or made a simple modification i need to do it 150 times ?

What are the solutions for this, anyone have a tip ?

Hi,

I know there are cleaner solutions, but I did it with around 400 servers through ssh:
we installed the “authorized_key” for ssh on all remote servers (that was the boring/hard part, because some had different passwords…).

Then, once done, we just had to “scp” the plugin, and do an “happend” (echo “command[my_command]=…” >> nrpe.cfg), and a restart of the nrpe via script.

I did work pretty well, except for the few odd servers we had to correct by hand (there are always those stupid servers that don’t react properly!).

As a side note: when I said “do a happend”, I advise you to check if the line isn’t already there (with a grep “your_line” nrpe.cfg| wc -l), so that you can run your script several times; because yes: you will have to! (because of those stupid servers :))

hope this helps;
but as I said, there might be better solutions :slight_smile:

You’re looking at ways to control a lot of servers at the same time. People usually make loops and run through, but there are applications if you want to get serious about it :slight_smile:
Heres a loop example that will make a file in your home dir with the contents “omgwtf”:

Make your server list rather than typing all these servers into your for loop:
cat > serverlist.txt
server1.com
server2.com
server3.com

Do your loop (those are backticks):
for i in cat serverlist.txt ; do ssh username@$i “echo omgwtf > /home/username/rawrrr.txt” ; done

Now, that’s annoying because you have to type your users login name for every single server iteration.

Loose mentions using SSH keys. Do a google search on how to set this up, and you will be able to run through that command above without typing a password.

But then, if you’re smart, you won’t allow root logins via ssh. Or maybe you dont have root on all boxes, just sudo. Adding a “sudo echo omgwtf >…” will work, but you’ll hvae to type your sudo password for every iteration. That’s total balls too.

Along comes pexpect. pexpect is a collection of python scripts that can automate your workload by “expecting” to get a password prompt, and automatically sending your password. Check it:

noah.org/wiki/Pexpect

Pexpect makes life a lot easier, but you have to a learn a little bit of python. Don’t worry, python is in my opinion the easiest scripting language you can learn a useful amount of.

So, pexpect has some quirks. Like Loose said, some servers are always going to be stupid. And pexpect doesn’t handle sudo gracefully…i mean, it’ll do it, but not perfectly every time in my experience. Maybe i just suck at python. Regardless, we’re looking into a package called “Func”. It allows mass control over mass amounts of servers. Right now i use a modified “hive.py” script from the pexpect package, and it is wicked, but the way its programmed limits its abilities to connecting to 30 servers or less at a time.

Func on the other hand sounds pretty good, though I haven’t tried it out yet. Will soon. You can check it out here:

fedorahosted.org/func

NO it’s not just for fedora!

Anyways hope this helps in your mass-control efforts.

guys thanks for the high value replys, you gave me some nice ideas to implement and manage lots of remote machines.

Now i have to think in a way to make sure that every host got the closest configuration possoble, this i will have more problems cuz many hosts are already installed and for what i see till now, they are all with different configurations… noobs work…

Once again thanks a lot :smiley: