The documentation is a little unclear on this point, so allow me to help!
all you have to do is tell it the database name, and the username and password you created for it.
./installdb -u nagios -p nagios -d nagios
That line assumes:
your database is located on the same server as nagios
your database is called "nagios"
the username and password you created are both set to "nagios"
if you put the line through that way, it should work.