So the service is checked every 5 minutes, if in your nagios.cfg you have interval_length=60. You have a retry of 1 and a max check attempts of 3, so that makes 8 muntes. Sounds like it's working just the way you have it defined, give or take 2 minutes.
But if what you really want, is to be notified approx in 5 minutes, then you need to define it that way.
per the docs "
max_check_attempts: This directive is used to define the number of times that Nagios will retry the service check command if it returns any state other than an OK state. Setting this value to 1 will cause Nagios to generate an alert without retrying the service check again."