If he kept y'all waiting too long don't be afraid to tell him "better luck next time", but if it's not going to change how soon you can arrive in town, then try to include him.