I figured that was intentionally set up so that you would fail first time, to let you know that it was ok to fail some missions. You had no reason to go to that area until attempting that mission, so likely wouldn't have cleared out the bandits before trying it.
That's quite an easy one to do. The trick is to go down there clear the area of bandits first and then do the mission.
Hit the first optional timed mission tonight, tried it once, failed, never again. At least that one was optional. Deliver mail. Yes, it really was as unexciting as it sounds.