I've had several people ping me in the last couple of days, including my Sorcerer's Apprentice, about starting points for designing an adventure. From talking to lots of gifted people over the last few years, I think I can definitely say that there is no objective "right" starting point. The most common things I have heard tend to divide between those who write monsters first, and those who draw maps first.
In my case, it's usually the map first, starting with really tentative sketches. Then coming up with a basic idea and writing it as the "Background" just so there are words on the blank page (a completely blank page is a real inhibitor for creativity).
Then I put some numbers on the map and begin filling in what's in those numbers. This process totally changes the background, and it often means going back to earlier encounter areas and changing them to match the new ideas that are developing. There's a lot of inefficient re-writing, but usually it's worth it as the adventure develops into something I really didn't expect at the beginning.
The final product is usually completely different from the starting point, and being surprised by the outcome is definitely part of the fun of writing an adventure. You never know what you're going to find in your own head.