Often, I have exactly the same experience with my sense of hearing. This occurs, for example, when I listen to music that I haven't heard for a long time, or that I spent a long time listening to while engaging in a particular activity. When listening again, I suddenly get a vibrant memory of whatever I happened to be doing the last time I listened to the track, album or artist several times over. It is not only smells and sounds that bring back such memories, but every sense.
Furthermore, the majority of smells I experience don't remind me of anything at all, let alone bring back vibrant memories. Of course, I usually forget these - they're just everyday experiences of no significance.
Baring these things in mind, it seems more likely that uncommon experiences with emotional significance or that occurred a lot in a relatively short period, rather than simply your sense of smell, invoke these strong memories. The fact that many smells are uncommon means that your sense of smell brings back more vibrant memories than your others senses. There may be an element of increased sensitivity due to a physical connection with memory in the brain, but I am sure it is not entirely responsible.