Why don't we remember everything? This an inefficient use of our resources. Our resources being mental capacity, energy and maybe even time.
Generally, we remember what is often used or linked with emotional stimulation (or, I suppose, strongly linked with something else).
The reason you can't remember what you ate for lunch yesterday is the same reason you remember so much about the world.