It sounds like I had the same childhood as you and I also grew out of those things and probably would have become a very sedentary person...who knows, but interesting to ponder...
However, I guess you could call me lucky, because when I was in the navy my ship went into the dry docks, in this case it was across the St Johns river, to get there I could either drive a ridiculous distance toward the west to find a bridge, then drive north, only to head back east to get the the dry docks I could practically see from my house...that was a crazy proposition. The second choice was to drive my car onto the St Johns river ferry cross the ferry and be basically there, very short commute, but there was one hitch... a $1.50 fee per crossing. However, I noticed on the price chart that pedestrians and bicyclists were able to cross for 10 cents. That was the answer!
I remember when I first started riding, I hadn't been on a bike in years, but I loved it and it wasn't just a short one-time nostalgic feel, I got hooked, so much so that I continued cycling after my ship returned to home port. Then I found myself taking leave so I could do long tours up and down the east coast...and now 30 years later I dare say I may just be addicted to riding. It is my primary form of transportation. Dogwood mentions how sugar can change one's brain chemistry, but I think cycling has changed my brain chemistry. I'll never stop.
So I kind of get your point about accidental exercise. Although, nowadays I've found that there are some elements of exercise that can't be all accidental. For instance, there are times to really hit it hard, i.e. go deep into the red (anaerobic), you can't really get the benefits of exercise without this, I found this relatively late in life, but since starting it, it has improved my health immensely; so much so that when I'm taking it easy I can go so much longer and my easy pace is so much faster than it use to be. When you first go into the red it hurts and it's something you don't want to do again, but over time it does become enjoyable, in a strange way. Now I have to force myself on some days not to do it too much, it's almost like a drug...dare I say that anaerobic training has changed my brain chemistry