At 11 weeks, your puppy almost certainly won't be fully protected from parvovirus (even if he has had the complete round of shots, it's not to do with the number of shots, but about the maternal antibodies in his system which counteract the vaccine, and that depends on many factors, but largely time), so you need to take your vet's guidance on whether he should be out walking yet or not. It depends very much on the risk in your area.
In the UK, most people choose to carry their puppies in their arms when out and about until a week after their final shots, which normally means about 13 weeks of age. The Guide Dogs puppies have to be carried until they are 17 weeks of age, as that is what they deem safe.
The "five minute rule" is a bit of a common-sense guideline, but it isn't based in any scientific research, so people have very different interpretations of it. But we do know some things: impact is bad for developing joints. Repetitive movement is bad for developing joints. Unnatural movement is bad for developing joints. So we need to minimise those things. Of course, we also don't want to wrap our puppies up in cotton wool and we need them to be able to explore their environments, different textures, wobbly surfaces etc, all as part of their habituation programme.
Just be aware that walking on hard surfaces such as asphalt is very hard on the joints, as is walking at an unnatural pace, which the puppy will do when on lead. Restrict retrieving activity to no more than a handful of throws every few days, as the impact through the elbows is very high. Don't allow your puppy to run up and down the stairs; although an introduction to stairs at an early age is no bad thing, it's obviously very high impact, so you need to minimise it. Teaching your puppy to go up and down stairs slowly is a very useful lesson.
I would take my latest puppy out for an hour by the time she was about 18 or 20 weeks, but a lot of that time was sitting down while she snuffled around in the woods around me. We didn't go on "walks" per se, just enjoyed nature, played little games together and built the foundations of behaviours that I knew I'd want - recall, walking next to me, choosing to be with me etc. We probably went less than 300m from the front door, all told!