Then stop wearing them on your head.
Winter: bamboo or silk long johns
Summer: Mix of commando, swimsuit with mesh support and nothing else, and cotton briefs when modesty prevails. I am still looking for a nice, wickable boxer that can double as a swimsuit without worrying if someone is looking too closely.
Spring & Fall: the above for summer plus a pair of bamboo long johns stuffed into sack just in case the temps plummet.