It doesn't really bother me to be called Ma'am. I have been, several times and have a very polite southern young man that works in our department that always says Ma'am. It seems to annoy some of my co-workers, but I pretty much take it as someone being courteous no matter who it's from.
The rarity of manners nowadays makes me pleased when someone says it. Besides, like someone else mentioned (forgot who now, sorry), it goes with martial arts. I got used to it when training in taekwondo. Plus I live in the southern US...hence the southern charm. Sir and Ma'am are pounded into many folks who grew up down here. I didn't grow up this far south so I don't do it much. Mostly just for someone in authority or an older person. Now I'm not a youngster any more so an "older person" is getting OLDER as I age too.
"I think if you try hard enough and make the best of a situation, the situation won't get the best of you." -MacGyver
Semper Vigilans - Civil Air Patrol