When you don’t ask for permission before taking a shot, you obviously expose yourself to all kind of reactions. In very few instances, people will get mad and start arguing with you. Their arguments almost exclusively revolve about the fact that they think it is illegal to have their picture taken in the street, which is incorrect. My best advice here is to just smile, apologize and go away. Don’t waste your time, you’ll never succeed to convince them and once you have engaged in the discussion, it will only be a slippery slope.
This being said, I never had an argument become physical. And only twice in the last three years did someone show me the middle finger when I took their picture. One of them happened actually last evening, though I am almost sure that this sympathetic woman didn’t really mean her gesture. (click on picture to enlarge)
Leica M9 with 35mm Summilux Asph at F1.4, 1/60, ISO200