The first time I got this was when I was accused of being Australian, the New Zealander, and finally South African (this was before the end of apartheid) while visiting California.

On the whole people are people and I've found that if their ethnic background becomes relevant they'll tell you. Everything else is just being annoying - deliberately or otherwise...