Turns out, it is totally racist!
"Late 19th century: from Paddy, associated with obsolete paddywhack 'Irishman (given to brawling')."
So two things:
1. That is a truly fantastic way of adding linguistic insult to colonial injury. Steal someone's country, kill a bunch of its people, outlaw their language, culture, and hairstyles, make up faux scientific theories about how they're the missing link between apes and humans... and then basically accuse the entire nation of being pathologically grumpy and prone to punching.
"Haha, we piss you off, and THEN we mock you for being pissed off! Gosh it's good to be the centre of the known universe."
2. Apparently broadening one's social circle to include people you haven't known since you were 15 is pretty useful in the life-long project to be a bit less of a dick.