Possible Clues:

• Bother a lot
• Get one's goat
• Rattle one's cage
• Ruffle
• Get on one's nerves
• Drive bananas
• Really annoy
• Irritate, annoy
• Test the patience of
• Make upset