If someone offers you a gift, and you decline to accept it, the other person still owns that gift. The same is true of insults and verbal attacks.