When you have a difficult person in your life—and who doesn't? —you have four basic options—much the same as how you would deal with a pair of shoes that hurt your feet because a nail has worked its way through the sole.  You can walk away from him, ignore him — perhaps getting calloused to his difficult personality, put some space between you and him (which allows you to live with the situation) or respond to the difficult person, which allows you to cope.  Usually this is the most positive and most satisfactory course of action when a difficult person really begins to get to you.