Is there anything wrong in socializing with less than righteous people, even if you tell yourself that you won't let their negative attitudes and values affect you?
In using the words la'asok bdivrei torah, the Rabbis are teaching us that our Torah study requires an investment of energy and thought similar to that of our business ventures.