There is a common error that if you are suddenly distracted in middle of your devotions, you weren't doing a good job. That would be true if you had but one soul –– but you don't. You have two. Just ignore that distracting voice as though it were a stranger outside of you. And if the going gets rough, don't forget to ask Hashem for help.