it is such a problematic feeling when you gotta say YES to someone instead of saying NO. as long as you sure about your decision then it is OK to say what you mean to say. being TRUE to yourself and listen to yourself is the most important thing to do. 

Happy Listening!