One suggestion is to have more practice speaking with an English native speaker so that it would be easier for you to understand this problem.

Join the waitlist for the complete English course: https://gonaturalenglish.com/prereg/

“I don’t care” can be very offensive because it could mean that it’s not important to me or it’s not much of a big deal for me. 

For example:

“Do you want to paint the wall blue or brown?”

“I don’t care.”

or

“I don’t care. Whatever you like.”

Your tone and your expression here is the most important tip in speaking English.

So if you want to be on the safe side, it’s recommendable to use, “I don’t mind.”