After Jacob dies, the older brothers wonder if Joseph will finally turn his back on them and pay them back for what they did to them. When Joseph hears that his brothers are insecure and uncertain at their standing with him, it broke his heart.