Although all the Shvotim were great Tzaddikim they needed to separate themselves from the world to serve Hashem that is why they where shepherds. Yoseph however did not have that choice, he was sold as a slave thrown in prison and eventually needed to run all Mitzraems finances and yet still maintained his full commitment to Torah and Mitzvahs.

לקו״ש כה