Based pm Rabbi Frand and Rabbi Kaminezky - The Ramban suggests an answer to this question. He says this corroborates Chazal’s teaching on the pasuk “As they (the Egyptians) persecuted them (the Jews), so did they multiply and so did they expand…” [Shemos 1:12]. The more the Egyptians tried to minimize us through their bondage and persecution, the moreHashemblessed us and allowed the Jewish women to have multiple births, creating a population explosion amongst the Children of
Israel. Chazal teach that the Tribe of Levi was not subjected to
the bondage of slavery. They were free from the work and the persecution suffered by the other tribes. Therefore, since they were not part of the persecution, they also were not part of the blessing of the population explosion, and consequently their total population at the end of the period of Egyptian slavery was much smaller than that of the other tribes. 

The Ohr HaChaim HaKadosh quotes this Ramban and is not happy with his suggestion. He offers his own unique—and in a sense, startling—answer: The Gemara says that Amram divorced his wife (after having only two children—Miriam and Aharon). His
rationale for doing so was that given the grim situation of the Jews in Egypt (Jewish male children being thrown at birth into the Nile, at that time), he did not want to bring any more children into the world. Amram was the gadol ha’dor (the leader of the generation). He was also the head of the Tribe of Levi. When the rest of his tribesmen saw that Amram divorced his wife, they all got up and divorced their wives as well. Even though the Gemara says
that Amram had second thoughts about the matter and remarried his wife Yocheved, the Ohr HaChaim suggests (this is speculation on his part) that the other Levites did not follow his lead in that action, and they remained separated from their wives. ........... 

....... The Baal HaTurim points out that there are only two places in all of Tanach where we find the expression “va’yelech ish“. One is the aforementioned pasuk “va’yelech ish m’Beis Levi va’yikach es bas Levi,” and the other is in the Book of Rus; “va’yelech ish m’Beis Lechem Yehudah…” (And a man went from Bethlehem in Judea to dwell in the fields of Moab) [Ruth 1:1].