He was born into a Christian family at Nikopolis in Armenia. When he was eighteen his parents died, and with twelve other young men he established a small monastery. After a few years, much against his will he was made Bishop of Colonia, but he continued to live the ascetic life of a monk. After nine years of service as bishop, discouraged by the worldliness and intrigue around him, he secretly left for Jerusalem to live as a monk. He was divinely guided to the monastery of St Sabas, who received him and, knowing nothing of his rank, assigned him a lowly place among the new monks. Saint John cheerfully undertook whatever task was given to him and served the other monks in humility and silence. After completing his novitiate he was given a cell where he lived in total silence, fasting five days a week. On Saturdays and Sundays he joined the brethren for prayer, Communion and meals; but even at these times the other monks were edified by his silence and unceasing compunction. Saint Sabas desired to make him a priest and took him to be ordained by the Patriarch of Jerusalem. Saint John asked the Patriarch for a private meeting and explained that he could not be ordained because he was already a bishop. The Patriarch returned St John to St Sabas, telling him only that it was impossible for him to ordain John, who should be allowed to live in silence and solitude. Saint Sabas was perplexed (thinking that some sin prevented the monk from being ordained), but soon received a revelation of John's true rank.

After many years of reclusion, St John withdrew further to a cave in the desert for nine years. He became known as a divinely-enlightened counselor and a wonderworker, and cheerfully received all who came to him for guidance or prayer. In 509 he returned to the monastery, where he lived as an anchorite in his cell, communicating with the world only through one of his disciples. For many years he lived only on thin porridge, into which he would mix ashes. One day a disciple saw him pouring ashes into his food, and John abandoned the practice, not wanting to be known for the practice of any virtue.

Once he asked God for a sign revealing whether he would be granted to enter the Kingdom of Heaven. Taking a fig-seed, he placed it on a bare rock outside his cell. Without soil or water, the seed brought forth a plant, put forth leaves and flowers, and produced three figs, which St John shared with his disciples. The Saint then made ready for death. He reposed in peace, at the age of 104.