He lived in the desert of Skete (Scetis) in Egypt during the fourth century, the golden age of the Desert Fathers. Nothing is known of his life in the world. He spent many years as the disciple of Abba Ammoes, who was very severe with him. Once the Elder took a dry stick, stuck it in the sand, and commanded John to water it every day until it bore fruit. Though this was plainly impossible, John performed the task uncomplainingly, walking a great distance to fetch the water, for three years. At the end of that time, the stick bore fruit. Abba Ammoes brought it to church the following Sunday and called out to the brethren, "Come and eat the fruit of obedience!" Though he had never praised or thanked his disciple, before he died Abba Ammoes said of John, "He is an angel, not a man." After his elder's repose, Abba John withdrew further into the desert, devoting all his time to vigil and prayer. As he prayed he would weave baskets, which he sold to meet his few needs. Sometimes he was so rapt in prayer that he would keep weaving until the basket reached an absurd size, filling his cell.

When, after many years, Abba John was delivered from all evil thoughts, Abba Poemen (commemorated August 27) told him to pray to God for another temptation to struggle against, for only in this way does the soul make progress. He rejoiced when he was insulted, was never known to be angry with anyone, and would run away as fast as he could if he ever saw men quarreling. He reposed in peace.

"Pray earnestly with compunction and vigilance. Pay no attention to the faults of others. Do not measure yourself against other people, for you are lower than every creature." — Abba John the Dwarf