This research introduces DISCIPL, a novel framework where language models dictate their own reasoning process for complex tasks. By having a Planner LM generate inference programs, DISCIPL orchestrates Follower LMs to solve problems, offering a more efficient and automated approach compared to existing methods like chain-of-thought or structured inference. The framework supports various inference patterns, including constrained generation and self-correction, and achieves significant performance gains on challenging tasks requiring structured reasoning and constraint satisfaction, as demonstrated through evaluations on diverse datasets. This work presents a new way to integrate code generation and probabilistic inference, enabling improved test-time adaptability for language models.