This paper introduces a novel approach called Alignment from Demonstrations (AfD) for aligning large language models (LLMs) using demonstration datasets instead of preference-based data. The paper frames this alignment problem within a reinforcement learning (RL) framework, specifically exploring connections to forward and inverse RL. It theoretically analyzes trajectory distribution matching objectives, linking supervised fine-tuning to forward KL divergence and adversarial learning to reverse KL divergence. Finally, the paper proposes a computationally efficient algorithm for AfD based on reward model extrapolation and presents experimental validation of its effectiveness.