Orthogonal Finetuning (OFT) is introduced to guide text-to-image diffusion models for downstream tasks, preserving hyper-spherical energy essential for semantic generation. Constrained Orthogonal Finetuning (COFT) improves stability. It outperforms existing methods in quality and speed.