This paper investigates the limitations of large language models (LLMs) as evaluators when directly scoring natural language generation quality, finding that existing calibration methods are insufficient to align their judgments with humans. Inspired by preference-based training in RLHF, the authors propose Pairwise-preference Search (PAIRS), an efficient, scalable method that reframes evaluation as a ranking problem using uncertainty-guided pairwise comparisons. PAIRS is shown to outperform direct scoring and some specialized metrics in aligning with human judgments across summarization and story generation tasks, while also offering insights into the transitivity of LLM evaluations and benefiting from calibration.