This research paper assesses the real-world path-planning capabilities of three large language models (LLMs): GPT-4, Gemini, and Mistral. The authors tested the LLMs across six diverse scenarios, including turn-by-turn navigation and vision-and-language navigation. The results revealed significant errors across all LLMs and scenarios, demonstrating their unreliability for real-world path planning. The study concludes that LLMs are currently unsuitable for vehicle navigation and proposes future research directions focusing on improved reality checks, enhanced transparency, and the potential of smaller, specialized models. The limitations of the study, such as its localized testing area, are also acknowledged.

https://arxiv.org/pdf/2411.17912