This research paper introduces a novel approach called Active In-context Prompt Design (AICL) for improving the performance of large language models (LLMs) through adaptive prompt tuning. The paper addresses the challenge of selecting the most informative examples to include in an LLM's prompt at inference time to optimize its predictions on a set of test queries. To achieve this, the authors propose two active learning algorithms: G-Optimal design (\go), inspired by optimal experimental design in linear models, and Simulation-Based Active Learning (\sal), which simulates the impact of labeling examples on the LLM's uncertainty. The paper presents theoretical analysis of these algorithms in the context of linear models and provides empirical evidence demonstrating their effectiveness across various tasks and LLMs compared to existing prompting strategies.