This comprehensive survey examines in-context learning (ICL) in large language models (LLMs), a capability that allows them to learn tasks from examples provided within the input. The paper explores advancements from theoretical viewpoints, such as mechanistic interpretability and mathematical foundations, and empirical perspectives, analyzing factors influencing ICL like pre-training data, model properties, and demonstration characteristics. Understanding ICL is crucial for improving LLM performance, utilizing their adaptable nature without retraining, and addressing potential risks such as bias and toxicity. The survey highlights open questions and suggests future research directions to move from correlational analysis to causal understanding and improve evaluation methods for ICL.