This academic paper introduces FIBO, a novel approach to Bayesian optimization (BO) that significantly streamlines the process. Traditional BO relies on sequentially building surrogate models and optimizing acquisition functions, which can be computationally expensive and time-consuming. FIBO bypasses these steps by employing a pretrained deep generative model that directly samples from the posterior distribution of the optimal point, achieving faster computation times without sacrificing optimization performance on synthetic and real-world tasks. This in-context and zero-shot method is demonstrated to be an approximation of Thompson sampling and shows substantial speed improvements, especially in batched optimization settings.