The provided text introduces ProjectEval, a new benchmark for automatically evaluating the project-level code generation capabilities of programming agents by simulating user interactions. ProjectEval aims to address the limitations of existing benchmarks, such as a lack of automated user-centric evaluation and result explainability. The benchmark includes diverse real-world tasks with varying input levels and employs automated test suites that mimic user behavior, alongside traditional code similarity metrics. Findings from ProjectEval highlight key capabilities necessary for programming agents to create practical projects and offer insights for future development in this field.