How will self-driving cars change urban mobility patterns? This talk examines scientific contributions in the field of reinforcement learning, presented in the context of enabling mixed-autonomy mobility-the gradual and complex integration of autonomous vehicles into existing traffic systems. We explore the potential impact of a small fraction of autonomous vehicles on low-level traffic flow dynamics, using novel techniques in model-free deep reinforcement learning. We share examples in the context of a new open-source computational platform and state-of-the-art microsimulation tools with deep-reinforcement libraries.