The paper introduces the fast feedforward (FFF) architecture, which offers comparable performance to feedforward networks at a fraction of the inference cost. FFFs can replace feedforward networks and mixture-of-expert networks in transformers. A vision transformer trained with FFF achieves single-neuron inferences with only a 5.8% performance decrease.