Scintillating CaWO4 single crystals are a promising multi-element target for rare-event searches and are currently used in the direct dark matter experiment CRESST (Cryogenic Rare Event Search with Superconducting Thermometers). The relative light output of different particle interactions in CaWO4 is quantified by quenching factors (QFs). These are essential for an active background discrimination and the identification of a possible signal induced by weakly interacting massive particles (WIMPs). We present the first precise measurements of the QFs of O, Ca and W at mK temperatures by irradiating a cryogenic detector with a fast neutron beam. A clear energy dependence of the QF of O and, less pronounced, of Ca was observed for the first time. Furthermore, in CRESST neutron-calibration data a variation of the QFs among different CaWO4 single crystals was found. For typical CRESST