While small earthquakes are expected to produce weak shaking, ground motion is highly variable and there are outlier earthquakes that generate more shaking than expected - sometimes significantly more. We explore datasets of M0.5-M8.3 earthquakes to determine the relative impact of common, smaller magnitude earthquakes that rarely produce strong ground motion to rare, large earthquakes that always cause strong shaking. We find that the natural variability of ground motion, combined with the Gutenberg-Richter magnitude-frequency relationship, ensures that most occurrences of any ground motion comes from earthquakes of smaller magnitude than expected, often >2 magnitude units smaller. This holds even for very strong shaking (>20%g), suggesting that M<7 earthquakes could be a significant source of damage.