We present a theory for the damping of layer-by-layer growth oscillations in molecular beam epitaxy. The surface becomes rough on distances larger than a layer coherence length which is substantially larger than the diffusion length. The damping time can be calculated by a comparison of the competing roughening and smoothening mechanisms. The dependence on the growth conditions, temperature and deposition rate, is characterized to be a power law. The theoretical results are confirmed by computer simulations.