Diastole is the sequence of physiological events that occur in the heart during ventricular filling and principally depends on myocardial relaxation and chamber stiffness. Abnormal diastolic function is related to many cardiovascular disease processes and is predictive of health outcomes, but its genetic architecture is largely unknown. Here, we use machine learning cardiac motion analysis to measure diastolic functional traits in 39,559 participants of the UK Biobank and perform a genome-wide association study. We identified 9 significant, independent loci near genes that are associated with maintaining sarcomeric function under biomechanical stress and genes implicated in the development of cardiomyopathy. Age, sex and diabetes were independent predictors of diastolic function and we found a causal relationship between genetically-determined ventricular stiffness and incident heart failure. Our results provide insights into the genetic and environmental factors influencing diastolic function that are relevant for identifying causal relationships and potential tractable targets.