The apolipoprotein E ε4 gene is the most important genetic risk factor for sporadic Alzheimer's disease, but the link between this gene and neurodegeneration remains unclear. Using array tomography, we analysed >50000 synapses in brains of 11 patients with Alzheimer's disease and five non-demented control subjects and found that synapse loss around senile plaques in Alzheimer's disease correlates with the burden of oligomeric amyloid-β in the neuropil and that this synaptotoxic oligomerized peptide is present at a subset of synapses. Further analysis reveals apolipoprotein E ε4 patients with Alzheimer's disease have significantly higher oligomeric amyloid-β burden and exacerbated synapse loss around plaques compared with apolipoprotein E ε3 patients. Apolipoprotein E4 protein colocalizes with oligomeric amyloid-β and enhances synaptic localization of oligomeric amyloid-β by >5-fold. Biochemical characterization shows that the amyloid-β enriched at synapses by apolipoprotein E4 includes sodium dodecyl sulphate-stable dimers and trimers. In mouse primary neuronal culture, lipidated apolipoprotein E4 enhances oligomeric amyloid-β association with synapses via a mechanism involving apolipoprotein E receptors. Together, these data suggest that apolipoprotein E4 is a co-factor that enhances the toxicity of oligomeric amyloid-β both by increasing its levels and directing it to synapses, providing a link between apolipoprotein E ε4 genotype and synapse loss, a major correlate of cognitive decline in Alzheimer's disease.