The mechanisms underlying human cell diversity are unclear. Here the authors provide a single-cell epigenome map of human neural organoid development and dissect how epigenetic changes control cell fate specification from pluripotency to distinct cerebral and retina neural types.