We propose an optical diffuser as a holographic lens exploiting the speckle-correlation scattering matrix with the spatially random property of diffused light. Using the proposed concept, we experimentally demonstrate a holographic camera by simply placing a commercial diffuser in front of an image sensor. We also perform numerical refocusing based upon a single captured hologram of a real object. Our holographic method is simple and compact; and does not require any assumptions, additional reference light, nor multiple measurements, which is the closest realization of the ideal holographic camera.