Using a Green's function method based on an elastic wave equation, the effects of surface roughness and the nanowire-contact interface scattering on phonon thermal conductivity are studied at low temperatures. It is found that the interface geometry between a nanowire and its contacts affects the transmission function at small energies related to the gapless modes and it gives rise to deviated results from the universal conductance. It is also shown that the surface roughness is crucial in the suppression of phonon thermal conductivity with reducing the nanowire size by averaging the transmission function over the rough-surface configurations. Furthermore, the phonon mean free path is proportional to the ratio of the correlation length and roughness heights quadratically as well as the cross-section area of the nanowire. (C) 2012 American Institute of Physics. [doi:10.1063/1.3684973]