Low cycle fatigue tests have been carried out with three different materials (1Cr-Mo-V steel, 12Cr-Mo-V steel and 304 stainless steel) for the investigation of the effect of surface roughness on the fatigue life. To see the effect systematically, we have chosen those materials which may or may not form grain boundary cavities. Test results show that the continuous fatigue life of 1Cr-Mo-V steel and aged 304 stainless steel with a rough surface is decreased compared with that of the specimens with a smooth surface. These two alloys are found to have no grain boundary cavities formed under creep-fatigue test conditions. On the contrary, the fatigue life of 12Cr-Mo-V steel and solutionized 304 stainless steel in which grain boundary cavities are formed under creep-fatigue test conditions is not influenced by the states of surface roughness. The characteristic test results strongly confirm that the fatigue life of the specimen under creep-fatigue interaction, during which creep cavities are forming, may be controlled by the cavity nucleation and growth processes rather than the process of surface crack initiation.