We found that Ce on the Si(111) system exhibited nonmetallic behavior at a critical coverage of 4.0 ML. We observed a large binding energy shift of 2.5 eV in x-ray photoelectron spectroscopy (XPS), and a band gap of 0.6 eV in scanning tunneling spectroscopy (STS), whereas normal metallic behavior was observed at other coverages. Moreover, a strong downward surface relaxation was observed at this critical coverage with increased surface roughness. We expect that this phase transition at a critical coverage is due to the breakdown of bulk modulus as a result of the release of tensile stress along the in-plane direction.