We study the scattering phase shift of Dirac fermions at graphene edge. We find that when a plane wave of a Dirac fermion is reflected at an edge of graphene, its reflection phase is shifted by the geometric phase resulting from the change of the pseudospin of the Dirac fermion in the reflection. The geometric phase is the Pancharatnam-Berry phase that equals the half of the solid angle on Bloch sphere determined by the propagation direction of the incident wave and also by the orientation angle of the graphene edge. The geometric phase is finite at the zigzag edge in general, while it always vanishes at the armchair edge because of intervalley mixing. To demonstrate its physical effects, we first connect the geometric phase with the energy band structure of graphene nanoribbon with the zigzag edge. The magnitude of the band gap of the nanoribbon, that opens in the presence of the staggered sublattice potential induced by edge magnetization, is related to the geometric phase. Second, we numerically study the effect of the geometric phase on the Veselago lens formed in a graphene nanoribbon. The interference pattern of the lens is distinguished between armchair and zigzag nanoribbons, which is useful for detecting the geometric phase.