Let M be a positive quaternionic kahler manifold of dimension 4m. In earlier papers, Fang and the first author showed that if the symmetry rank is greater than or equal to [m/2]+ 3, then M is isometric to HPm or Gr(2)(Cm+2). The goal of this paper is to give a more refined classification result for positive quaternionic Kahler manifolds (in particular, of relatively low dimension or with even m) whose fourth Betti number equals one. To be precise, we show in this paper that if the symmetry rank of M with b(4)(M) = 1 is no less than [m/2] + 2 for in >= 5, then M is isometric to HPm.