This paper presents that the joint torque control capability can be enhanced by adding physical damper to a series elastic actuator (SEA). Joint torque tracking of standard SEA has known limitations that the torque dynamics has an relative order of two, and, as a consequence, the torque controller often requires acceleration feedback when the desired torque is defined by a function of velocity (for example, compliance control). This limitation can be removed by introducing physical damping, reducing the relative degree of torque dynamics by one. Based on this observation, we design a robust controller using the disturbance observer technique. The resulting control law is given by a feed-forward term combined with PI control. The proposed controller is verified in simulation and experiment.