This paper proposes a practical and less conservative passivity analysis for series elastic actuators (SEAs) by introducing load port definition and shows that the achievable stiffness by the impedance control of SEA can be set higher than the inherent stiffness of SEA depending on the condition of the load dynamics. Since SEA can inherently measure or estimate a transmitted force thanks to its embedded spring element, impedance control is often exploited to render compliant behaviors related between the motion and the force. Although the stability of the SEA control system is of great importance, the conventional passivity analysis gives conservative criteria, and indeed limits the actual actuator performance. To tackle the conservatism of the conventional passivity in SEAs, we first explore the dynamic characteristics of SEA including load dynamics, which has been ignored for the sake of simplicity of the passivity analysis by excluding uncertain load dynamics. The inclusion of the load dynamics into the passivity analysis allows us to properly derive the less conservative limit of achievable stiffness by impedance control and the factors that determine the limit. The proposed analysis is verified by numerical simulations and applied to a passivity observer design for experimental validation on an actual SEA setup.