This article offers a new insight to the achievable stiffness limitation of the impedance control of series elastic actuators (SEAs) by suggesting a comparative analysis of energy ports and passivity control framework at the energy port to enhance the maximum achievable rendered stiffness of SEAs. To this end, it is explored that SEAs have two different port definitions to assess energy and passivity of the system-spring port and load port; and conservatisms for passivity evaluation of two ports are investigated and compared utilizing frequency characteristics. The results reveal that the load port passivity exhibits less conservatism, which allows to render larger achievable stiffness in impedance-controlled SEAs. Moreover, key parameters that determine the passivity characteristic of the SEA impedance control are discovered based on the load port passivity analysis. A novel passivity control framework that incorporates the time-domain passivity observer and the passivity controller is designed utilizing the load port energy monitoring, which offers a less conservative assessment of the systems passivity. Throughout this novel port passivity analysis and the passivity control, it can be achieved to render maximum rendered stiffness of the SEA impedance control much higher than the limitation that has been perceived as the maximum value.