A recently proposed method for stabilizing haptic interfaces and teleoperation systems was tested with a PHANTOM commercial haptic device. The passivity observer (PO) and passivity controller (PC) stabilization method was applied to stabilize the system but also excited a high-frequency mode in the device. To solve this problem, we propose a method to use a time-varying desired energy threshold instead of fixed zero energy threshold for the PO, and make the actual energy input follow the time-varying energy threshold. With the time-varying energy threshold, we make the PC control action smooth without sudden impulsive behavior by distributing the dissipation. The proposed new PO/ PC approach is applied to PHANTOM with high stiffness (K = 5000 N/m), and stable and smooth contact is guaranteed. Resetting and active environment display problems also can be solved with the reference energy following idea.