A recently proposed method for stabilizing haptic interfaces and teleoperation systems was tested with a “PHANToM” commercial haptic device. The Passivity Observer (PO) and Passivity Controller (PC) stabilization method was applied to stabilize the system but also excited a high frequency mode in the device. To solve this problem, we propose a method to use a time-varying desired energy threshold instead of fixed zero energy threshold for the PO, and make the actual energy input follow the time-varying energy threshold. With the time-varying energy threshold, we make the PC control action smooth without sudden impulsive behavior by distributing the dissipation. The proposed new PO/PC approach is applied to PHANToM with high stiffness (K = 5000N/m), and stable and smooth contact is guaranteed. Resetting and active environment display problems also can be solved with the reference energy following idea.