With the rise of interest in watch-type and wristband-type devices, wrist-worn devices have become a growing market in the field of wearable activity tracking devices. However, activity tracking with wrist-worn device is more challenging than the task with the other parts of the body, since upper limb movements are uncoupled from the action of walking and reveal irregular movements. This paper proposes a walking speed estimation method with data from a six-axis inertial measurement unit (IMU), which is commonly mounted in wrist-worn devices, and user's height information. The proposed method provides accurate walking speed estimation results under different sensor-carrying modes and walking speeds. The estimation is based on sensor-carrying mode detection with the TreeBagger model, and on Gaussian process regression (GPR) models, which are adapted to seven predetermined sensor-carrying modes. The speed estimation is done by calculating a weighted sum of multiple GPR models with the probabilities from TreeBagger model. To evaluate the superiority of our method, we implement it on a hardware. An experimental evaluation is performed on 18 healthy subjects with a treadmill. The experimental results show that the proposed method outperforms existing studies and comparable commercial devices for all sensing conditions. The averaged error of the proposed method is about 5% for all sensing conditions, while others show error of more than 15% in different sensing conditions. The results show that proposed method provides better accuracy compared to existing studies in terms of estimating walking speed accurately under changing sensing conditions only with single IMU sensor.