Unmanned vehicles are significant systems in the military, industrial, scientific research, investigation of hazardous environments and rescue fields. Particularly, visual servoing through a visual image has the advantage of ease of use in various fields. Commonly, studies of visual servoing do not consider external environmental factors in most cases. However, in this paper, the proposed visual servoing framework supports any feature vision information which enhances feature-independence as a general visual servoing framework. Furthermore, the visual servoing control framework which includes environment variables can be applied to real-world environments such as the surface of water or rough terrain to perform the docking and following of an object. To adapt to uncertain environments, a controller with a compensator for environmental factors is implemented. Furthermore, applying improved reinforcement learning, the proposed system can operate in highly uncertain environments. Finally, the proposed approach was designed for applicability to an actual system. Thus, the performance was evaluated by applying it to an unmanned surface vehicle and an unmanned ground vehicle.