Autonomous ship berthing assisted by tugboats using reinforcement learning

Abstract

The support of a tugboat is essential for large vessels with limited maneuverability at low speeds in precise operations such as berthing. The berthing operation using a tugboat is a highly difficulty operation in which the tugboat captain manipulates the tugboat according to the instructions of the pilot, and requires many highly trained personnel such as pilots and towing personnel. Recently, the application of automation and autonomy technology has been attempted to these difficult problems. Thus, automation of tugboats is a challenging research topic. To automate the tugboat, the tugboat for the tugboat operation must be accurately modelled. In this research, the tugboat dynamic was defined as a hybrid system in which continuous time space and non-continuous time space interact in order to represent the operation time and control problems of the tugboat simultaneously. Since these complex hybrid system problems are difficult to solve with typical algorithms, reinforcement learning was applied. The reinforcement learning algorithm used for this problem was a proximal policy optimization algorithm, one of the probability policy gradient algorithms, and it has been widely used in recent reinforcement learning research. In this research, the problem was solved by simulating an actual port environment. Three scenarios were conducted to demonstrate the general possibility of solving the autonomous berthing problem using reinforcement learning: the berthing problem with the interaction between the towing and berthing modes of the tugboat, the problem of the ship’s starboard-berthing with the interaction between the self-propelled and the berthing modes, and finally, the ship’s port-berthing problem with the interaction between the self-propelled and the berthing modes. Through applying reinforcement learning to various berthing problems, this research demonstrated the performance and usefulness of the proposed algorithm through simulation.