A hierarchical spatio-temporal reinforcement learning framework for cooperative multi-USV search and interception

Multi-USV cooperative search and target interception holds value in maritime security, yet existing methods face three challenges: multi-timescale decision coupling, insufficient heterogeneous observation fusion, and inefficient exploration under sparse rewards. This paper proposes a multi-USV decision-making framework based on Hierarchical Spatio-Temporal Proximal Policy Optimization (HST-PPO) with three contributions. A three-layer spatio-temporal decision architecture spanning strategic, tactical, and execution layers is coupled via soft goal passing and trained end-to-end. A Hierarchical Causal Observation Fusion Network (HCOF-Net) uses causal mask attention and temporal decay encoding to adaptively fuse four categories of observations. A Distributed Random Network Distillation exploration mechanism (DRND-NovelD) combines ensemble novelty estimation with a multi-agent diversity reward to accelerate convergence under sparse rewards. HST-PPO is validated across Python simulation, Unity high-fidelity simulation, and real-world lake trials, and benchmarked against ten baselines spanning six non-hierarchical MARL methods and four hierarchical RL methods adapted to the same setting. HST-PPO achieves a Search Coverage Rate of 82.3% and Interception Success Rate of 80.0%, surpassing the strongest baseline by 1.7 and 1.1 percentage points and outperforming every hierarchical baseline. In real-world trials, HST-PPO retains 96.2% and 81.3% of its simulated Search Coverage Rate and Interception Success Rate, validating its zero-shot sim-to-real transfer capability.

• Record URL:
  • https://doi.org/10.1016/j.oceaneng.2026.128026
• Record URL:
  • http://www.sciencedirect.com/science/article/pii/S0029801826038606
• Availability:
  • Find a library where document is available. Order URL: http://worldcat.org/issn/00298018
• Supplemental Notes:
  • © 2026 Elsevier Ltd. All rights are reserved, including those for text and data mining, AI training, and similar technologies. Abstract reprinted with permission of Elsevier.
• Authors:
  • Liu, Wang
  • 0009-0003-5235-5567
  • Xu, Xiaowei
  • 0000-0003-0577-6337
  • Deng, Mingxing
  • 0009-0005-9559-3105
  • Qi, Qinghua
  • 0000-0002-8136-1275
• Publication Date: 2026-11-15

Language

• English

Media Info

• Media Type: Web
• Features: References;
• Pagination: 128026
• Serial:
  • Ocean Engineering
  • Volume: 368
  • Issue Number: 0
  • Publisher: Pergamon
  • ISSN: 0029-8018
  • EISSN: 1873-5258
  • Serial URL: http://www.sciencedirect.com/science/journal/00298018

Subject/Index Terms

• TRT Terms: Autonomous ships; Data fusion; Decision making; Machine learning; Security
• Subject Areas: Data and Information Technology; Marine Transportation; Vehicles and Equipment;

Filing Info

• Accession Number: 02004881
• Record Type: Publication
• Files: TRIS
• Created Date: Oct 2 2026 4:54PM