基于DPPO的双智能体协同渗透测试方法研究
网络安全与数据治理
李科,霍朝宾,贺敏超,杨继
华北计算机系统工程研究所
摘要： 基于强化学习的自动化网络渗透测试方法近年来受到广泛关注，但现有研究多局限于中小规模网络环境，难以应对实际网络中主机数量庞大、拓扑复杂的挑战。提出一种基于DPPO(Dualagent Proximal Policy Optimization)的双智能体协同渗透测试框架，通过分工协作机制提升大规模网络环境下的攻击决策效率。该框架包含目标决策智能体(TargetDecider)与攻击执行智能体(AttackExecutor)，分别负责目标节点筛选与具体漏洞利用执行，二者通过共享环境状态进行协同，以协作奖励机制引导，从而实现分工合作。实验在基于CyberBattleSim构建的仿真网络环境中进行，并与传统的DQN、PPO、SAC等智能体方法进行了比较，结果表明DPPO框架攻陷关键目标所需的步骤数更少、波动更小，显示出更高的攻击效率与策略稳定性。同时，其敏感主机攻陷比率维持在较高水平，表明该框架能可靠完成核心渗透任务。在累计奖励方面，DPPO亦表现出持续优化与稳健收敛的趋势。实验结果表明，该框架能够有效适应大规模静态仿真网络中的渗透测试任务。该框架对动态防御和真实网络环境的适用性仍需在后续研究中进一步验证。
中图分类号：TP393.08文献标志码：ADOI:10.19358/j.issn.2097-1788.2026.06.001中文引用格式：李科,霍朝宾,贺敏超，等.基于DPPO的双智能体协同渗透测试方法研究［J］.网络安全与数据治理，2026，45（6）：1-8.
英文引用格式：Li Ke,Huo Chaobin,He Minchao,et al.Research on dualagent collaborative penetration testing method based on DPPO［J］.Cyber Security and Data Governance，2026，45（6）：1-8.
英文引用格式：Li Ke,Huo Chaobin,He Minchao,et al.Research on dualagent collaborative penetration testing method based on DPPO［J］.Cyber Security and Data Governance，2026，45（6）：1-8.
Research on dual-agent collaborative penetration testing method based on DPPO
Li Ke,Huo Chaobin,He Minchao,Yang Ji
National Computer System Engineering Research Institute of China
Abstract： The automated network penetration testing method based on reinforcement learning has received widespread attention in recent years.However,the existing research is mostly limited to small and mediumsized network environments,making it difficult to address the challenges posed by the vast number of hosts and complex topology in realworld networks.This study proposes a dualagent collaborative penetration testing framework based on DPPO (Dualagent Proximal Policy Optimization),which enhances the efficiency of attack decisionmaking in largescale network environments through a division of labor and collaboration mechanism.The framework comprises a target decisionmaking agent (TargetDecider) and an attack execution agent (AttackExecutor),responsible for target node selection and specific vulnerability exploitation execution,respectively.The two agents collaborate through shared environmental states,guided by a collaborative reward mechanism,to achieve division of labor and cooperation.Experiments were conducted in a simulated network environment built based on CyberBattleSim,and compared with traditional agent methods such as DQN,PPO,and SAC.The results show that the DPPO framework requires fewer steps and exhibits less fluctuation to compromise key targets,demonstrating higher attack efficiency and strategic stability.At the same time,its sensitive host compromise rate remains at a high level,indicating that the framework can reliably complete core penetration tasks.In terms of cumulative reward,DPPO also shows a trend of continuous optimization and robust convergence.The results indicate that the framework can effectively adapt to penetration testing tasks in largescale static simulated networks,while its applicability to dynamic defense scenarios and real network e
Key words : automated penetration testing; dualagent collaboration; deep reinforcement learning
引言
随着网络攻击手段日趋复杂，传统静态防御体系已难以有效应对新型威胁。自动化渗透测试作为主动安全评估的重要手段，能够通过模拟攻击行为提前发现系统脆弱性，但其传统实现模式高度依赖专家经验，存在成本高、适应性差等问题［1］。近年来，强化学习技术因其在序列决策问题上的优势，被引入自动化渗透测试领域，通过将网络环境建模为马尔可夫决策过程［2］，使智能体能够自主学习攻击策略。然而，现有方法在面对大规模复杂网络时仍面临显著挑战：状态与动作空间随网络规模指数增长导致探索效率低下，以及渗透测试任务中奖励信号稀疏导致的收敛困难。
针对上述问题，本文提出一种“目标决策—攻击执行”双智能体协同训练框架。该框架将渗透测试任务分解为宏观目标选择和微观攻击执行两个层次，分别由目标决策智能体(TargetDecider)和攻击执行智能体(AttackExecutor)承担，二者通过状态共享与协同奖励机制实现高效分工协作。本文主要贡献包括：(1)设计了面向大规模网络的双智能体分层强化学习架构，有效降低了学习复杂度；(2)提出了与分工机制相匹配的协同训练方法及分层奖励函数；(3)在CyberBattleSim仿真环境中验证了方法的有效性，实验表明本框架在攻击效率与稳定性方面优于传统单智能体方法。
本文详细内容请下载：
http://www.chinaaet.com/resource/share/2000007120
作者信息：
李科,霍朝宾,贺敏超,杨继
(华北计算机系统工程研究所，北京100083)
此内容为AET网站原创，未经授权禁止转载。