Meta Generative Flow Networks with Personalization for Task-Specific Adaptation

arXiv (Cornell University)

Abstract

Multi-task reinforcement learning and meta-reinforcement learning have been developed to quickly adapt to new tasks, but they tend to focus on tasks with higher rewards and more frequent occurrences, leading to poor performance on tasks with sparse rewards. To address this issue, GFlowNets can be integrated into meta-learning algorithms (GFlowMeta) by leveraging the advantages of GFlowNets on tasks with sparse rewards. However, GFlowMeta suffers from performance degradation when encountering heterogeneous transitions from distinct tasks. To overcome this challenge, this paper proposes a personalized approach named pGFlowMeta, which combines task-specific personalized policies with a meta policy. Each personalized policy balances the loss on its personalized task and the difference from the meta policy, while the meta policy aims to minimize the average loss of all tasks. The theoretical analysis shows that the algorithm converges at a sublinear rate. Extensive experiments demonstrate that the proposed algorithm outperforms state-of-the-art reinforcement learning algorithms in discrete environments.

Authors 6

1. Xinyuan Ji Aachen Leiden Institute of Advanced Computer Science

   Leiden University · Xi'an Jiaotong University

   Affiliation as printed

   Faculty of Electronic and Information Engineering , Xi'an Jiaotong University , China

   Leiden Institute of Advanced Computer Science (LIACS) , Leiden University , Netherlands

2. Xu Zhang

   Chinese Academy of Sciences · Academy of Mathematics and Systems Science

   Affiliation as printed

   LSEC , Academy of Mathematics and Systems Science , Chinese Academy of Sciences , China

3. Wei Xi

   Xi'an Jiaotong University

   Affiliation as printed

   Faculty of Electronic and Information Engineering , Xi'an Jiaotong University , China

4. Haozhi Wang

   Tianjin University

   Affiliation as printed

   School of Electrical and Information Engineering , Tianjin University , China

5. Olga Gadyatskaya Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science (LIACS) , Leiden University , Netherlands

6. Yinchuan Li

   Huawei Technologies (China)

   Affiliation as printed

   Noah's Ark Lab , Huawei Technologies , China