Composing Task Knowledge With Modular Successor Feature Approximators

ICLR 2023

Wilka Carvalho1, Angelos Filos2, Richard L. Lewis1, Honglak Lee1,3, Satinder Singh1
University of Michigan1, University of Oxford2, LG AI Research3
arXiv Code

Abstract

Recently, the Successor Features and Generalized Policy Improvement (SF&GPI) framework has been proposed as a method for learning, composing and transferring predictive knowledge and behavior. SF&GPI works by having an agent learn predictive representations (SFs) that can be combined for transfer to new tasks with GPI. However, to be effective this approach requires state features that are useful to predict, and these state-features are typically hand-designed. In this work, we present a novel neural network architecture, “Modular Successor Feature Approximators” (MSFA), where modules both discover what is useful to predict, and learn their own predictive representations. We show that MSFA is able to better generalize compared to baseline architectures for learning SFs and a modular network that discovers factored state representations.

Interpolate start reference image.

Prior work: Feature-Attending Recurrent Modules (FARM, 1) learns multiple state modules. This enables discovering useful state features in different modules and promotes generalization to novel environment configurations. However, it has no mechanism for combining task solutions. Universal Successor Feature Approximators (USFA, 2) learns a single monolothic architecture for predicing SFs and can combine task solutions. However, it relies on hand-designed state features and has no mechanism for generalization to novel environment configurations.

Modular Successor Feature Approximators: We combine the benefits of both We have modules that discover their own state features and learn their own SFs. This enables combining task solutions in the face of novel environment configurations.

BibTeX


    @article{carvalho2023composing,
      author    = {Carvalho, Wilka and Filos, Angelos and Lewis, Richard L. and Lee, Honglak and Singh, Satinder},
      title     = {Composing Task Knowledge With Modular Successor Feature Approximators},
      journal   = {ICLR},
      year      = {2023},
      }