Open-World Mobile Manipulation

Abstract

Deploying robots in open-ended unstructured environments such as homes has been a long-standing research problem. However, robots are often studied only in closed-off lab settings, and prior mobile manipulation work is restricted to pick-move-place, which is arguably just the tip of the iceberg in this area. In this paper, we introduce Open-World Mobile Manipulation System, a full-stack approach to tackle realistic articulated object operation, e.g. real-world doors, cabinets, drawers, and refrigerators in open-ended unstructured environments. We propose an adaptive learning framework in which the robot initially learns from a small set of data through behavior cloning, followed by learning from online self-practice on novel variations that fall outside the BC training domain. We develop a low-cost mobile manipulation hardware platform capable of repeatedly safe and autonomous online adaptation in unstructured environments with a cost of around 20k USD. We conducted a field test on 20 novel doors across 4 different buildings on a university campus. In a trial period of less than one hour, our system demonstrated significant improvement, boosting the success rate from 50% of BC pre-training to 95% of online adaptation without any human intervention.

Online Adaptation.

We start by initializing our policy using a limited set of training data through behavior cloning. The central challenge we face is operating with new objects that fall outside the training domain. To address this, we've developed a system capable of fully autonomous Reinforcement Learning (RL) adaptation, utilizing a Vision-Language Model (VLM) for reward function In a trial period of less than one hour, this system demonstrated significant improvement, boosting the success rate from 50% to 95% without any human intervention.

All videos play at 1x speed. Test adapted policies.

Media

新智元

BibTeX


@article{xiong2024adaptive,
  title={Adaptive Mobile Manipulation for Articulated Objects In the Open World},
  author={Xiong, Haoyu and Mendonca, Russell and Shaw, Kenneth and Pathak, Deepak},
  journal={arXiv preprint arXiv:2401.14403},
  year={2024}
}