LANCAR: Leveraging Language for Context-Aware Robot Locomotion in Unstructured Environments

Robotic locomotion is a challenging task, especially in unstructured terrains. In practice, the optimal locomotion policy can be context-dependent by using the contextual information of encountered terrains in decision-making. Humans can interpret the environmental context for robots, but the ambiguity of human language makes it challenging to use in robot locomotion directly. In this paper, we propose a novel approach that introduces a context translator that works with reinforcement learning (RL) agents for context-aware locomotion. Our formulation allows a robot to interpret the contextual information from environments generated by human observers or Vision-Language Models (VLM) with Large Language Models (LLM) and use this information to generate contextual embeddings. We incorporate the contextual embeddings with the robot's internal environmental observations as the input to the RL agent's decision neural network. We evaluate with contextual information in varying ambiguity levels and compare its performance using several alternative approaches. Our experimental results demonstrate that our approach exhibits good generalizability and adaptability across diverse terrains, by achieving at least 10% of performance improvement in episodic reward over baselines.

@misc{shek2023lancar,
      title={LANCAR: Leveraging Language for Context-Aware Robot Locomotion in Unstructured Environments}, 
      author={Chak Lam Shek and Xiyang Wu and Dinesh Manocha and Pratap Tokekar and Amrit Singh Bedi},
      year={2023},
      eprint={2310.00481},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}