Abstract

Reinforcement Learning (RL) and Imitation Learning (IL) offer complementary capabilities: RL can learn high-performing policies but is data-intensive, whereas IL enables rapid learning from demonstrations but is limited by the demonstrator’s quality. Combining them offers the potential for improved sample efficiency in learning high-performing policies, yet naïve integrations often suffer from two fundamental issues: (1) negative transfer, where optimizing the IL loss hinders effective RL fine-tuning, and (2) gradient conflict, where a negative inner product between IL and RL gradients might result in a counter-productive policy update step. Addressing both challenges, we introduce the Conflict-Averse IL-RL (CAIR) framework. Given IL and RL baseline algorithms, CAIR combines them through two components: (1) Loss Manipulation: adaptive weighting of the IL and RL losses targeting maximal RL weight subject to a capped gradient misalignment between the policy update direction and the IL gradient. (2) Gradient Manipulation: Conflict-Averse Gradient Descent (CAGrad) adjusts the resulting policy update step to balance the normalized predicted first-order decreases of both objectives (IL and RL). For an idealized stagewise optimization process, we establish conditional monotonic non-deterioration in expected return between successive stages. We evaluate CAIR in (1) sparse-reward robotic manipulation (Fetch and Adroit), where existing RL algorithms (without demonstrations) struggle to converge, and (2) dense-reward continuous control locomotion (Gymnasium MuJoCo), where existing RL algorithms (without demonstrations) can converge effectively yet demonstrations can be used to improve sample efficiency. We evaluate CAIR over various IL (BC, DAgger) and RL (DDPG, SAC, PPO) combinations. The results show that the BC+SAC variant and DAgger+DDPG variant attain the highest average normalized reward among the baseline algorithms in the sparse-reward and dense-reward domains respectively. Evaluations across IL and RL combinations demonstrate CAIR’s applicability to diverse IL–RL pairings, with pairing- and domain-dependent performance and no consistently dominant combination.