LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

Manith Adikari 1,4 , Bei Peng 2 , Samuele Vinanzi 3 , Angelo Cangelosi 1,4 Corresponding Author: manith.adikari@manchester.ac.uk

Abstract Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.

1 Introduction

Reinforcement Learning (RL) has achieved remarkable success in training autonomous agents, from game-playing (Mnih et al. 2013 ) to robotics (Tan et al. 2018 ) . By formalizing learning as the maximization of cumulative rewards through trial and error learning (Sutton and Barto 1998 ) , RL provides a natural framework for training autonomous ‘goal-seeking’ agents (McCarthy 1997 ) . However, standard RL relies on two critical assumptions: that the goal can be represented by a single scalar reward , and that this reward function is well-specified . In practice, these assumptions rarely hold in complex, real-world domains.

Real-world tasks often involve multiple, competing objectives (Dulac-Arnold et al. 2019 ) , such as balancing speed versus safety in autonomous driving (Wang et al. 2026 ) , or throughput versus energy efficiency in robotics (Huang et al. 2022 ; Kouritem et al. 2022 ) .

The field of Multi-Objective Reinforcement Learning (MORL) addresses this by modeling rewards as vectors to find a set of Pareto-optimal policies (Roijers et al. 2013 ) : policies where no objective’s expected returns can be increased without decreasing the returns of other objectives. However, existing MORL approaches typically assume that the ground-truth reward function for each objective is accessible and manually specified (Hayes et al. 2022b ) . Thus, they inherit the same challenges of reward specification from the single-objective RL domain, now across multiple objectives. Manually designing a reward function that can achieve an adequate balance between competing objectives is challenging and can lead to oversimplification, which results in suboptimal policies (Knox et al. 2012 ) and possible reward exploitation (Amodei et al. 2016 ) . To avoid complicated reward engineering, Preference-based RL (PbRL) learns reward models directly from human feedback (Christiano et al. 2017 ) , which leads to better alignment between the system’s behavior and human preferences. Despite progress in reward learning for single-objective RL, this problem remains largely unexplored in the MORL setting. While some works consider reward learning in MORL, they are typically limited to Large Language Model (LLM) post-training settings or narrow natural language-based tasks (Bakker et al. 2022 ; Rame et al. 2023 ; Yang et al. 2024 ) , and do not study the joint learning of policies and rewards in environments with multiple, competing objectives.

Reward specification is already a major bottleneck in scaling single-objective RL settings, and becomes even more critical in MORL. Two key challenges arise. First, the agent must optimize multiple objectives whose underlying reward functions are complex, implicit, or inaccessible. Second, even when human feedback is available, collapsing multi-criteria reward signals into a single scalar obscures the very trade-off structure that MORL is designed to optimize (Vamplew et al. 2011 ; Roijers et al. 2013 ; Sorensen et al. 2024 ) . Consider training a robotic system: non-expert operators can reliably judge task-level success, such as whether objects were grasped or placed correctly, while expert operators are required to assess fine-grained criteria such as grasp stability, long-term wear, or safety-related objectives. Collapsing such heterogeneous feedback into a single reward risks conflating distinct objectives and diluting expert signals, motivating the need for separate , objective-specific reward models learned from the appropriate source of feedback.

This creates a crucial gap: How can agents learn optimal trade-offs between conflicting objectives when the reward functions are unknown and must be inferred from feedback?

To address this gap, we introduce LEMUR : Le arning to Align with Mu lti-Objective R einforcement Learning from Preference Feedback, a framework for learning to balance multiple objectives without pre-defined reward functions. Our approach jointly learns policies and multiple objective-specific reward models from preference feedback, enabling agents to effectively balance competing objectives during learning. By removing the assumption of known reward functions and explicitly modeling multiple objectives, our method tackles a key challenge to scaling MORL in real-world, human-aligned domains. To summarize, our main contributions are:

• We propose a novel framework, LEMUR, which learns multiple objective-specific reward models from preference feedback and then optimizes policies against these reward models using multi-objective reinforcement learning. This enables agents to effectively solve multi-objective decision-making tasks without access to pre-defined reward functions, while naturally accommodating heterogeneous sources of feedback, where different annotators may hold expertise over different objectives.

• Our extensive experiments demonstrate that LEMUR outperforms baselines across a range of benchmark multi-objective environments, namely high-dimensional continuous control tasks, and further show LEMUR’s robustness to label noise, constrained feedback budgets, and scaling to more objectives.

Figure 1: Illustration of our framework LEMUR: (1) Unsupervised Pre-training for the MORL agent to explore and collect diverse experiences via maximizing state entropy H(s). (2) Reward learning from Preference feedback, where each reward model is learned separately from the preferences queried from each teacher. The reward models are used to dynamically relabel the state-action pairs as a reward vector for each objective (i.e., each teacher’s preferences). (3) Multi-Objective RL agent denoted by π ϕ \pi_{\phi} uses each of the trained reward models to do multi-objective policy optimization to maximize the expected vector rewards.

2 Preliminaries

Multi-Objective Reinforcement Learning. We formulate the problem as a Multi-Objective Markov Decision Process (MOMDP) (White 1982 ) , defined by the tuple ⟨ 𝒮 , 𝒜 , 𝒯 , γ , 𝐫 ⟩ \langle\mathcal{S},\mathcal{A},\mathcal{T},\gamma,\mathbf{r}\rangle . Here, 𝒮 \mathcal{S} and 𝒜 \mathcal{A} denote state and action spaces respectively, 𝒯 \mathcal{T} the transition dynamics, and γ ∈ [ 0 , 1 ) \gamma\in[0,1) the discount factor. Unlike standard RL, the reward is a vector 𝐫 ​ ( s , a ) ∈ ℝ m \mathbf{r}(s,a)\in\mathbb{R}^{m} comprising m m distinct objectives. The agent’s goal is to maximize the expected discounted vector returns J ​ ( π ) = 𝔼 π ​ [ ∑ t = 0 ∞ γ t ​ 𝐫 ​ ( 𝐬 t , 𝐚 t ) ] J(\pi)=\mathbb{E}{\pi}[\sum{t=0}^{\infty}\gamma^{t}\mathbf{r}(\mathbf{s}{t},\mathbf{a}{t})] . A policy π \pi maps states to action distributions. Since no single policy typically maximizes all objectives in MORL, the agent learns a set of policies Π \Pi representing optimal trade-offs, defined by Pareto dominance (Hayes et al. 2022a ) . A policy π \pi dominates π ′ \pi^{\prime} (denoted J ​ ( π ) ≻ J ​ ( π ′ ) J(\pi)\succ J(\pi^{\prime}) ) if it is superior in at least one objective and no worse in others. The solution set is the Pareto Frontier ℱ = { π ∈ Π ∣ ∄ ​ π ′ ∈ Π : J ​ ( π ′ ) ≻ J ​ ( π ) } \mathcal{F}={\pi\in\Pi\mid\nexists\pi^{\prime}\in\Pi:J(\pi^{\prime})\succ J(\pi)} .

Soft Actor-Critic (SAC). SAC (Haarnoja et al. 2018 ) is an off-policy actor-critic algorithm grounded in the maximum entropy framework. It augments standard single-objective RL with an entropy term to encourage exploration. The agent aims to maximize J ( π ) = 𝔼 π [ ∑ t γ t ( r t + α ℋ ( π ( ⋅ | 𝐬 t ) ) ) ] J(\pi)=\mathbb{E}{\pi}[\sum{t}\gamma^{t}(r{t}+\alpha\mathcal{H}(\pi(\cdot|\mathbf{s}{t})))] .

Reward Learning from Preference Feedback. We follow the standard Preference-based RL (PbRL) framework, which uses ‘latent rewards’ as a proxy for values (Christiano et al. 2017 ) . Each human’s reward function is learned independently using their respective preference feedback. Similar to prior work, we use preferences over pairs of trajectory segments, ( σ 0 , σ 1 ) (\sigma^{0},\sigma^{1}) . An expert teacher (e.g., human) provides a label y ∈ { 0 , 1 , 0.5 } y\in{0,1,0.5} to indicate their preference. To learn a reward function r ^ \hat{r} parameterized by ψ \psi , we employ the Bradley-Terry model (Bradley and Terry 1952 ) , modeling the preference probability as P ​ [ σ 1 ≻ σ 0 ; ψ ] = exp ​ ∑ r ^ ​ ( σ 1 ) exp ​ ∑ r ^ ​ ( σ 1 ) + exp ​ ∑ r ^ ​ ( σ 0 ) P[\sigma^{1}\succ\sigma^{0};\psi]=\frac{\exp\sum\hat{r}(\sigma^{1})}{\exp\sum\hat{r}(\sigma^{1})+\exp\sum\hat{r}(\sigma^{0})} . Given a dataset 𝒟 \mathcal{D} , the reward function is trained by minimizing the cross-entropy loss:

ℒ C ​ E ​ ( ψ ) = − 𝔼 𝒟 ​ [ ( 1 − y ) ​ log ⁡ P ​ [ σ 0 ≻ σ 1 ] + y ​ log ⁡ P ​ [ σ 1 ≻ σ 0 ] ] . \mathcal{L}^{CE}(\psi)=-\mathbb{E}_{\mathcal{D}}\Big[(1-y)\log P[\sigma^{0}\succ\sigma^{1}]+y\log P[\sigma^{1}\succ\sigma^{0}]\Big]. (1)

The learned reward function can then be used to update the policy with any RL algorithm to maximize expected returns.

3 Problem Setup

In this section, we present our formulation of Multi-Objective Reinforcement Learning (MORL) without pre-defined reward functions for multiple, conflicting objectives.

Latent Reward Vector.

In standard MORL, the reward function is a known vector 𝐫 ​ ( s , a ) ∈ ℝ m \mathbf{r}(s,a)\in\mathbb{R}^{m} . However, in our work, the agent does not have access to the ground-truth reward function. Instead, we assume the existence of m m distinct multiple, conflicting human users (objectives), where each dimension r i r{i} corresponds to the latent reward function of the i i -th specific user. Since these rewards are inaccessible, we must approximate them. We define a parameterized reward vector 𝐫 ^ ψ ​ ( s , a ) = [ r ^ ψ 1 ​ ( s , a ) , … , r ^ ψ m ​ ( s , a ) ] T \mathbf{\hat{r}}{\psi}(s,a)=[\hat{r}{\psi{1}}(s,a),\dots,\hat{r}{\psi{m}}(s,a)]^{T} , where each component is a reward model learned from human preference feedback (Christiano et al. 2017 ) .

Multi-Objective RL Optimization.

The agent’s goal is to maximize the expected discounted vector returns. We adopt the most prevalent MORL formalism called the utility-based approach (Roijers et al. 2013 ) , where we define a scalarization function f 𝐰 ​ ( 𝐫 ) = 𝐰 ⊤ ​ 𝐫 f{\mathbf{w}}(\mathbf{r})=\mathbf{w}^{\top}\mathbf{r} , where 𝐰 ∈ ℝ m \mathbf{w}\in\mathbb{R}^{m} is a preference weight vector on the simplex (i.e., ∑ w i = 1 \sum w{i}=1 ). Thus, we define an optimal MORL agent to be the policies belonging to the Convex Coverage Set (CCS), the subset of ℱ \mathcal{F} optimal for linearly scalarized preferences (Roijers et al. 2013 ) .

To summarize, the MORL agent learns these policies by maximizing the expected returns of the learned latent reward vector linearly scalarized by the weight w {w} , which determines the trade-offs between objectives:

J ​ ( π ) = 𝔼 π ​ [ ∑ t = 0 ∞ γ t ​ 𝐰 T ​ 𝐫 ^ ψ ​ ( s t , a t ) ] . J(\pi)=\mathbb{E}{\pi}\left[\sum{t=0}^{\infty}\gamma^{t}\mathbf{{w}}^{T}\mathbf{\hat{r}{\psi}}(s{t},a_{t})\right]. (2)

4 LEMUR

LEMUR ( Le arning to Align with Mu lti-Objective R einforcement Learning from Preference Feedback), illustrated in Figure 1 proceeds in three stages: (1) unsupervised pre-training, where the agent explores via intrinsic rewards to collect diverse experiences (Section 4.1 ); (2) reward learning, where multiple human teachers are queried for preference feedback to train objective-specific reward models (Section 4.2 ); and (3) multi-objective RL training against the learned reward models (Section 4.3 ). Stages 2 and 3 repeat, continually improving both the reward models and the multi-objective policies. Full pseudocode is provided in Appendix B .

4.1 Unsupervised Pre-training

Standard PbRL suffers from uninformative queries caused by the limited coverage of random initialization. To generate informative queries, LEMUR employs an unsupervised pre-training phase driven by intrinsic motivation (Lee et al. 2021b ) . We encourage exploration by maximizing state entropy, approximated via a particle-based k k -nearest neighbors ( k k -NN) estimator (Liu and Abbeel 2021 ) . The intrinsic reward r i ​ n ​ t ​ ( s t ) = log ⁡ ( ‖ s t − s t k ‖ ) r^{int}(s{t})=\log(|s{t}-s{t}^{k}|) is the normalized distance to the k k -th nearest neighbor in ℬ \mathcal{B} , and the agent maximizes 𝒥 i ​ n ​ t ​ ( ϕ ) = 𝔼 π ϕ ​ [ ∑ t = 0 T γ t ​ r i ​ n ​ t ​ ( s t ) ] \mathcal{J}{int}(\phi)=\mathbb{E}{\pi{\phi}}[\sum{t=0}^{T}\gamma^{t}\textbf{r}^{int}(s{t})] . This populates the buffer with diverse behaviors, accelerating the subsequent multi-objective reward learning.

4.2 Reward learning of Multiple Objectives from Preferences

A core challenge in our setup is that the agent does not have access to the ground-truth rewards, and the m m conflicting objectives are characterized instead by the conflicting preferences of m m humans. Following the PbRL formulation in Section 2 , LEMUR learns a separate reward model 𝐫 ^ ψ j ​ ( s , a ) \hat{\mathbf{r}}{\psi{j}}(s,a) per teacher, trained by minimizing the cross-entropy loss between the model’s predictions and that teacher’s labels (Equation 1 ).

Weight-Conditioned Reward Models. Rather than learning each teacher’s reward in isolation, we condition every objective-specific model on the shared objective space. Each teacher j j is assigned a reward model 𝐫 ^ ψ j ​ ( s , a ) \hat{\mathbf{r}}{\psi{j}}(s,a) , a lightweight MLP predicting the full objective vector, whose scalar utility is obtained by projecting onto that teacher’s preference anchor, r ^ j ​ ( s , a ) = 𝐚 j ⊤ ​ 𝐫 ^ ψ j ​ ( s , a ) \hat{r}{j}(s,a)=\mathbf{a}{j}^{\top}\hat{\mathbf{r}}{\psi{j}}(s,a) . This couples the m m learned models to a common vector-reward structure, so that a policy conditioned on 𝐰 \mathbf{w} reads a consistent per-teacher utility at inference, and the reward models remain directly comparable as the number of conflicting teachers grows. We deliberately adopt this simple architecture, as in prior approaches (Mu et al. 2025 ) ; we find it sufficient to recover strong compromise policies while keeping reward learning fast enough to remain in the loop with online policy optimization.

Query Sampling Strategy. We sample trajectory pairs ( σ 0 , σ 1 ) (\sigma^{0},\sigma^{1}) uniformly at random from the buffer ℬ \mathcal{B} , so that queries span the diverse state-action distributions explored by all policies. While more sophisticated disagreement-based strategies exist, uniform sampling offers simplicity and avoids bias toward particular regions of the objective space during early training.

4.3 Multi-Objective RL Training

Given the parameterized reward vector 𝐫 ^ ψ ​ ( s , a ) \mathbf{\hat{r}}_{\psi}(s,a) , LEMUR trains the MORL agent to maximize expected latent vector rewards (Equation 2 ). For policy optimization we leverage MORL/D, a state-of-the-art Multi-Objective Soft Actor-Critic (MO-SAC) algorithm (Felten et al. 2024 ) which learns a set of independent SAC policies and applies an evolutionary strategy for policy search. This off-policy choice is deliberate: reusing past experience from the replay buffer is essential for sample efficiency under a limited human feedback budget.

Weight Vector Initialization and Adaptation. The scalarization weight vectors { 𝐰 } {\mathbf{w}} determine the trade-offs between objectives. Given 𝐰 ∈ ℝ m \mathbf{w}\in\mathbb{R}^{m} , we optimize for policies using SAC (Haarnoja et al. 2018 ) on the scalarized reward of Equation 2 . We employ a Pareto Simulated Annealing (PSA) approach similar to (Felten et al. 2024 ) , adapting the weight vector in response to the current policies and their distance to non-dominated solutions, allowing the agent to focus training on feasible regions of the objective space while maintaining policy diversity.

Cooperation via Shared Buffer. To facilitate information exchange across policies learning different trade-offs, all policies store and sample from a common replay buffer ℬ \mathcal{B} . This enables policies to learn from diverse experiences collected under different preference weightings, improving sample efficiency, a critical consideration given the limited human feedback budget.

Relabeling of Vector Rewards . Combining off-policy RL with a reward function learned from preferences introduces non-stationarity: as the reward model is updated with new feedback, rewards associated with past transitions in the buffer become stale, destabilizing learning. To address this, LEMUR employs a vector reward relabeling strategy inspired by prior work (Lee et al. 2021b ) . Rather than storing rewards, we store only the transitions, and compute vector rewards on the fly when a batch is sampled, using the most up-to-date reward models. This ensures the agent always trains on updated rewards, synchronizing policy and reward learning while preserving the sample efficiency of our off-policy approach.

5 Experiments

Our experiments address three questions: (1) Can LEMUR learn multi-objective policies that balance multiple learned reward models from multiple teachers? (2) How does LEMUR compare to existing baselines on multi-objective benchmarks? (3) Does explicitly learning multiple reward models for conflicting feedback outperform aggregating feedback into a single reward model? For all experiments, we report the mean across five random seeds with standard error. Additional implementation details are reported in Appendices D & E .

Benchmark Environments & Setup. We evaluate LEMUR on high-dimensional environments from the MORL-Generalization benchmark (Teoh et al. 2025 ) . Following standard practice in PbRL (Lee et al. 2021a ; Christiano et al. 2017 ) , we use scripted teachers that generate feedback according to the components of the ground-truth vector reward, enabling quantitative evaluation; the ground-truth rewards remain inaccessible to the agent, which must jointly learn the conflicting preferences and optimize to find a balance. Our main experiments use two conflicting teachers, the fundamental version of the problem; Section 5.1 demonstrates scaling to more objectives. We evaluate on: MO-Lunarlander , where Teacher A rewards precise, stable landings and Teacher B prioritizes fuel conservation; MO-Hopper and MO-Cheetah , continuous control locomotion tasks where Teacher A prefers fast locomotion and Teacher B prefers slow, energy-efficient gaits; and MO-MetaWorld (Drawer-Close), a robotic-manipulation task from the Meta-World suite (Yu et al. 2020 ) . Meta-World tasks are natively single-objective; we convert Drawer-Close into a two-objective task by pairing the native task-progress reward (Teacher A) with a control-effort penalty (Teacher B), mirroring the reward decomposition standard in the MORL benchmark suite (Teoh et al. 2025 ) . Full environment details are given in Appendix G .

Figure 2: Learning curves on all benchmark environments: MO-LunarLander , MO-Hopper , MO-Cheetah , and MO-MetaWorld . Curves depict the true objective returns (inaccessible to the agent), averaged across five seeds, with shaded regions representing standard error. LEMUR (blue) most closely tracks the Oracle (red) on both objectives simultaneously. Baselines that aggregate conflicting feedback ( Naive , Utilitarian ) fail to make progress, while the external baselines ( MORAL , PbMORL , FPbRL ) learn but consistently trail LEMUR.

Baselines. We compare against five baselines spanning distinct strategies for preference aggregation and learning from multiple objectives: (1) a Utilitarian agent, a single SAC agent optimizing the arithmetic mean of the independently learned rewards; (2) Naive data pooling, which trains one monolithic reward model on all conflicting feedback, akin to standard Reinforcement Learning from Human Feedback (RLHF); (3) MORAL (Peschl et al. 2022 ) , which recovers per-teacher rewards via Adversarial Inverse Reinforcement Learning (AIRL) and learns a scalarization over them; (4) PbMORL (Mu et al. 2025 ) , a recent preference-based multi-objective method learning a weight-conditioned vector reward from pairwise feedback; and (5) FPbRL (Siddique et al. 2023 ) , which aggregates learned per-teacher rewards through a Generalized Gini Welfare scalarization to optimize for fairness. We additionally report an (6) Oracle trained on ground-truth rewards as an upper bound. Unless otherwise noted, every baseline shares LEMUR’s interactive learning loop, reward-model architecture, pre-training stage, teacher weight vectors, query budget, and environment-step budget; the primary distinction lies in how conflicting reward signals are aggregated and optimized . Where a baseline’s original policy optimizer would disadvantage it in our environments, we adapt in the baseline’s favour; all deviations are disclosed in Appendix E .

5.1 Results & Analysis

Figure 2 presents the learning curves for all methods across the benchmark environments. Across every environment, LEMUR is the method that most closely tracks the Oracle on both objectives simultaneously, effectively recovering policies that balance multiple, conflicting objectives. The Utilitarian and Naive agents remain flat with suboptimal returns throughout, supporting our hypothesis that aggregating conflicting reward signals into a single scalar degrades performance in such multi-objective settings.

Figure 3: Task Success Rate (%) learning curves on MetaWorld (Drawer Close Task).

While MORAL improves early in LunarLander and Hopper, it fails to sustain this progress: in MO-Hopper its returns peak mid-training and then steadily declines. This is consistent with the known failure of out-of-distribution optimization due to static reward models; MORAL infers its per-teacher rewards offline from expert demonstrations via AIRL and holds them fixed. LEMUR instead does online policy optimization and addresses non-stationarity through vector reward relabeling .

PbMORL and FPbRL, which both learn vector rewards from preference feedback, perform better than the aggregation baselines, yet still fall short of LEMUR. We attribute this to the assumptions both methods inherit. PbMORL trains a single weight-conditioned reward model over pooled feedback, implicitly assuming all preferences originate from one single teacher; under conflicting teachers the pooled model must average over conflicting labels, degrading the reward signal, most visibly in MO-Hopper and MO-Cheetah where it consistently trails LEMUR on both objectives. FPbRL preserves the vector structure but employs a fixed Generalized Gini welfare scalarization a priori, converging to a single welfare-optimal policy rather than a set of trade-offs: it achieves reasonable returns on MetaWorld, but fails to achieve task success (shown in Figure 3 ) nor make progress on the other environments. LEMUR avoids both failure modes by maintaining objective-specific reward models and adapting the trade-off online. On MO-MetaWorld, Figure 3 reports the task success rate: LEMUR reaches closest to the Oracle, while PbMORL plateaus.

Multi-Objective Metrics.

Environment Hypervolume ( ↑ \uparrow ) Sparsity ( ↓ \downarrow )

LEMUR PbMORL LEMUR PbMORL

MO-LunarLander 1.10 × 𝟏𝟎 𝟒 \mathbf{1.10\times 10^{4}} 1.09 × 10 4 1.09\times 10^{4} 134.5 134.5 7.8 \mathbf{7.8}

MO-Hopper 3.67 × 𝟏𝟎 𝟔 \mathbf{3.67\times 10^{6}} 2.24 × 10 6 2.24\times 10^{6} 294.7 \mathbf{294.7} 1731.7 1731.7

MO-HalfCheetah 4.86 × 𝟏𝟎 𝟕 \mathbf{4.86\times 10^{7}} 4.78 × 10 7 4.78\times 10^{7} 294.7 \mathbf{294.7} 2191.0 2191.0

MO-MetaWorld 2.15 × 𝟏𝟎 𝟔 \mathbf{2.15\times 10^{6}} 1.43 × 10 6 1.43\times 10^{6} 436.3 \mathbf{436.3} 5564.7 5564.7

Table 1: Hypervolume and Sparsity for LEMUR and PbMORL. Full results are in Appendix F.1 .

We evaluate LEMUR using standard multi-objective metrics (Hayes et al. 2022a ) . Hypervolume (HV) measures the volume of objective space, rewarding policies that are both high-performing and broadly spread (Teoh et al. 2025 ) ; Sparsity (SPS) measures the average distance between policies along the front, with lower values indicating more uniform coverage (Teoh et al. 2025 ) . As summarised in Table 1 , LEMUR attains the highest or comparable Hypervolume across all four environments while achieving markedly lower sparsity than PbMORL, indicating that it learns policies that is both higher-performing and more uniformly distributed over the trade-off space. MORAL is excluded from these set-based metrics, as its single-objective policy optimization against a scalarized reward recovers only one solution rather than a front. Full results are reported in Appendix F.1 .

Reward Model Alignment.

We additionally evaluate the learned reward models directly against the ground-truth teacher rewards, following established PbRL evaluation practice (Lee et al. 2021a ) . We report Spearman rank correlation, which measures how accurately the learned reward models rank individual states compared to the teachers’ ground-truth reward; the Trajectory Alignment Coefficient (TAC) (Muslimani et al. 2025 ) , which compares rankings over whole trajectories rather than individual transitions. Table 2 shows that LEMUR’s reward models recover their teachers’ preference orderings with consistently strong correlation, and outperform both PbMORL and FPbRL (Appendix F.2 ).

Environment Spearman ( ρ ↑ \rho\uparrow ) TAC ( ↑ \uparrow )

MO-Hopper 0.945 ± 0.002 0.945\pm 0.002 0.856 ± 0.014 0.856\pm 0.014

MO-HalfCheetah 0.710 ± 0.005 0.710\pm 0.005 0.898 ± 0.041 0.898\pm 0.041

MO-MetaWorld 0.520 ± 0.007 0.520\pm 0.007 0.347 ± 0.026 0.347\pm 0.026

Table 2: LEMUR reward model alignment, reporting Spearman rank correlation and Trajectory Alignment Coefficient (TAC). Per-metric comparisons against PbMORL and FPbRL are in Appendix F.2 .

Scaling to More Objectives.

Figure 4: LEMUR scalability to higher-dimensional objective spaces. Episode returns for (a) 3-objective and (b) 4-objective tasks, comparing policies trained with LEMUR (blue) versus ground-truth oracle rewards (orange).

(a) Buffer Relabeling

(b) Noisy Labels

(c) Query Budget Figure 5: Ablation studies on MO-Cheetah evaluating the impact of (a) the shared buffer, vector reward relabeling, and unsupervised pre-training, (b) noisy teacher labels, and (c) varying the total query budget per teacher on agent returns for both objectives; by default LEMUR uses 3900 queries (green). The results are averaged over multiple runs across five seeds.

We next vary the teacher configuration on MO-Cheetah . Figure 4 extends LEMUR to three and four conflicting teachers: the learned-reward policies closely track the ground-truth oracle across all objectives, demonstrating the per-teacher decomposition scales without modification, each additional objective adding one reward model.

5.2 Ablation Studies

To validate the components of LEMUR, we conduct ablation studies on the high-dimensional MO-Cheetah domain; Figure 5 visualizes the learning curves.

Impact of Shared Buffer, Relabeling, and Pre-training. Figure 5 (a) isolates the contributions of the shared replay buffer, vector reward relabeling, and unsupervised pre-training. Disabling the shared buffer causes the most severe degradation. With the shared buffer intact, removing relabeling alone produces a modest but consistent drop relative to full LEMUR, confirming that recomputing rewards under the current models stabilizes learning against reward non-stationarity. Removing pre-training accelerates the earliest phase of training, but converges to lower final returns, indicating that the diverse initial buffer ultimately yields better reward models and policies.

Robustness to Label Noise. Figure 5 (b) corrupts a fraction of teacher labels (flipping preferences with probability up to 15%), following PbRL benchmarking protocol (Lee et al. 2021a ) . LEMUR degrades gracefully: performance is essentially unaffected up to 10% noise, and at 15% the agent still learns effective compromise policies on both objectives, albeit with slower convergence and higher variance, indicating tolerance to levels of annotator error.

Impact of Feedback Budget. Figure 5 (c) varies the total query budget from 260 to 5,200 per teacher. Performance improves with budget, and larger budgets learn faster; notably, even 260 total learns adequate policies on both objectives. This feedback efficiency is particularly important in preference-based RL, where human queries are limited.

Reward Model Ablation. To verify that LEMUR’s gains stem from its weight-conditioned reward model rather than the surrounding pipeline, we re-run LEMUR replacing this component with an ensemble of three unconditioned MLPs (Christiano et al. 2017 ; Lee et al. 2021b ) , holding all else fixed. The weight-conditioned variant converges to 6 , 812 ± 39 6{,}812\pm 39 and 4 , 404 ± 22 4{,}404\pm 22 on the two objectives, against 4 , 556 ± 369 4{,}556\pm 369 and 2 , 902 ± 245 2{,}902\pm 245 for the ensemble. (For full results, refer to Appendix C.7 ).

Additional Experiments. In Appendices C.1 - C.7 , we demonstrate that LEMUR accommodates changing teachers/objectives mid-training, varying levels of conflict between teachers, and also non-stationary preferences without reinitialization. Query ablations reveal that segment length is important to performance, and we verify that LEMUR maintains performance even when teachers are in agreement with overlapping preferences.

6 Related Work

Reward Learning from Preference Feedback.

Designing reward functions is a primary bottleneck in scaling RL, as manual crafting is impractical and can induce unsafe behavior (Amodei et al. 2016 ) ; prior work instead learns rewards from demonstrations (Ng and Russell 2000 ; Abbeel and Ng 2004 ) , language (Lin et al. 2022 ) , or human feedback (Christiano et al. 2017 ) . Preference-based RL (PbRL) learns rewards from pairwise comparisons (Christiano et al. 2017 ; Lee et al. 2021b ) ; popularized as RLHF (Stiennon et al. 2020 ; Ouyang et al. 2022 ) , it typically trains a single reward model, aggregating diverse feedback into one scalar (Ouyang et al. 2022 ) and failing to capture the multi-objective nature of human values (Sorensen et al. 2024 ) . Offline variants train rewards on fixed datasets before policy optimization (Shin et al. 2023 ) , but static models suffer distribution shift as the policy diverges from the offline coverage, causing reward exploitation (Gao et al. 2023 ; Ye et al. 2024 ) . Online, iterative RLHF mitigates this by collecting feedback alongside policy optimization (Dong et al. 2024 ; Gao et al. 2023 ; Ye et al. 2024 ) , which is more critical still in MORL, where the agent must span a space of diverse policies (Hayes et al. 2022a ) ; offline MORL instead presupposes specified rewards or adequate dataset coverage (Yuan et al. 2024 ; Zhu et al. 2023 ) . LEMUR circumvents this by jointly and interactively optimizing both the reward models and the multi-objective policy online.

Multi-Objective Reinforcement Learning (MORL).

MORL learns a set of policies approximating the Pareto frontier (Roijers et al. 2013 ; Hayes et al. 2022b ) , via single-policy scalarization, weight-conditioned, or multi-policy methods; multi-task and Meta-RL are closely related (Chen et al. 2019 ; Abdolmaleki et al. 2020 ; Sener and Koltun 2018 ) . Most of this literature assumes a vector of ground-truth reward functions (Hayes et al. 2022b ) , which is impractical in complex, real-world tasks. Reward-free MORL (Chen et al. 2026 ) relaxes this only partially, using reward-free exploration as an auxiliary objective while still assuming an extrinsically specified ground-truth reward. LEMUR extends the MORL paradigm to the setting where the objectives are never observed and must be inferred directly from preferences.

Learning & Alignment with Diverse Objectives.

Standard RLHF fails to capture the pluralistic nature of human values (Sorensen et al. 2024 ) , and existing remedies rely on manual aggregation functions (Rodriguez-Soto et al. 2023 ) , expensive consensus datasets (Tessler et al. 2024 ) , static offline learning (Bakker et al. 2022 ) , or model heterogeneous feedback as hidden context without optimizing the trade-off between preferences (Siththaranjan et al. 2024 ) . Our closest baselines learn rewards for multiple objectives but inherit strong coherence assumptions: MORAL (Peschl et al. 2022 ) requires expert demonstrations and freezes AIRL-learned rewards (Fu et al. 2018 ) , PbMORL (Mu et al. 2025 ) assumes a single teacher, and FPbRL (Siddique et al. 2023 ) fixes a welfare scalarization a priori. LEMUR instead learns objective-specific reward models from separate feedback streams and jointly optimizes a multi-objective policy online via vector reward relabeling , without offline pre-training or expert demonstrations.

7 Conclusion & Limitations

We propose LEMUR, a framework for Multi-Objective RL in domains where reward functions are unknown and must be inferred from the conflicting preferences of multiple teachers. LEMUR jointly learns the objectives and the policies that balance them, without expert demonstrations, pre-defined rewards, or a priori aggregation rules that existing methods require. Across multi-objective RL control and robotic manipulation benchmark environments, LEMUR outperforms aggregation baselines and recent preference-based multi-objective methods, and remains robust to label noise, reduced feedback budgets, and scaling to additional objectives. Several directions for future work are as follows. Our evaluation uses scripted teachers, standard practice in PbRL for controlled and reproducible comparison (Lee et al. 2021a ) ; our noise ablations suggest the framework tolerates the label error real annotators exhibit, and a human study is the natural next validation. We adopt linear scalarization, and extending to non-linear scalarization would allow the framework to learn policies in non-convex regions of the front (Roijers et al. 2013 ; Hayes et al. 2022b ) . Finally, active querying (Akrour et al. 2012 ) offers a route to further reducing the number of queries, which is a promising path for deploying preference-based MORL in real-time.