Missing eval function in morld.py

[nkirschi]

An eval function analogous to pcn.py would be handy

[ffelten]

Hi,

These are two very different algorithms, I am not sure what you want exactly in such a function for MORL/D?

[nkirschi]

Afaik the eval functions in this library correspond to evaluating the learnt deterministic policy a = pi(o, w) for a given observation o and conditioning input w. This is something that every MORL algorithm should be able to provide, no?

[ffelten]

Oh I see. Not all algorithms support that (actually, I believe only the algorithms relying on conditioned networks do). See for instance, PQL: https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/pareto_q_learning/pql.py#L282C9-L282C27

Does it make sense to implement it for MORL/D?
The issue is that in MORL/D, policies are not tied to a single weight vector; the weight vectors change during training. Theoretically, MORL/D should also work with conditioned networks but I never implemented it 😅.

With the current version, we could implement something in the vein of an eval based on a weight vector by taking the policy in the Pareto set that has the closest evaluation to the direction given by the weight vector, e.g., argmax_{\pi \in PS} np.dot(\vec{w}, \vec{v}_\pi). Now MORL/D also supports non-linear utilities, so we would need to generalize the scalarization function, but it's quite easy to do. Does that sound like what you want?

Side note: there are a lot of other ways to choose a policy from a PF than scalarizing (and relying on a weight vector), e.g., https://www.sciencedirect.com/science/article/pii/S0360835222000924.

[nkirschi]

Thanks for your insights! It makes sense to offer a version of eval that reflects how MORL/D policy evaluation works, which, admittedly, I'm not knowledgeable about.

[ffelten]

Another note: you actually already have the evaluations stored in the archive for MORL/D.
https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/morld/morld.py#L208-L208

Here is what you'll find in morld.archive:

morl-baselines/morl_baselines/common/pareto.py

Lines 122 to 123 in 8f04aae

	self.individuals: list = []
	self.evaluations: List[np.ndarray] = []

A good example of evaluation is here:

morl-baselines/examples/morld_lunar_lander_restore.py

Lines 60 to 73 in fbd0399

	for i, policy in enumerate(algo.archive.individuals):
	returns = []
	for ep in range(5): # Run 5 episodes per policy
	_, _, _, discounted_reward = policy.wrapped.policy_eval(
	eval_env,
	weights=policy.weights,
	scalarization=algo.scalarization,
	log=False,
	)
	returns.append(discounted_reward)
	mean_return = np.mean(returns, axis=0)
	print(
	f"Policy {i} mean discounted return over 5 episodes: {mean_return}, was {algo.archive.evaluations[i]} in the archive"
	)

So I'm not sure we really "need" an eval, given that it's up to the user to choose which policy they want from the archive. Any thoughts?

[nkirschi]

Can the policies in the archive be indexed by some interpretable quantity? If not, the only straightforward way I see is argmaximising over the immediate reward that the policies achieve for my current observation

[ffelten]

Each policy is associated with an evaluation vector, i.e., morld.archive.individuals[i] == pi_i and morld.archive.evaluations[i] == \vec{v}^{pi_i}.

[nkirschi]

Then I guess it would be nice to have a dict mapping an evaluation vector to the associated policy and have eval (which btw is a confusing name, maybe predict is less ambiguous) index into this dict, so a user could first inspect all evaluation vectors and consequently choose the policy having produced the most suitable vector