Description
Add support for Kueue (basically revives: #822) as a scheduler plugin.
Detailed Proposal
The implementation itself should be pretty straight forward. So will discuss where the plugin should live.
We are moving away from registering plugins via entrypoints to using namespace packages (PEP-420). Essentially we'd create {git_root}/torchx_plugins/schedulers/kueue.py and register the KueueScheduler plugin as:
# in kueue.py
class KueueScheduler(Scheduler...):
...
@register_scheduler(name="kueue")
def kueue_scheduler() -> Scheduler:
return KueueScheduler(...)
See torchx.plugins documentation for details.
Schedulers registered this way are automatically discovered at runtime as long as the wheel that provides torchx_plugins/schedulers/kueue.py is installed in the current python environment.
We need to create a top-level torchx_plugins/* dir and move all the builtin plugins that are currently in torchx/schedulers/*_scheduler.py to torchx_plugins/schedulers/builtins/*_schedulers.py and add torchx_plugins/* to the torchx wheel.
Description
Add support for Kueue (basically revives: #822) as a scheduler plugin.
Detailed Proposal
The implementation itself should be pretty straight forward. So will discuss where the plugin should live.
We are moving away from registering plugins via entrypoints to using namespace packages (PEP-420). Essentially we'd create
{git_root}/torchx_plugins/schedulers/kueue.pyand register theKueueSchedulerplugin as:See
torchx.pluginsdocumentation for details.Schedulers registered this way are automatically discovered at runtime as long as the wheel that provides
torchx_plugins/schedulers/kueue.pyis installed in the current python environment.We need to create a top-level
torchx_plugins/*dir and move all the builtin plugins that are currently intorchx/schedulers/*_scheduler.pytotorchx_plugins/schedulers/builtins/*_schedulers.pyand addtorchx_plugins/*to the torchx wheel.