return average cpu usage , memory, and io, similarly to nextflow report

[stemangiola]

Prework

• [ x ] Read and agree to the Contributor Code of Conduct and contributing guidelines.
• [ x ] If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• [ x ] New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a GitHub discussion.
• [ x ] Format your code according to the tidyverse style guide.

Proposal

Nextflow can report average cpu usage , memory, and io(https://www.nextflow.io/docs/latest/metrics.html). I am wondering whether crew_controller_slurm could do the same.

[wlandau]

@stemangiola, this seems like a useful idea for crew in general. I am not sure if there is a reliable platform-agnostic way to get these metrics, or how best to store/display them since there can be multiple instances of a given worker and each instance could run more than one task. But I will keep it in mind.

[multimeric]

It's true that many of these platforms (Slurm, AWS Batch etc) can report resource metrics for a given task, which could be added via a new method in the controller (launcher?). But as you say, the persistent worker strategy makes it difficult to map these metrics to single tasks.

[stemangiola]

But if the one-job-one-task strategy is used (e.g. crew_controller_slurm ?), then the memory and CPU profiling would make a lot of sense.

[multimeric]

crew_controller_slurm is still a persistent worker. I think all of crew works with that model. You can set tasks_max=1 to make each worker transient, but:

crew does not excel with perfectly transient workers because it does not micromanage the assignment of tasks to workers, it is recommended to set tasks_max to a value greater than 1.

[wlandau]

crew indeed uses a mostly persistent worker model, although I wrote that comment back when crew was having trouble with backlogged tasks (c.f. wlandau/crew#79 and other issues). Things have gotten much better since then, both in crew and its backend mirai, and it should be able to handle tasks_max = 1.

[stemangiola]

Apologies for my ignorance, but with a persistent worker model, still, is each task limited to one worker? If so, that worker could output the average usage for its tasks.

With the assumption that a queue type (i.e. worker?, e.g. 100Gb memory, 10 cores, 10 hours) is associated with a homogenous set of tasks, this is still incredibly useful as one of the biggest time-consuming and disruptive things in practical pipelining is to profile the resource usage of an arbitrary pipeline to project resource requests in busy computer clusters (e.g. SLURM).

If I want to deploy a pipeline, I would like to get a very good sense of the resources I have to request for a given input size. Without profiling, efficient workflows become impractical.

This is a major feature, in my opinion.