Add a k8up_last_schedule_job_succeded metric

[danielpodwysocki]

Summary

As a platform engineer
I want to be able to alert only when the last job didn't succeed
So that I reduce alert fatigue in my team

Context

What I'm after is essentially alerting only when the last job for a schedule did not pass.

I'd also like to have the alert per-schedule, so that I can amend things such as what is its name, namespace, etc.

I was thinking of a k8up_schedule_last_job_succeded gauge, with a value of 1 for when we are ok and 0 when it failed.

Out of Scope

No response

Further links

No response

Acceptance Criteria

• A metric exists with enough labels to allows a user to alert on:
  • last job failure - meaning if a backup succeded an hour ago and failed 23h ago, I get no alert
  • specific namespace
  • specific schedule name

Implementation Ideas

It'd be another metric - let me know if that sounds good and fits the project well and I'd also happy to get it contributed.

[danielpodwysocki]

Thinking about it, what I suggested would be for a narrow use case and hide flakiness that other users might want to detect.

Perhaps a k8up_schedule_failed_jobs_gauge , counting how many currently failed jobs belong to a schedule would be better?

It could then easily be translated into the same requirements I was after, while also covering other people's potentially different use cases - I could use a time range to query what I need, while others could alert per-schedule/namespace/whatever they need.

Does that make sense? I think that'd get the most granularity people might need.

[danielpodwysocki]

Actually, there seems to be a very clean way to solve this in kube-state-metrics that requires no work/code: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/extend/customresourcestate-metrics.md

Cool, TIL :D

If that's the case, not sure if you still want something similar in the metrics of the project itself? Or should I just close the issue?