[hma] Split /status into /livez and /readyz#1974
Open
aokolish wants to merge 3 commits intofacebook:mainfrom
Open
[hma] Split /status into /livez and /readyz#1974aokolish wants to merge 3 commits intofacebook:mainfrom
aokolish wants to merge 3 commits intofacebook:mainfrom
Conversation
Kubernetes HTTP probes only inspect the response status code, not the body. The combined /status endpoint returned 503 for both INDEX-NOT-LOADED (cold start) and INDEX-STALE, so pointing both livenessProbe and readinessProbe at it was effectively the same probe - and the cold-start 503 caused CrashLoopBackOff before the index could finish loading. Add /livez (always 200 if Flask can serve) and /readyz (mirrors the existing /status checks) as separate endpoints. /status is left untouched for backwards compatibility and marked deprecated in its OpenAPI description. Closes facebook#1905.
github-actions
Bot
added
the
hma
Drop k8s-specific framing in the OpenAPI descriptions and test comment. The endpoints work the same way under any health-check client that acts on HTTP status codes (k8s, ALB/NLB, GCP LB, Consul, Envoy, etc.); k8s is just one prominent example. Status-code-only is the lowest common denominator the split is designed for, but the body strings remain useful for body-aware clients (HAProxy http-check expect, Docker HEALTHCHECK shell scripts, Azure Application Gateway match conditions).
Dcallies
approved these changes
May 4, 2026
Contributor
Dcallies
left a comment
Dcallies
left a comment
There was a problem hiding this comment.
Since the ready endpoint is the same as status, let's keep using that instead of creating a new endpoint, then you don't need to go through the steps of deprecated it. Other comments inline.
| return "I-AM-ALIVE", 200 | ||
|
|
||
| @app.get( | ||
| "/livez", |
Contributor
There was a problem hiding this comment.
Can we nest this under status?
/status/live
| return "OK", 200 | ||
|
|
||
| @app.get( | ||
| "/readyz", |
Contributor
There was a problem hiding this comment.
Can we nest this under status?
/status/ready
Comment on lines
+295
to
+300
| "Liveness check. Returns 200 if the process can serve HTTP. Does" | ||
| " not check index state - this is intentional, so that" | ||
| " health-check clients acting on status codes alone do not" | ||
| " restart the instance during cold start before the index has" | ||
| " loaded. Failing this should imply the process is wedged and" | ||
| " needs to be restarted." |
Contributor
There was a problem hiding this comment.
Since this appears to be motivated by kubernetes needs, I would have expected to see that mentioned here.
| }, | ||
| summary="Health check", | ||
| description="Liveness/readiness check", | ||
| summary="Health check (deprecated)", |
Contributor
There was a problem hiding this comment.
Suggested change
| summary="Health check (deprecated)", | |
| summary="Health check", |
| " the database)." | ||
| ), | ||
| ) | ||
| def readyz(): |
Contributor
There was a problem hiding this comment.
blocking: Wait, this seems to be the same implementation as the original status. Why don't we leave the old one so you don't need to deprecate it?
| " clients that act on HTTP status codes alone (most load balancers" | ||
| " and orchestrators) cannot distinguish the failure modes this" | ||
| " endpoint reports, so using it as a liveness check causes restart" | ||
| " loops during cold start while the index is still loading." |
Contributor
There was a problem hiding this comment.
Suggested change
| " loops during cold start while the index is still loading." | |
| "Returns 200 if the server is healthy and should serve traffic. If you need a separate check that the server is alive, check for a non-empty return from this endpoint or use a 200 return from /status/live" |
Per Dcallies' review on facebook#1974: - Drop /readyz: it duplicates /status. Keep /status as the readiness check (un-deprecated) with the reviewer's suggested description that points body-aware clients at /status and status-code-only liveness clients at /status/live. - Rename /livez to /status/live to nest under /status. - Mention Kubernetes (and other orchestrators / load balancers that act on status codes alone) in the /status/live description, since that's the motivating case. - Return "I-AM-ALIVE" from /status/live so its body matches /status, letting body-aware clients use either endpoint identically.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the combined
/statushealth check into separate/livez(liveness) and/readyz(readiness) endpoints. Closes #1905.Why the previous fix didn't work
#1912 disambiguated the cold-start case (
INDEX-NOT-LOADED) from staleness (INDEX-STALE) in the response body, but most health-check clients only inspect the response status code. From their perspective both still return503, so pointing both a liveness and a readiness check at/statusis effectively the same check, and during index reloading those503responses can cause a restart loop before the index can finish loading.What this PR does
/livez— minimal liveness. Failing this should mean the instance needs to be restarted./readyz— readiness. Mirrors the existing/statuschecks/status— left untouched for backwards compatibility.Migration
Example (Kubernetes):
Test plan