Reuse staging endpoints

[smashwilson]

As you observed this morning, there's definitely a bunch of redundancy between this and the changes I've been planning to do to support pull request builds. For both use cases, we want to host new metadata envelopes "in front of" a production content service. From this repo I'm pretty certain you'll be able to directly use whatever "staging" endpoints are included as part of that effort - the content from the PR build will be sent from a preparer running on a Strider PR build while the content from the web client will be sent directly from the web client. We'll just have to be careful that our URL prefixes don't collide somehow.

I wrote up the changes that I'd had in mind today:

• Proxy content lookup failures to another content service content-service#68
• Staging mode: Retry failed content IDs without their first path segment content-service#69
• Staging mode: Remove the first path segment from the presented path presenter#97

To compare our approaches:

• I like using URL segments to identify the "environment" rather than hacky regexp matching. (I'm increasingly thinking that we should explicitly state the content IDs are URLs and do all the normalization, etc that's implied by that, but that's a bigger discussion I think.)
• The presenter-manager is something I explicitly didn't want to do. Launching a separate presenter container for every "build" will quickly overwhelm us once we start doing builds for every commit pushed to a PR across 20-ish content repositories. I'm trying to avoid needing to reap old builds away or put time limits on them. Also, without a wildcard cert, it'll be much harder to use HTTPS for them.
• I'm not certain whether we want to have separate proxy services or not. I'm currently thinking that I won't be able to do the presented path manipulation that I want to do to by just putting a proxy in front of the presenter - when I see a path of /build-whatever/foo/bar/baz/, I want to (a) use /foo/bar/baz/ to find the template and the content ID prefix, but (b) use /build-whatever/foo/bar/baz/ to construct the content ID of https://github.com/someorg/somerepo/v9000/build-whatever/foo/bar/baz/ that's used to actually find the envelope; otherwise we lose information between the presenter-proxy and the content-service-proxy. (See the point above about not running separate services for every build.) There might be a way to make this work, though?
• You've tempted me to bring up Store metadata envelopes in MongoDB content-service#24 again. So much simpler than the Cloud Files wrangling...

[smashwilson]

otherwise we lose information between the presenter-proxy and the content-service-proxy

Hmm, we could do this with a custom HTTP header. The presenter-proxy could add an X-ContentID-Middle header of some kind that the presenter uses to assemble the content ID:

<content ID prefix> + <header value> + <presented path suffix>

[ktbartholomew]

My only qualm about trying to piggyback on the PR builder is that one of the goals of this tool was to allow simple editing/previewing completely outside of version control. If the content service middleman only looks up PRs, I won't be able to reuse it for this. What could work really well though, is if the content service middleman reads a generic revisionID from a URL segment, then feeds that to a pluggable middleware that does whatever it needs to in order to show the revised content. The "staging" instance would inject a middleware that translates revisionID into a PR commit, and the web client instance would inject a middleware that translates it into a local set of changes made via the web UI.

The presenter-manager is something I explicitly didn't want to do.

I wasn't really happy with this solution either, but I was trying to avoid changing any core deconst components to get this prototype working. If we could have a presenter dynamically create its content service URL on each request, that entire hack would be unnecessary, as the presenter would adjust its content service URLs on the fly.

The presenter-proxy could add an X-ContentID-Middle header of some kind that the presenter uses to assemble the content ID

This sounds like it would solve the problem nicely, and give the presenter the information it needs to dynamically build its content service URLs. I think X-RevisionID as used above may read a little nicer though. 😄

[ktbartholomew]

Ugh. I'm stupid. I'm realizing now that your content service issues have nothing to do with how revision-ified content gets to the content service, meaning I could just send users' changes to the staging URL. Very easy.

Namespacing the revisionID segment could be an easy way to ensure nothing collides. The PR build could generate a revisionID like github-123abcdef and the web client would generate it as webclient-123abcdef.

[smashwilson]

If we could have a presenter dynamically create its content service URL on each request, that entire hack would be unnecessary, as the presenter would adjust its content service URLs on the fly.

👍

I think X-RevisionID as used above may read a little nicer though. 😄

Hah, names far from final. "X-RevisionID" isn't bad, I like it.

I'm realizing now that your content service issues have nothing to do with how revision-ified content gets to the content service, meaning I could just send users' changes to the staging URL. Very easy.

Well, I hadn't actually, you know, written them down until earlier today, so don't be too hard on yourself 😉

Namespacing the revisionID segment could be an easy way to ensure nothing collides.

Totally. I'm leaning toward "build-" instead of "github-" because they don't have to come from "pull requests on GitHub" per se.