Build: use host key checking

[agjohnson]

In some recent changes we disabled host key checking on SSH Git operations:

• Build: Skip host key verification when syncing versions #12724
• Build: add GIT_SSH_COMMAND env var VCS environment variables #12656
• Build: set GIT_SSH_COMMAND to avoid host key checking #12649

The reason was that the build process was prompting on missing host keys (StrictHostKeyChecking=ask, the default). I don't know why the prompt was happening generally though, the host keys for known hosts should be available on build instances.

We currently have the option set to StrictHostKeyChecking=no, which doesn't prompt but also doesn't use existing host keys for verification.

The option to use to avoid prompting but to still use existing known hosts is StrictHostKeyChecking=accept-new:

OpenSSH ssh_config man page

The Ubuntu 24.04 man pages don't hint at this option being available, it should be in the OpenSSH package installed. This option might not yet be usable in 24.04.

[stsewd]

I don't know why the prompt was happening generally though, the host keys for known hosts should be available on build instances.

Those keys are in the host, not the docker container.

[agjohnson]

Yeah, but I thought we were configuring the container SSH from the host. I'm not finding this if we were though.

But yeah, then the container needs a copy of known_hosts from the host in order to provide host key checking on GitHub/GitLab/Bitbucket via accept-new. This is exactly what CircleCI does, though through commands instead of a managed known_hosts.

From a build job checkout:

Creating .ssh directory
Adding the following entries to known_hosts:
github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
bitbucket.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE=
bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO
bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M=
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=

[humitos]

Yeah, we just need to mount the /home/docs/.ssh/known_hosts file from the host inside the container and use StrictHostKeyChecking=accept-new 👍🏼

[stsewd]

We shouldn't mount it, only copy the files. We don't want users changing that file for everyone.

[humitos]

It can be mounted as readonly, but yeah. It's the same

[stsewd]

We still want users to be able to update that file inside the container. We have copy the file. Maybe mount the file as read only on /tmp and then copy it to the expected location. But we should be able to just copy the file from the host to the container, no need for mount points.

[agjohnson]

Also, just to clarify, we do need to verify the option is available in Ubuntu 24.x. It was not clear to me in research if it was. It is possible it's available but not documented.

[humitos]

yeah, the feature is available in Ubuntu 24 and also in 22 https://manpages.ubuntu.com/manpages/jammy/en/man5/ssh_config.5.html -- so we are fine here 👍🏼

[humitos]

I took a quick look at this and it seems there are two alternatives:

1. use Docker put_archive API which requires us to create a TarFile first
2. dump the file into a shell command using exec_create as we do for the other regular commands