CUS-349: CI: enable remote execution in jobs that can use it

[jayconrod]

Added platform and C++ toolchain configurations for platforms we use
that have configured remote execution pools. This is mostly copied
from github.com/EngFlow/examples.

In .bazelrc, added platform configurations like remote_linux_x64.
One of these needs to be specified when using remote execution.
Also, rename the opal configuration to engflow and split out engflow_bes.
We'll only support one cluster for this repo.

In workflows, added --config=<remote_platform> and replaced the
CLUSTER_CONFIG environment variable with either engflow or engflow_bes
as appropriate. The jobs driven with 'bazel run' cannot use remote
execution because that always runs on the host, so they're engflow_bes.

[jayconrod]

Test runs:

• main: https://github.com/EngFlow/auth/actions/runs/10638044492
• release: https://github.com/EngFlow/auth/actions/runs/10638055339 (expected failure, can't be run on dev branch. I'll verify it works in a follow-up.)

[ola-rozenfeld]

I don't actually understand the bazel changes.

[ola-rozenfeld]

Note: glass will be faster, because the CI runner is on glass, so it will be faster if the RE action is on the same cluster. Fine to switch in a followup (or create separate configs for glass and opal so that the switch for both CI Runners and RE can be done more easily).

[jayconrod]

Can't use Glass yet, mainly because it's not authorized for open source, though I think we can change that. It also doesn't have a Mac RE pool, though we don't strictly need that.

[ola-rozenfeld]

• release: https://github.com/EngFlow/auth/actions/runs/10638055339 (expected failure, can't be run on dev branch. I'll verify it works in a follow-up.)

About that: I was also annoyed that I had to find and comment out the code that does this check when I migrated the job to CI Runners (and it was not trivial). I think this should be easier (for example, an input parameter that needs to be manually checked in order to enable releasing from a dev branch). Everything should be possible from a dev branch, imo, otherwise debugging it is a pain; maybe the outputs of a release from a dev branch should be easily distinguishable from those of master, so that they couldn't be used?

[ola-rozenfeld]

Oh, just saw #46, you're taking care of that -- great!

[minor-fixes]

How is this timeout sized, now that it's extended to 30m?

[jayconrod]

10m wasn't fast enough to reliably complete a clean build on Opal. It's usually enough, but not always.

I hit a problem that was kind of a worst case: completely cold cache on a pool that had queueing but hadn't scaled up yet. The GoStdLib actions are particularly slow with a cold cache because they're not optimized for RE, and there are a lot of them because we're cross-compiling to every platform on every build.

[jayconrod]

I don't actually understand the bazel changes.

Which changes do you mean? Is there something I can explain?

I assume you mean the stuff in //platform. We need a platform and a C++ toolchain for each pool we want to use. The C++ toolchain is 99% of this diff. I copied it from the example repo, which copies whatever Bazel generates when run locally inside the worker's container.

[ola-rozenfeld]

I don't actually understand the bazel changes.

Which changes do you mean? Is there something I can explain?

I assume you mean the stuff in //platform. We need a platform and a C++ toolchain for each pool we want to use. The C++ toolchain is 99% of this diff. I copied it from the example repo, which copies whatever Bazel generates when run locally inside the worker's container.

Thank you, that helps! And all our customers do it like this, too, right? I'm wondering if there is any way it can be simplified for them -- something where they need to configure the minimal things for everything to work with their remote cluster, at least for the simpler use-cases...

[jayconrod]

There are a couple alternatives, but a user who needs a remote C++ toolchain needs something equivalent of this. Most of the rules depend on C++ in some way.

If the C++ toolchain is part of the remote container image, you need an explicit configuration like this. There's no way to generate that as part of the build because it's needed before the analysis phase. But I think it would be possible to write a tool that executes a remote action then generates the configuration for you.

The main alternative is a hermetic toolchain, where all files are provided as inputs to actions that use the toolchain, and the toolchain configuration is included in the package. That has some overhead, at least in constructing the input tree, but I haven't measured how big the difference is. It's certainly easier to use.