[Bug]: Building wheels failing on self hosted actions, working on github's runners

[damccorm]

What happened?

Starting Dec 10, building Python wheels started failing with a bunch of seg faults. It seems likely due to something with the underlying hardware or image. I tried updating to using GitHub hosted runners and it seems like this works. We can do this as a workaround, but we should understand the problem and switch back to self-hosted to avoid being blocked on GitHub quota.

Example failure - https://github.com/apache/beam/actions/runs/12625457564

This also impacts some other workflows which I will switch over to GitHub hosted, but we should similarly switch back

Workflows where we've seen this issue along with the PR used to temporarily mitigate:

• .github/workflows/build_wheels.yml - Build wheels on gha runners instead of local runners #33505
• .github/workflows/republish_released_docker_containers.yml - Use self-hosted runners for republish workflow #33507
• .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml - Temporarily use self hosted runners for snapshots #33563
• .github/workflows/beam_PostCommit_Python_Arm.yml - Update beam_PostCommit_Python_Arm.yml #33564

We should figure out what is causing the problem and then revert all these PRs

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[damccorm]

@claudevdm did some good investigation here - Claude would you mind adding in any investigation you've done/things you've tried

[claudevdm]

Sure, here is what I found

• The workflow started failing consistently on Dec 10
• The only difference I could find since they started failing is a runner/host Kernel Version: 6.1.100+ to Kernel Version: 6.1.112+ change
• The kernel date change lines up with a release on Dec 10
• The arm workflows have to do some cross-compilation thing using qemu
• cibuildwheels pulls quay.io/pypa/manylinux2014_aarch64 which uses gcc 10 to compile the cyton code
• It seems there are some incompatible changes with the new kernel release and cross compiling with gcc 10

[damccorm]

FYI @Amar3tto @mrshakirov @akashorabek - this should fix a few flaky workloads, but it is just a patch. It would be great to dig in on this one and try to figure out what in our self-hosted runner infrastructure is causing the flakes so that we can move these workloads back to our self-hosted runners.