Regression running inner container

[duck-rh]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

We run podman inside podman with inner systemd support. There's a few configuration tweaks but it used to work up to podman 3.4.7 and stopped working with 4.1.x (for running the inner container, see environment info below for the outer one).

Steps to reproduce the issue:

1. as non-root: podman rm outer; podman run --name=outer --cap-add=sys_admin,net_admin,net_raw --sysctl=net.ipv4.ip_forward=1 --sysctl=net.ipv6.conf.all.forwarding=1 --device=/dev/fuse --security-opt=label=disable --security-opt=seccomp=unconfined --uts=private --systemd=always quay.io/podman/stable sleep 99999999

2. in another non-root terminal:

• podman exec -ti outer /bin/bash
• echo "root:100000:1000000000" >>/etc/subuid
• echo "root:100000:1000000000" >>/etc/subgid
• podman network create --subnet 10.89.0.0/16 --ipv6 --subnet fdd1:0:0:0::/64 testnetwork
• sed -i '/net.ipv4.ping_group_range=/ d' /usr/share/containers/containers.conf
• podman run --rm --name=inner -it --device=/dev/fuse --systemd=always --env=container=podman --uts=private --network=testnetwork --hostname=ansible-test-outside.example.com --dns-search=example.com --log-level=debug quay.io/pilou/debian-systemd:bullseye /sbin/init

Describe the results you received:

Outer and inner container start and systemd works inside the inner one.
Here is the log of a working step 2 with podman 3.4.7:
20220706_podman_inner_systemd_ok.log

Describe the results you expected:

The inner container fails to start with: netavark (exit code 1): io error: read-only file system (os error 30)
Here is the log of a failing step 2 with podman 4.1.1:
20220706_podman_inner_systemd_ko.log

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Wed Jun 15 14:31:58 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 73.29
    systemPercent: 6.03
    userPercent: 20.68
  cpus: 4
  distribution:
    distribution: fedora
    variant: container
    version: "36"
  eventLogger: file
  hostname: 380763b19616
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.0-3-amd64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 608759808
  memTotal: 33441599488
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.5-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 394h 6m 58.39s (Approximately 16.42 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.9-1.fc36.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.9
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 224136298496
  graphRootUsed: 84338401280
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1655303518
  BuiltTime: Wed Jun 15 14:31:58 2022
  GitCommit: ""
  GoVersion: go1.18.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.1.1-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes and Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

The environment is either F35 (podman 3.4.7) or Debian unstable (podman 3.4.7 or 4.1.1) with the same results.

[Luap99]

Is /proc/sys mounted readonly?

In the container could you install cni and set the network backend to cni to test if this works (CNI was always used before 4.0, now we use netavark by default)

[duck-rh]

Quack @Luap99 ,

Indeed /sys is ro, no idea why. IIUC --systemd=always takes care of that right?

Switching back to CNI worked; at least I have a workaround now, thanks a lot.

\_o<

[Luap99]

I move this to netavark then, looks like you already set the correct sysctl value that netavark tries to set so netavark should just ignore the error

[duck-rh]

@Luap99 thanks for working on it :-)