Podman rootless container in bridge network with port-forwards not working

[doppelrittberger]

Issue Description

I try to create a bridged network with
podman network create heureka
Then I try to create a container using this network and defining port mappings:
podman run --network heureka --network-alias heureka -p 10001:10001 <some-image>
I configured the containers.conf to use slirp4netns as netns and netavark as network backend.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman network create heureka
2. podman run --network heureka --network-alias heureka -p 10001:10001
3. -> Fails

Describe the results you received

I receive an error message and the container does not start:

WARN[0004] failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: read-only file system 
Error: netavark: Sysctl error: IO Error: Read-only file system (os error 30)

Describe the results you expected

I expect the container to start

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 98.71
    systemPercent: 0.45
    userPercent: 0.83
  cpus: 48
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "38"
  eventLogger: file
  freeLocks: 2048
  hostname: no-priv
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    - container_id: 65535
      host_id: 65536
      size: 100001
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    - container_id: 65535
      host_id: 65536
      size: 100001
  kernel: 4.18.0-477.13.1.el8_8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 84203294720
  memTotal: 269347774464
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.7-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.7
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230823.ga7e4bfb-1.fc38.x86_64
    version: |
      pasta 0^20230823.ga7e4bfb-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 1212h 30m 27.00s (Approximately 50.50 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 1530142773248
  graphRootUsed: 125049090048
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1691705273
  BuiltTime: Thu Aug 10 22:07:53 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

containers.conf:

[containers]
volumes=["/proc:/proc","/sys:/sys"]
default_sysctls=[]
netns="slirp4netns"
[network]
network_backend="netavark"
network_cmd_options="enable_ipv6=false"

Additional information

/proc and /sys are read-only. I don´t know if I can mount them from kubernetes host into podman pod and if this would help. Rootful podman is unfortunately no option

[Luap99]

We need to configure some sysctl's to make routing work so this is expected to fail if you have a read only /proc.
I am not sure if it makes to make the error non fatal as the general network setup likely still works correctly but it could result in incorrect routing so that would things harder to discover it things fail later on but I never tried to see what happens if we do not set any sysctl's.

[doppelrittberger]

Ok. Is there a way to mak proc writable in rootless environement?

[Luap99]

Your outer container needs to mount /proc rw, that usually means running it "privileged". No idea how this would look when run via kubernetes.

[rhatdan]

--security-opt unmask=/proc/*

Kubernetes has procMount=Unmasked

[Luap99]

I created containers/netavark#825 to see if we should ignore read only sysctl issues.

Otherwise I don't see anything that can be done here, in general I find most of the nested container setups somewhat questionable. The more you are nesting network namespaces with this network configuration the lower the throughput will be.