Unable to read or use the SSH key inside the custom pipe

Summary

Unable to read or use the SSH key inside the custom pipe (Child container). If a user is using any custom pipe and the pipe is trying to read the private SSH key mounted from the build container, they would get a permission denied error.

Environment

To replicate this issue, create a child container in a container with the usernamespace enabled and access the mounted volume.

Diagnosis

• The child container (pipe) is unable to access the parent container (build container) data even though the volume is mounted

  1 2 --volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \ --volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro

• The issue is reproducible internally. When examining the permissions, we can see the permissions to read/write/execute are set to nobody causing a permission denied error.

  1 2 3 4 5 6 7 8 9 10 Listing SSH Keys /opt/atlassian/pipelines/agent/ssh total 4 drwxr-xr-x 3 nobody nogroup 140 Jul 21 10:39 . drwxr-xr-x 4 root root 4096 Jul 21 10:40 .. drwxr-xr-x 2 nobody nogroup 100 Jul 21 10:39 ..2022_07_21_10_39_44.810911684 lrwxrwxrwx 1 nobody nogroup 31 Jul 21 10:39 ..data -> ..2022_07_21_10_39_44.810911684 lrwxrwxrwx 1 nobody nogroup 13 Jul 21 10:39 id_rsa -> ..data/id_rsa lrwxrwxrwx 1 nobody nogroup 17 Jul 21 10:39 id_rsa_tmp -> ..data/id_rsa_tmp lrwxrwxrwx 1 nobody nogroup 18 Jul 21 10:39 known_hosts -> ..data/known_hosts

Cause

For each container that is spun up, the Docker user namespace feature will assign a specific UID/GID to it. With that UID/GID in place, Linux systems limit the file access permissions to the specific UID/GID. By default, your main build container will be root. Therefore, the main container can access the other containers that you create within it.

The user namespace feature will assign the child containers with a large UID/GID that it will recognize the container by. However, the child container won’t know that the ID it’s being recognized by. File/Directory owned by root on the parent container will not be inherited by the child container. This is why we’re seeing the ownership as “Nobody”.

According to the Docker documentation on userns-remap:

The best way to prevent privilege-escalation attacks from within a container is to configure your container’s applications to run as unprivileged users. For containers whose processes must run as the root user within the container, you can re-map this user to a less-privileged user on the Docker host. The mapped user is assigned a range of UIDs which function within the namespace as normal UIDs from 0 to 65536, but have no privileges on the host machine itself.

Basically, when we run a container, by default it will run as root. The root user's UID inside the container matches the UID on our host and can lead to container escape vulnerabilities.

The change has been introduced to improve the security in pipelines as discussed in the security restrictions. 

Solution

To resolve this issue, the below steps need to be executed within a custom pipe script. This would create a directory with .ssh and copy the contents of SSH from the parent container (build container) to the child container (pipe). 

Modify the permission of the .ssh folder. It removes read/write/execute permissions from groups and others but preserves whatever permissions the owner had and it applies to the entire contents of the .ssh directory.