tree afa8607137ccb258e9868743f8442a9825cd246a
parent 4f65e2f483b6f764c15094d14dd53dda048a4048
author Christian Brauner <christian.brauner@ubuntu.com> 1589979467 +0200
committer Christian Brauner <christian.brauner@ubuntu.com> 1589986236 +0200
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQRAhzRXHqcMeLMyaSiRxhvAZXjcogUCXsVDvAAKCRCRxhvAZXjc
 orbfAPwKlZC6+G0A96e5jR4VUrvfskp4v+OJjyctVnH1AwaUsQD/RSHBiTrMBNHu
 n9LtuOrbXk1Woso4vmqtRgcHv7ocjgA=
 =gw3S
 -----END PGP SIGNATURE-----

ipv6/route: inherit max_sizes from init netns

During NorthSec (cf. [1]) a very large number of unprivileged
containers and nested containers are run during the competition to
provide a safe environment for the various teams during the event. Every
year a range of feature requests or bug reports come out of this and
this year's no different.
One of the containers was running a simple VPN server. There were about
1.5k users connected to this VPN over ipv6 and the container was setup
with about 100 custom routing tables when it hit the max_sizes routing
limit. After this no new connections could be established anymore,
pinging didn't work anymore; you get the idea.

The kernel helpfully logs a message pointing out that bumping max_sizes
should be considered to fix that problem. Here's where it gets tricky.
Routing tables are namespaced and as such max_sizes along with the other
sysctls is a per-netns limit, not a global limit. Each network namespace
by default gets a limit of 4096 routes. Network namespaces owned by the
initial user namespace can change this limit afterwards but network
namespaces owned by non-initial user namespaces can't.

The workload outlined is not an uncommon one. Running VPNs in
unprivileged containers for a large number of users is something people
do and in general large routing tables in unprivileged containers are
pretty common.

There are multiple ways to fix or at least alleviate the problem:
1. Fully namespace at least max_sizes. Right now the max_sizes sysctl
   doesn't show up in network namespaces owned by non-initial user
   namespaces and hence, can't be written to by user namespace root. We
   could simply expose it to non-initial network namespaces owned by
   non-initial user namespaces.
   This solution is the cleanest one conceptually: each container
   already has a separate routing table regardless if privileged or
   unprivileged so allow them to determine the size of it too. The issue
   I see with this solution is that it would potentially make it easier
   to dos the system or even exceed kernel memory. If I'm not mistaken,
   a container could bump max_sizes to INT_MAX(2147483647) and keep
   allocating routes.
   An argument against this worry is that this is technically already
   possible by simply creating a huge number of network namespaces, each
   with a 4096 route limit and allocating 4096 routes in each of them.
   An unprivileged user could for example create at least
   /proc/sys/user/max_net_namespaces network namespaces. The limit on
   the number of network namespaces on my host is e.g. 62690 which means
   256778240 routes. So by exposing max_sizes we don't really add a new
   attack surface.
2. Namespace max_sizes but limiting it to init_net.sysctl.max_sizes. This
   would mean that a container would start with the 4096 limit but if
   the initial network namespace bumps max_sizes later the container
   could bump it too. This has two problems, first the container doesn't
   know what the host bumped max_sizes too and would need to guess by
   trying to write a larger value. Second, this would break network
   namespaces owned by the initial user namespace as they would suddenly
   be limited by the initial network namespace.
3. Inherit the limit from the initial network namespace at
   container/network namespace creation time. This would mean we don't
   fully namespace max_sizes but allow the host to set a limit that it
   is fine with each container to inherit.
   That sounds acceptable but will mean that a nested container can e.g.
   get a larger max_sizes value than the container it was created in.
4. Inherit the limit from the network namespace the current network
   namespace. This to means we don't fully namespace max_sizes but allow
   the current network namespace to choose a limit it is comfortable
   with inheriting.

[1]: https://nsec.io/
Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>