peteris.rocks

NFSv4 with only one open port

How to run NFSv4 without rpcbind on Debian and Ubuntu Server

Last updated on

I pay extra attention to what network ports are open on my machines.

My go to command is netstat -putnl (you can remember the arguments as "put new line").

Here's what happens if you want to mount an NFS share on your machine:

$ sudo apt-get install -y nfs-common

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      380/sshd
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init
tcp6       0      0 :::22                   :::*                    LISTEN      380/sshd
tcp6       0      0 :::111                  :::*                    LISTEN      1/init
udp        0      0 0.0.0.0:68              0.0.0.0:*                           348/dhclient
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init
udp6       0      0 :::111                  :::*                                1/init

Look at all those open 111 port numbers! They are used by rpcbind.

It's even worse if you run your own NFS server.

If you use NFS version 4, you don't need rpcbind and its friends and you can disable it.

Here's how.

Debian Buster

I'll use Vagrant with the following Vagrantfile to demonstrate this:

Vagrant.configure("2") do |config|
  config.vm.box = "debian/buster64"

  config.vm.provider "virtualbox" do |vb|
    vb.cpus = 1
    vb.memory = 256
  end

  config.vm.define "nfs-server" do |srv|
    srv.vm.hostname = "nfs-server"
    srv.vm.network :private_network, ip: "192.168.9.3"
  end

  config.vm.define "nfs-client" do |srv|
    srv.vm.hostname = "nfs-client"
    srv.vm.network :private_network, ip: "192.168.9.4"
  end
end

Server

vagrant up nfs-server
vagrant ssh nfs-server

sudo apt-get update
sudo apt-get install -qq vim net-tools

Install NFS server:

sudo apt-get install -y nfs-kernel-server

Create something to share over NFS so we can test it if it works:

sudo mkdir /var/nfs
sudo touch /var/nfs/hello.txt

echo '/var/nfs *(rw,sync,fsid=0,no_root_squash,no_subtree_check)' | sudo tee -a /etc/exports

sudo systemctl restart nfs-server

Let's see what ports we have open...

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      368/sshd
tcp        0      0 0.0.0.0:46043           0.0.0.0:*               LISTEN      2705/rpc.mountd
tcp        0      0 0.0.0.0:40765           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:38409           0.0.0.0:*               LISTEN      2705/rpc.mountd
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init
tcp        0      0 0.0.0.0:40689           0.0.0.0:*               LISTEN      2705/rpc.mountd
tcp6       0      0 :::48245                :::*                    LISTEN      2705/rpc.mountd
tcp6       0      0 :::22                   :::*                    LISTEN      368/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
tcp6       0      0 :::111                  :::*                    LISTEN      1/init
tcp6       0      0 :::34097                :::*                    LISTEN      -
tcp6       0      0 :::39793                :::*                    LISTEN      2705/rpc.mountd
tcp6       0      0 :::35763                :::*                    LISTEN      2705/rpc.mountd
udp        0      0 0.0.0.0:2049            0.0.0.0:*                           -
udp        0      0 0.0.0.0:45345           0.0.0.0:*                           2705/rpc.mountd
udp        0      0 0.0.0.0:42554           0.0.0.0:*                           -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           346/dhclient
udp        0      0 0.0.0.0:59743           0.0.0.0:*                           2705/rpc.mountd
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init
udp        0      0 0.0.0.0:37234           0.0.0.0:*                           2705/rpc.mountd
udp6       0      0 :::43171                :::*                                2705/rpc.mountd
udp6       0      0 :::2049                 :::*                                -
udp6       0      0 :::33029                :::*                                2705/rpc.mountd
udp6       0      0 :::40970                :::*                                -
udp6       0      0 :::42834                :::*                                2705/rpc.mountd
udp6       0      0 :::111                  :::*                                1/init

Insane.

None of that is necessary for NFSv4.

All we need is one open port which is TCP 2049 by default.

Disable all versions that are not v4:

$ sudo vim /etc/default/nfs-kernel-server
...
RPCMOUNTDOPTS="--no-nfs-version 2 --no-nfs-version 3 --nfs-version 4 --no-udp"
RPCNFSDOPTS="--no-nfs-version 2 --no-nfs-version 3 --nfs-version 4 --no-udp"
...

Retart NFS server:

$ sudo systemctl restart nfs-server

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      368/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init
tcp6       0      0 :::22                   :::*                    LISTEN      368/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
tcp6       0      0 :::111                  :::*                    LISTEN      1/init
udp        0      0 0.0.0.0:68              0.0.0.0:*                           346/dhclient
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init
udp6       0      0 :::111                  :::*                                1/init

Much better but not good enough. Those 111 ports opened by rpcbind are not needed either.

Let's disable rpcbind:

sudo systemctl disable --now rpcbind.service rpcbind.socket
sudo systemctl mask rpcbind.service rpcbind.socket

Since rpcbind is needed by nfs-server regardless of which NFS versions are used, it'll be started again unless we mask it. Masking replaces the systemd unit file with a symlink to /dev/null so it cannot be started.

Let's check now:

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      368/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      368/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           346/dhclient

Much, much better. NFSv4 has only one open port which is TCP 2049.

Quick reboot to check that this change will persist:

$ sudo reboot
$ vagrant ssh nfs-server

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      387/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      387/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           359/dhclient

All good.

Client

Let's check that after all this we can still mount NFS shares on other machines.

vagrant up nfs-client
vagrant ssh nfs-client
sudo apt-get update
sudo apt-get install -qq vim net-tools

Install NFS client:

sudo apt-get install -y nfs-common

Mount the share:

sudo mkdir /mnt/nfs
echo '192.168.9.3:/ /mnt/nfs nfs4 intr 0 0' | sudo tee -a /etc/fstab
sudo mount -a
$ ls -l /mnt/nfs
-rw-r--r-- 1 root root 0 Nov 23 00:43 hello.txt

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      368/sshd
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init
tcp6       0      0 :::22                   :::*                    LISTEN      368/sshd
tcp6       0      0 :::111                  :::*                    LISTEN      1/init
udp        0      0 0.0.0.0:68              0.0.0.0:*                           348/dhclient
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init
udp6       0      0 :::111                  :::*                                1/init

We are not running an NFS server on this machine yet port 111 is open.

This time it's enough to just disable rpcbind, no masking is needed here.

sudo systemctl disable --now rpcbind.service rpcbind.socket

Let's check:

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      368/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      368/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           348/dhclient

Wonderful.

Double check if it stays like that after a reboot:

$ sudo reboot
$ vagrant ssh nfs-client

$ ls -l /mnt/nfs
-rw-r--r-- 1 root root 0 Nov 23 00:43 hello.txt

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      382/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      382/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           357/dhclient

All good.

We can clean up now:

vagrant destroy -f

Debian Stretch

I'll be using the same Vagrantfile except for this change:

Vagrant.configure("2") do |config|
  config.vm.box = "debian/stretch64"
  ...

Server

The Debian Buster instructions above work for Debian Stretch as well. But you'll encounter one nasty surprise.

If you reboot your machine, you'll find that it takes unusually long to boot up.

On the screen, you'll see something like this:

[ ] A start job is running for NFS server and services (1min 40s / no limit)

In my case it took about 3 minutes.

systemd-analyze blame shows that it was nfs-server that was so slow.

$ sudo systemd-analyze blame | head
    2min 36.460s nfs-server.service
           591ms keyboard-setup.service
           529ms dev-sda1.device
           223ms proc-fs-nfsd.mount
           215ms run-rpc_pipefs.mount
           196ms networking.service
           151ms systemd-udev-trigger.service
           138ms systemd-journald.service
            76ms dev-mqueue.mount
            75ms dev-hugepages.mount

It turns out it only happens when rpcbind is masked. If you allow rpcbind to be started, nfs-server starts up instantly.

It looks like it's a bug and it was fixed in, I believe, a later version of nfs-common.

The way I understand it is that if rpcbind is not running, the kernel NFS server will wait for it and eventually time out. So it seems you must have rpcbind running when you start the NFS server or it'll take a long time.

Installing Debian Buster's versions of nfs-common was not straightforward because of other dependencies.

I couldn't figure out a nice way of disabling rpcbind except for stopping it after the NFS server is started:

$ time sudo systemctl restart nfs-server
real    3m38.706s

$ sudo systemctl unmask rpcbind.service rpcbind.socket
$ sudo systemctl daemon-reload

$ time sudo systemctl restart nfs-server
real    0m0.077s

$ sudo systemctl stop rpcbind.service rpcbind.socket

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      356/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      356/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           371/dhclient

We can run systemctl stop rpcbind.service rpcbind.socket as an ExecStartPost command in the nfs-server.service systemd unit file:

$ sudo vim /lib/systemd/system/nfs-server.service
...
ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
ExecStartPost=/bin/systemctl stop rpcbind.service rpcbind.socket
ExecStop=/usr/sbin/rpc.nfsd 0
...

Let's test it:

$ sudo systemctl daemon-reload

$ sudo systemctl stop nfs-server
$ sudo systemctl start nfs-server

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      356/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      356/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           371/dhclient

It worked.

You can also check by running sudo systemctl status nfs-server where you should see that ExecStartPost was run and with sudo systemctl status rpcbind you can confirm it's stopped.

It's not a nice solution to edit system files but that'll work for now. If you want a clean solution, upgrade to Debian Buster which does not have this problem.

Client

Client instructions for Debian Buster above work also for Debian Stretch.

The bug affects NFS servers not NFS clients, so everything should work just fine.

Ubuntu Server 18.04

I'll be using the same Vagrantfile but with this change:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"

If you see this error message:

bash: line 4: /sbin/ifdown: No such file or directory

Do this to fix it:

vagrant ssh nfs-server

sudo apt-get update
sudo apt-get install -y ifupdown

vagrant reload nfs-server

Debian Buster instructions above work for Ubuntu Server as well.

Server

Before:

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:44525           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      2052/rpcbind
tcp        0      0 0.0.0.0:46165           0.0.0.0:*               LISTEN      3102/rpc.mountd
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      718/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      937/sshd
tcp        0      0 0.0.0.0:38587           0.0.0.0:*               LISTEN      3102/rpc.mountd
tcp        0      0 0.0.0.0:53087           0.0.0.0:*               LISTEN      3102/rpc.mountd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::41673                :::*                    LISTEN      3102/rpc.mountd
tcp6       0      0 :::111                  :::*                    LISTEN      2052/rpcbind
tcp6       0      0 :::42485                :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      937/sshd
tcp6       0      0 :::56991                :::*                    LISTEN      3102/rpc.mountd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
tcp6       0      0 :::43937                :::*                    LISTEN      3102/rpc.mountd
udp        0      0 0.0.0.0:2049            0.0.0.0:*                           -
udp        0      0 0.0.0.0:45062           0.0.0.0:*                           3102/rpc.mountd
udp        0      0 0.0.0.0:38426           0.0.0.0:*                           -
udp        0      0 127.0.0.53:53           0.0.0.0:*                           718/systemd-resolve
udp        0      0 10.0.2.15:68            0.0.0.0:*                           647/systemd-network
udp        0      0 0.0.0.0:35944           0.0.0.0:*                           3102/rpc.mountd
udp        0      0 0.0.0.0:111             0.0.0.0:*                           2052/rpcbind
udp        0      0 0.0.0.0:55676           0.0.0.0:*                           3102/rpc.mountd
udp        0      0 0.0.0.0:956             0.0.0.0:*                           2052/rpcbind
udp6       0      0 :::47577                :::*                                -
udp6       0      0 :::45820                :::*                                3102/rpc.mountd
udp6       0      0 :::2049                 :::*                                -
udp6       0      0 :::111                  :::*                                2052/rpcbind
udp6       0      0 :::56988                :::*                                3102/rpc.mountd
udp6       0      0 :::44468                :::*                                3102/rpc.mountd
udp6       0      0 :::956                  :::*                                2052/rpcbind

After:

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      723/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1025/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      1025/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
udp        0      0 127.0.0.53:53           0.0.0.0:*                           723/systemd-resolve
udp        0      0 10.0.2.15:68            0.0.0.0:*                           651/systemd-network

Client

Before:

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      2058/rpcbind
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      727/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      914/sshd
tcp6       0      0 :::111                  :::*                    LISTEN      2058/rpcbind
tcp6       0      0 :::22                   :::*                    LISTEN      914/sshd
udp        0      0 10.0.2.15:68            0.0.0.0:*                           665/systemd-network
udp        0      0 0.0.0.0:111             0.0.0.0:*                           2058/rpcbind
udp        0      0 0.0.0.0:962             0.0.0.0:*                           2058/rpcbind
udp        0      0 127.0.0.53:53           0.0.0.0:*                           727/systemd-resolve
udp6       0      0 :::111                  :::*                                2058/rpcbind
udp6       0      0 :::962                  :::*                                2058/rpcbind

After:

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      727/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      914/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      914/sshd
udp        0      0 10.0.2.15:68            0.0.0.0:*                           665/systemd-network
udp        0      0 127.0.0.53:53           0.0.0.0:*                           727/systemd-resolve

Bonus: disabling IPv6

If you don't use IPv6, you can reduce the number of open ports in the netstat output further from

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      366/sshd
tcp6       0      0 :::2049                 :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      366/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           350/dhclient

to

$ sudo netstat -putnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      390/sshd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           350/dhclient

by doing

$ sudo vim /etc/default/grub
...
GRUB_CMDGRUB_CMDLINE_LINUX_DEFAULT="... ipv6.disable=1"
...

$ sudo update-grub
$ sudo reboot

Disabling IPv6 with sysctl alone doesn't seem to have the desired effect, that's why I'm using kernel options in the boot loader.