Michael's Daemonic Doodles

...blogging bits of BSD

Trusted Package Distribution with pkgng

pkgng is FreeBSD's new approach to replace its aging pkg_* utility infrastructure. In this post I'll show how to set up a private package repository and - after applying a patch to libfetch - use SSL certificates to establish bidirectional trust. This is on top of pkgng's existing repository signing mechanism, which I will cover at the end of this post.

Motivation

By default pkgng does not use any kind of authentication mechanism. It uses libfetch for transport, which allows for SSL encrypted transport, but only without any certificate checking. Since you cannot verify the peer, using any kind of password based authentication mechanism to authenticate the client would be dangerous. All of this is no problem for FreeBSD itself, since it is an open source project and therefore client authentication is not a real priority (spreading software is one of the main goals of the project after all).

It's different when you start building your own packages, which may contain proprietary software, intellectual property or any other private data. Thanks to pkgng, building packages got so easy that it became attractive for many different applications, so I could see how these uses of packages will increase in the future. So in this case you really want to make sure that only authorized clients have access to your package repository. Using firewall rules only gets you this far though, so using client certificates are a relatively straightforward and well understood mechanism to segregate permissions for different hosts within your organization.

Note

This tutorial assumes that you're working on a newly installed host (e.g. FreeBSD jails). It will work on a configured host as well, but you'll be in charge of figuring out what the changes mean for your particular setup.

Architecture

The idea is to create a package site using the nginx web server that can host multiple package repositories for various (groups of) hosts. All transport will happen over HTTPS, so it will be encrypted and the client can verify it is talking to a legitimate host before any transmissions happen (and potential weaknesses in the pkg code can be exploited). At the same time, the client presents a certificate to the package host, which allows it to authenticate the client and select the correct package repository.

This tutorial involved three hosts:

  • package master (master)
  • package client (client)
  • package CA (ca)

Usually you'll have at least one master and one client. Storing the CA on a separate host is considered best practice, but you can also store this on the master (for small setups this can be sufficient).

Note

This tutorial shows only one possible option for how to manage your certificates. You might already have advanced certificate management policies in place or just use your favorite tool like security/tinyca or openssl ca. It's also assumed that you don't create certificate revocation lists (which you could add easily to this setup), but just replace the certificate infrastructure in the event of a suspected security incident. Also, I will not give any advice on how to chose strong passwords.

Setting up the general infrastructure

This will only need to be done once.

Install pkgng (master)

Make sure to install pkgng from ports, since it always has the latest version.

cd /usr/ports/ports-mgmt/pkg
make install clean
cd /var/tmp
pkg_create -b pkg-\*
pkg2ng
echo 'WITH_PKGNG=yes' >> /etc/make.conf

Note

pkg_create is used to create a binary package of pkg which you can then install on client hosts using pkg_add.

Install nginx (master)

Make sure to enable the HTTP_SSL option in its configuration dialog.

cd /usr/ports/www/nginx
make config
make install clean
echo 'nginx_enable="YES"' >> /etc/rc.conf

Create nginx server certificate (master)

There is no real reason why the server certificate should be issued by the same CA as client certificates (unless you have already have an elaborate PKI in place). So for the sake of this tutorial we'll just create a self- signed certificate clients will trust.

For the sake of example this is called pm01.example.org (for package master 1), since we might want to add additional package hosts to our setup later on.

cd /usr/local/etc/nginx
openssl req -x509 -nodes -days 3650 \
-subj '/C=DE/ST=BY/L=Munich/O=Example Ltd/CN=pm01.example.org' \
-newkey rsa:2048 -keyout pm01.example.org.key -out pm01.example.org.crt
chmod 400 *.key *.crt

Note

In case you want to use a password on the server key, remove the nodes parameter from the command above. You'll have to enter its password on every startup of nginx though.

Create simple CA (ca)

Best case, the CA should be on a dedicated device (e.g. laptop not connected to a network). For most setups a separate server or even placing it on the master should be acceptable.

OpenSSL will ask you for a password for the CA key.

mkdir /usr/local/etc/pkgca
chown 700 /usr/local/etc/pkgca
cd /usr/local/etc/pkgca
openssl req -x509 -days 3650 \
-subj '/C=DE/ST=BY/L=Munich/O=Example Ltd/OU=pkgng/CN=PackageCA' \
-newkey rsa:2048 -keyout pkgca.key -out pkgca.crt
echo "01" > serial
chmod 400 *.key *.crt

Configure and start nginx (master)

Copy pkgca.crt from ca host to the master to /usr/local/etc/nginx/pkgca.crt.

Create a custom nginx.conf:

cat >/usr/local/etc/nginx/nginx.conf <<"EOF"
worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    keepalive_timeout  65;

    ssl                  on;
    ssl_certificate      pm01.example.org.crt;
    ssl_certificate_key  pm01.example.org.key;

    ssl_session_timeout  5m;

    ssl_protocols  SSLv3 TLSv1;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;
    ssl_client_certificate pkgca.crt;
    ssl_verify_client on;
    ssl_verify_depth 1;

    map $ssl_client_s_dn $ssl_client_s_dn_cn {
        default "";
        ~/CN=(?<CN>[^/]+) $CN;
    }

    log_format  main  '$remote_addr - $ssl_client_s_dn_cn [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx-access.log  main;

    server {
        listen       443 ssl default_server;
        server_name  pm01.example.org;
        root   /usr/local/www/nginx;

        # redundant check
        if ($ssl_client_verify != SUCCESS) {
            return 496;
        }

        # make sure there are no fishy characters in cn
        if ($ssl_client_s_dn_cn !~ "^[a-z0-9]{1,10}$") {
            return 496;
        }

        location / {
            root   /usr/local/www/pkg/$ssl_client_s_dn_cn;
        }
    }
}
EOF
service nginx start

This configuration ensures only trusted clients (read: clients using a certificate signed by the package CA) are allowed to access the server. The CN part of the certificate is used as the user name and will show up in nginx-access.log.

Each package repository is stored in /usr/local/www/pkg/<CN>. You can use symbolic links to group clients.

Create default repository (master)

The next step is to create a default repository that will contain the basic package repository most hosts will access. Typically most hosts will just need a simple vanilla set of packages.

mkdir -p /usr/local/www/pkg/default
cd /usr/local/www/pkg/default
pkg create -a
pkg repo .

Client configuration

This needs to be done for every client.

Install pkgng (client)

First, copy the tbz file created on the master above (/var/tmp/pkg-version.tbz) to the client. Install it and make convert to pkgng:

pkg_add /path/to/pkg-x.y.tbz
pkg2ng
echo 'WITH_PKGNG=yes' >> /etc/make.conf

Patching libfetch (client)

pkgng uses libfetch (which is part of base and therefore doesn't add any additional dependencies to pkg). Libfetch's support for HTTPS is fairly minimal, so the first step was to implement proper SSL support for it.

I submitted a patch - and several improvements - as a PR to the FreeBSD project, which is still under review. This patch was created on FreeBSD 9.1-RELEASE, but should also apply for the most part on other recent versions of the OS.

So the first step is to patch libfetch:

  1. Obtain the current version of patch
  2. Apply the patch
cd /usr/src
patch < /path/to/patch
cd /usr/src/lib/libfetch
make obj && make depend && make && make install
cd /usr/src/usr.bin/fetch
make obj && make depend && make && make install

Note

For the purpose of this tutorial, patching the client is sufficient. On the other hand, making libfetch verify server certificates is a valuable feature in general, so I'd suggest to patch all your systems involved. If you do that, I'd also recommend installing security/ca_root_nss with the ETCSYMLINK option enabled.

Create client CSR (client)

cd /etc/ssl
openssl req \
-subj "/C=DE/ST=BY/L=Munich/O=Example Ltd/OU=pkgng/CN=client01" \
-new -newkey rsa:2048 -out pkgclient01.csr \
-keyout pkgclient01.key -nodes
chmod 400 *.key

Note

You can also use a password on this key, in which case you'll have to enter its password on every single fetch request. This can get pretty annoying and therefore is not recommended.

Sign the client CSR (ca)

Copy the certificate request created above to the ca host to /usr/local/etc/pkgca/pkgclient01.csr.

Sign it using openssl:

cd /usr/local/etc/pkgca
openssl x509 -req -days 3650 -in pkgclient01.csr -out pkgclient01.crt \
-CA pkgca.crt -CAkey pkgca.key -CAserial serial

Finish client configuration (client)

  1. Copy pkgclient01.crt to /etc/ssl/pkgclient01.crt
  2. Place the content of pm01.example.org.crt from master into /etc/ssl/pkgmaster.pem.
  3. Make sure all files in /etc/ssl are only writable by root and key files are only readable by root.
  4. Change packagesite to the example host (this overwrites your existing pkg.conf):
echo "packagesite: https://pm01.example.org" > /usr/local/etc/pkg.conf
  1. In case you're not using official DNS names or working within a private network without access to DNS, add the host to /etc/hosts, e.g.
192.168.100.2 pm01.example.org

Grant access to client (master)

There are two different options:

  1. The client should get access to our default set of packages:
cd /usr/local/www/pkg
ln -s default client01
  1. The client should get access to a custom package repository (e.g. only containing dialog4ports):
mkdir /usr/local/www/pkg/client01
cd /usr/local/www/pkg/client01
pkg create dialog4ports
pkg repo .

To revoke access, simple remove the directory or symlink matching the client's CN.

Testing (client)

Before testing, issue the following command to create an alias for the pkg command:

alias pkg="SSL_CLIENT_KEY_FILE=/etc/ssl/pkgclient01.key \
SSL_CLIENT_CERT_FILE=/etc/ssl/pkgclient01.crt \
SSL_CA_CERT_FILE=/etc/ssl/pkgmaster.pem pkg"

It's best to add this to your/the root users .profile so you won't have to issue it every time you're using pkg.

Next, try to update the catalog and install dialog4ports:

pkg update
pkg install -y dialog4ports

Adding repository signatures

Now that access to the package repository is controlled and all parties are trusted at the transport level, it's time to add repository signatures. This provides additional advantages:

  • The integrity of the repository can be verified by the client at any time (not just protecting it from malicious users, but also human error).
  • Package repositories for different clients can be hosted on the same package master, but signed by different authorities/teams within the organization (like the database team provides its own repository and the database hosts only trust repositories signed by their key) and therefore provides
  • Segregation of power

For the rest of this tutorial it's assumed the pkg repository is created on the master.

Create a repository signing key (master)

For the sake of example we put the into root's home directory. The exact location depends on who's doing the actual signing.

OpenSSL will ask you for a password for the signing key.

openssl genrsa -aes256 -out /root/reposign.privkey 2048
chmod 400 /root/reposign.privkey
openssl rsa -in /root/reposign.privkey -pubout -out /root/reposign.pubkey

Sign the repository (master)

Signs the repository for client01. Apply the same procedure for default and all other repositories.

cd /usr/local/www/pkg/client01
pkg repo . /root/reposign.privkey

Configure client to verify signature (client)

On the client, copy reposign.pubkey to /etc/ssl/reposign.pubkey and add signature checking to the configuration file:

echo "pubkey: /etc/ssl/reposign.pubkey" >> /usr/local/etc/pkg.conf

Testing (client)

Note

This assumes you issued the alias command described above or already put it into your .profile

pkg update -f
pkg delete -y dialog4ports
pkg install -y dialog4ports

Feel free to play with this a little to make sure the signature gets actually checked.

Conclusion

pkgng is a great improvement in how FreeBSD binary packages are created and managed. Adding a little bit of extra security beyond what's provided out of the box was mostly a matter of improving libfetch. I hope my patches will be incorporated into the project in one way or another, since fetch and its users could greatly benefit from real SSL support anyway. I put a lot of effort into the patch, so it conforms to all the relevant RFCs, including RFC6125, sections 6.4.3 and 7.2, which clarifies RFC2818 and RFC3280.

The entire procedure is not too complicated and client setup especially could be easily scripted to further ease the process.

Feel free to contact me if you find any mistakes in this tutorial or to share your ideas for improvements.