Trusted Package Distribution with pkgng
pkgng is FreeBSD's new approach to replace its aging pkg_* utility infrastructure. In this post I'll show how to set up a private package repository and - after applying a patch to libfetch - use SSL certificates to establish bidirectional trust. This is on top of pkgng's existing repository signing mechanism, which I will cover at the end of this post.
Motivation
By default pkgng does not use any kind of authentication mechanism. It uses libfetch for transport, which allows for SSL encrypted transport, but only without any certificate checking. Since you cannot verify the peer, using any kind of password based authentication mechanism to authenticate the client would be dangerous. All of this is no problem for FreeBSD itself, since it is an open source project and therefore client authentication is not a real priority (spreading software is one of the main goals of the project after all).
It's different when you start building your own packages, which may contain proprietary software, intellectual property or any other private data. Thanks to pkgng, building packages got so easy that it became attractive for many different applications, so I could see how these uses of packages will increase in the future. So in this case you really want to make sure that only authorized clients have access to your package repository. Using firewall rules only gets you this far though, so using client certificates are a relatively straightforward and well understood mechanism to segregate permissions for different hosts within your organization.
Note
This tutorial assumes that you're working on a newly installed host (e.g. FreeBSD jails). It will work on a configured host as well, but you'll be in charge of figuring out what the changes mean for your particular setup.
Architecture
The idea is to create a package site using the nginx web server that can host multiple package repositories for various (groups of) hosts. All transport will happen over HTTPS, so it will be encrypted and the client can verify it is talking to a legitimate host before any transmissions happen (and potential weaknesses in the pkg code can be exploited). At the same time, the client presents a certificate to the package host, which allows it to authenticate the client and select the correct package repository.
This tutorial involved three hosts:
- package master (master)
- package client (client)
- package CA (ca)
Usually you'll have at least one master and one client. Storing the CA on a separate host is considered best practice, but you can also store this on the master (for small setups this can be sufficient).
Note
This tutorial shows only one possible option for how to manage your certificates. You might already have advanced certificate management policies in place or just use your favorite tool like security/tinyca or openssl ca. It's also assumed that you don't create certificate revocation lists (which you could add easily to this setup), but just replace the certificate infrastructure in the event of a suspected security incident. Also, I will not give any advice on how to chose strong passwords.
Setting up the general infrastructure
This will only need to be done once.
Install pkgng (master)
Make sure to install pkgng from ports, since it always has the latest version.
cd /usr/ports/ports-mgmt/pkg make install clean cd /var/tmp pkg_create -b pkg-\* pkg2ng echo 'WITH_PKGNG=yes' >> /etc/make.conf
Note
pkg_create is used to create a binary package of pkg which you can then install on client hosts using pkg_add.
Install nginx (master)
Make sure to enable the HTTP_SSL option in its configuration dialog.
cd /usr/ports/www/nginx make config make install clean echo 'nginx_enable="YES"' >> /etc/rc.conf
Create nginx server certificate (master)
There is no real reason why the server certificate should be issued by the same CA as client certificates (unless you have already have an elaborate PKI in place). So for the sake of this tutorial we'll just create a self- signed certificate clients will trust.
For the sake of example this is called pm01.example.org (for package master 1), since we might want to add additional package hosts to our setup later on.
cd /usr/local/etc/nginx openssl req -x509 -nodes -days 3650 \ -subj '/C=DE/ST=BY/L=Munich/O=Example Ltd/CN=pm01.example.org' \ -newkey rsa:2048 -keyout pm01.example.org.key -out pm01.example.org.crt chmod 400 *.key *.crt
Note
In case you want to use a password on the server key, remove the nodes parameter from the command above. You'll have to enter its password on every startup of nginx though.
Create simple CA (ca)
Best case, the CA should be on a dedicated device (e.g. laptop not connected to a network). For most setups a separate server or even placing it on the master should be acceptable.
OpenSSL will ask you for a password for the CA key.
mkdir /usr/local/etc/pkgca chown 700 /usr/local/etc/pkgca cd /usr/local/etc/pkgca openssl req -x509 -days 3650 \ -subj '/C=DE/ST=BY/L=Munich/O=Example Ltd/OU=pkgng/CN=PackageCA' \ -newkey rsa:2048 -keyout pkgca.key -out pkgca.crt echo "01" > serial chmod 400 *.key *.crt
Configure and start nginx (master)
Copy pkgca.crt from ca host to the master to /usr/local/etc/nginx/pkgca.crt.
Create a custom nginx.conf:
cat >/usr/local/etc/nginx/nginx.conf <<"EOF" worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; keepalive_timeout 65; ssl on; ssl_certificate pm01.example.org.crt; ssl_certificate_key pm01.example.org.key; ssl_session_timeout 5m; ssl_protocols SSLv3 TLSv1; ssl_ciphers HIGH:!aNULL:!MD5; ssl_prefer_server_ciphers on; ssl_client_certificate pkgca.crt; ssl_verify_client on; ssl_verify_depth 1; map $ssl_client_s_dn $ssl_client_s_dn_cn { default ""; ~/CN=(?<CN>[^/]+) $CN; } log_format main '$remote_addr - $ssl_client_s_dn_cn [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx-access.log main; server { listen 443 ssl default_server; server_name pm01.example.org; root /usr/local/www/nginx; # redundant check if ($ssl_client_verify != SUCCESS) { return 496; } # make sure there are no fishy characters in cn if ($ssl_client_s_dn_cn !~ "^[a-z0-9]{1,10}$") { return 496; } location / { root /usr/local/www/pkg/$ssl_client_s_dn_cn; } } } EOF service nginx start
This configuration ensures only trusted clients (read: clients using a certificate signed by the package CA) are allowed to access the server. The CN part of the certificate is used as the user name and will show up in nginx-access.log.
Each package repository is stored in /usr/local/www/pkg/<CN>. You can use symbolic links to group clients.
Create default repository (master)
The next step is to create a default repository that will contain the basic package repository most hosts will access. Typically most hosts will just need a simple vanilla set of packages.
mkdir -p /usr/local/www/pkg/default cd /usr/local/www/pkg/default pkg create -a pkg repo .
Client configuration
This needs to be done for every client.
Install pkgng (client)
First, copy the tbz file created on the master above (/var/tmp/pkg-version.tbz) to the client. Install it and make convert to pkgng:
pkg_add /path/to/pkg-x.y.tbz pkg2ng echo 'WITH_PKGNG=yes' >> /etc/make.conf
Patching libfetch (client)
pkgng uses libfetch (which is part of base and therefore doesn't add any additional dependencies to pkg). Libfetch's support for HTTPS is fairly minimal, so the first step was to implement proper SSL support for it.
I submitted a patch - and several improvements - as a PR to the FreeBSD project, which is still under review. This patch was created on FreeBSD 9.1-RELEASE, but should also apply for the most part on other recent versions of the OS.
So the first step is to patch libfetch:
- Obtain the current version of patch
- Apply the patch
cd /usr/src patch < /path/to/patch cd /usr/src/lib/libfetch make obj && make depend && make && make install cd /usr/src/usr.bin/fetch make obj && make depend && make && make install
Note
For the purpose of this tutorial, patching the client is sufficient. On the other hand, making libfetch verify server certificates is a valuable feature in general, so I'd suggest to patch all your systems involved. If you do that, I'd also recommend installing security/ca_root_nss with the ETCSYMLINK option enabled.
Create client CSR (client)
cd /etc/ssl openssl req \ -subj "/C=DE/ST=BY/L=Munich/O=Example Ltd/OU=pkgng/CN=client01" \ -new -newkey rsa:2048 -out pkgclient01.csr \ -keyout pkgclient01.key -nodes chmod 400 *.key
Note
You can also use a password on this key, in which case you'll have to enter its password on every single fetch request. This can get pretty annoying and therefore is not recommended.
Sign the client CSR (ca)
Copy the certificate request created above to the ca host to /usr/local/etc/pkgca/pkgclient01.csr.
Sign it using openssl:
cd /usr/local/etc/pkgca openssl x509 -req -days 3650 -in pkgclient01.csr -out pkgclient01.crt \ -CA pkgca.crt -CAkey pkgca.key -CAserial serial
Finish client configuration (client)
- Copy pkgclient01.crt to /etc/ssl/pkgclient01.crt
- Place the content of pm01.example.org.crt from master into /etc/ssl/pkgmaster.pem.
- Make sure all files in /etc/ssl are only writable by root and key files are only readable by root.
- Change packagesite to the example host (this overwrites your existing pkg.conf):
echo "packagesite: https://pm01.example.org" > /usr/local/etc/pkg.conf
- In case you're not using official DNS names or working within a private network without access to DNS, add the host to /etc/hosts, e.g.
192.168.100.2 pm01.example.org
Grant access to client (master)
There are two different options:
- The client should get access to our default set of packages:
cd /usr/local/www/pkg ln -s default client01
- The client should get access to a custom package repository (e.g. only containing dialog4ports):
mkdir /usr/local/www/pkg/client01 cd /usr/local/www/pkg/client01 pkg create dialog4ports pkg repo .
To revoke access, simple remove the directory or symlink matching the client's CN.
Testing (client)
Before testing, issue the following command to create an alias for the pkg command:
alias pkg="SSL_CLIENT_KEY_FILE=/etc/ssl/pkgclient01.key \ SSL_CLIENT_CERT_FILE=/etc/ssl/pkgclient01.crt \ SSL_CA_CERT_FILE=/etc/ssl/pkgmaster.pem pkg"
It's best to add this to your/the root users .profile so you won't have to issue it every time you're using pkg.
Next, try to update the catalog and install dialog4ports:
pkg update pkg install -y dialog4ports
Adding repository signatures
Now that access to the package repository is controlled and all parties are trusted at the transport level, it's time to add repository signatures. This provides additional advantages:
- The integrity of the repository can be verified by the client at any time (not just protecting it from malicious users, but also human error).
- Package repositories for different clients can be hosted on the same package master, but signed by different authorities/teams within the organization (like the database team provides its own repository and the database hosts only trust repositories signed by their key) and therefore provides
- Segregation of power
For the rest of this tutorial it's assumed the pkg repository is created on the master.
Create a repository signing key (master)
For the sake of example we put the into root's home directory. The exact location depends on who's doing the actual signing.
OpenSSL will ask you for a password for the signing key.
openssl genrsa -aes256 -out /root/reposign.privkey 2048 chmod 400 /root/reposign.privkey openssl rsa -in /root/reposign.privkey -pubout -out /root/reposign.pubkey
Sign the repository (master)
Signs the repository for client01. Apply the same procedure for default and all other repositories.
cd /usr/local/www/pkg/client01 pkg repo . /root/reposign.privkey
Configure client to verify signature (client)
On the client, copy reposign.pubkey to /etc/ssl/reposign.pubkey and add signature checking to the configuration file:
echo "pubkey: /etc/ssl/reposign.pubkey" >> /usr/local/etc/pkg.conf
Testing (client)
Note
This assumes you issued the alias command described above or already put it into your .profile
pkg update -f pkg delete -y dialog4ports pkg install -y dialog4ports
Feel free to play with this a little to make sure the signature gets actually checked.
Conclusion
pkgng is a great improvement in how FreeBSD binary packages are created and managed. Adding a little bit of extra security beyond what's provided out of the box was mostly a matter of improving libfetch. I hope my patches will be incorporated into the project in one way or another, since fetch and its users could greatly benefit from real SSL support anyway. I put a lot of effort into the patch, so it conforms to all the relevant RFCs, including RFC6125, sections 6.4.3 and 7.2, which clarifies RFC2818 and RFC3280.
The entire procedure is not too complicated and client setup especially could be easily scripted to further ease the process.
Feel free to contact me if you find any mistakes in this tutorial or to share your ideas for improvements.