Org-in-a-Box - Kerberos & NFS
The second post covering my personal project of an "organization in a box". Based on the initial architecture this article goes over partially setting up the first two authentication servers in the project leveraging MIT Kerberos and NFSv4. These hosts will serve as the base for the rest of the Org-in-a-Box project and a subsequent post will cover the LDAP and DBIS configuration on these hosts.
Host Prep Work
Before we start down the Kerberos and NFS path we have a bit of prep work to get done. The very first thing is setting a copy of Raspberry Pi OS installed on our Pis following the installation guide provided. Secondly, since I'm running with the headless version to cut back on resource usage we need to make sure to include a file on the SD card, aptly called "ssh", to ensure that SSH is enabled by default. In recent versions of the OS they disabled this service by default and in a headless setup becomes a pain to configure without having to play with HDMI cables.
Since I opted for a proper Pi case with built in fans, we also need to make sure that the pin setup is attached to the board properly.
Last but not least, once we have logged on the host for the first time we should go ahead and do an initial vanilla update to ensure we apply any available patches or new versions.
With that we can move on to the good stuff - getting Kerberos up and running.
Most of the installation process is well documented under MIT's Kerberos 1.17 guide. While many sources of information were used MIT's guide served as the base.
Let us start by getting the relevant 1.17 binaries installed.
With no cold feet in sight, let's dive on in and create our
PADRAIGNIX.COM Kerberos realm.
Before we start creating new principals we also want to ensure we enable admin entitlements for our
/admin accounts. We do this by uncommenting the line in kadm5.acl.
Lastly we need to set up our kdc.conf file which serves as the main configuration for the service. Encryption types, ticket options such as renewable and forwadable, and KDC servers in our environment are all defined in this file. You'll also notice I cheated and put
auth02 (which will be our replica KDC) as I knew I was going to be building that one shortly as well.
With a quick service restart we confirm everything is up and running and we can start populating the Kerberos database.
Alright, time to add myself, my /admin variant, and the auth01 host principal to our database. I should have configured policies prior to doing so, however I plan on going back later on and creating those to enforce password complexity and various other nuances. Since we don't have any principals at the moment we will use kadmin.local instead of kadmin to interact with the database. The difference between the two is that kadmin.local directly accesses the KDC database, while kadmin performs operations using kadmind - a kerberizes service. We will use kadmin to create the host key once our user principals are created.
Excellent we now have our user, admin, and host principals created, so now what? Let's tie this in to the login authentication. The plan is to base our host login on principal/password combinations held within Kerberos, rather than the usual passwd/shadow method. Let's see if we can add a few quality of life items while we are at it as well.
For our initial approach to work we will need to manually add our user to the host and the various groups required. This will not be the long term way of doing so as we will be leveraging LDAP and DBIS in the future. However, to get things started, let us get it done real quick.
Now we have our local user padraignix, and we have a Kerberos principal padraignix, however if we tried to use the Kerberos password to assume the identity (su, ssh, etc.) we would see it fails with incorrect password. In fact, the local user does not even have a password set at the moment. What we will need to do is get an additional set of PAM libaries to help bridge that authentication for us.
What we aiming for is to start with Kerberos authentication and if that is successful, permit the login as well as get an initial ticket issued by the KDC. If it is not successful, then we default back to local Unix authentication without any Kerberos tickets. As an added bonus with the PAM configuration we have the ability to create a homedirectory on first login if it does not exist already - so let's select that option as well.
Lessons Learned Section
Unless you like to live dangerously make sure you are also leaving Unix authentication enabled. Without it, should you have any corruption with your Kerberos database or service, or if you're like me and you try to fit a square peg into a round hole and brick your host, you will not have the ability to get back on to fix whatever you broke.
I have an out-of-band (OOB) account that I added to the hosts without a corresponding Kerberos principal. The plan was per the above if anything went wrong, I could log in without going through Kerberos and fix anything needed. However with that PAM option disabled this account became as useless as the host.
With both Kerberos and Unix options selected the PAM configuration looks like the following - first attempting Kerberos and falling back to local auth if unsuccessful.
Now all that is left is to log in as padraignix and see if it works.
So far so good. Our password was accepted and the homedirectory was created on initial login. Now how about Kerberos tickets?
Excellent! At this point we earned a quick breather... alright breather done. Time to move on to NFS!
With our Kerberos service running time to see if we can get NFSv4 working and secure it. Since this is generally a test infrastructure I choose a small form factor USB dongle as the eventual NFS share. The plan is to tie it together by not only leveraging Kerberos for the communication, but use the NFS share to create a shared
/home space that our hosts will leverage.
The first step was to get more information on our USB stick.
Alright time to get it mounted normally.
Unfortunately the stick was formatted FAT32 by default. To leverage it for our NFS purposes we will need to format it to ext4 so let's get that done.
If we add the uuid to
/etc/fstab we should be able to access it locally on the mounted host. Before we get fancy with our second host it would be a good idea to confirm everything is working so far post reboot.
Great, time to bump up the difficulty! Starting off, we should ensure that only V4 is running as we want to enforce Kerberized security. Debian has a good NFS Server Guide that helped understand the various options and configurations we would need.
To which we updated.
To ensure owner and group information is available from client hosts the imapd service Mapping needs to be configured and running on the NFS server.
Lastly on our first host, we need to create a new nfs Kerberos principal and add it to our host keytab.
At this point we are automatically mounting /home from our USB stick locally on
auth01. With the configuration work done above all that is left is to serve the share using NFS onthe LAN by configuring
There are three different Kerberos options for serving NFS. Debian has a good NFS Kerberos page that breaks down those options:
- krb5 Use Kerberos for authentication only.
- krb5i Use Kerberos for authentication, and include a hash with each transaction to ensure integrity.
- krb5p Use Kerberos for authentication, and encrypt all traffic between the client and server.
I will skip over the initial setup as it was covered above. The only difference is that no new Kerberos realms are initialized on the second host, we only make sure the client utilities are available for now.
auth01 we will go ahead and create a new set of principals - a host principal and nfs principal for
auth02. We will add them to a new keytab and send that keytab over to the host.
With the keytab in place we should have everything needed to be able to mount our share. Let's try doing that manually.
It took entirely too long to realize I was using sec=krb5 instead of sec=krb5p as described in the previous section. Once that hiccup was cleared I was able to get it mounted properly.
... well almost properly. The share IS mounted and the the location is R/W, however it seems there is something wrong with the ownership and groups. It took me a bit of reading to understand what was going on. Ultimately two things failed me here. First, I needed to have the same user and group information on
auth02 as I did on
auth01. This was simple, yet annoying, to fix and I was able to create my uid=1001 padraignix account locally on
auth02. Manually creating accounts on various hosts sounds like it will be a messy, error-prone situation however and we will be solving this requirement further on with LDAP/DBIS. However for the moment, the second issue we had to fix once local accounts were created was the imapd service mapping. If you remember above when I covered the imapd configuration there was a [Mapping] section. Turns out we needed to make sure this was commented out to allow the service to serve the existing owners and groups. A quick modification and refresh on auth01...
and then it looks much better on auth02!
One more configuration successful. No time stop now however as there is one more aspect we want to address in this post - Kerberos replication.
Alright last topic for this post. You would think that setting up replication between two Kerberos instances should be trivial compared to getting NFSv4 setup, however this proved to not be accurate. It was during this portion that my "bricking" situation happened.
The first requirement is getting
kpropd installed on both hosts.
Continuing with the Kerberos 1.17 guide process we create/add the replica KDC host principal to the Keytabs and subsequently add the primary KDC host principal to the replica's keytab.
Then we follow up by copying over the following configuration files to
Having manually started
kpropd on the replica KDC I then attempted manually triggering a replication. This is the moment that started the unfortunate bricking situation started.
That doesn't seem right... I took a look at the logs to see if I could make sense of it.
What I came to understand was that there was a KVNO (Key version number) difference in my host keytabs. Unfortunately I did not figure this out before attempting to reboot
auth01, which combined with the removal of the PAM Unix local auth configuration, essentially locked me out of the host. The irony was that the host was able to boot successfully and serve Kerberos requests from
auth02 which made the decision to blow away 01 and start from scratch all the more painful. Thankfully my notes were detailed enough to go back through the entire process under an hour, so I will chalk it up as a valuable learning experience.
With keytabs sorted out and everything back in working order it is time to re-attempt the manual sync.
There we go! A quick test by commenting out the primary KDC from kdc.conf on
auth02 and trigerring a kdestroy/kinit, confirmed by local logs on 02, was performed and successful. All that is left is to automate the process and make sure everything starts on boot properly.
Step 1 is easy enough. Script up the replication process and add it as a cron job on
Another quick manual test to confirm it's working.
Then adding the proper crontab entry to run this every 10 minutes.
Unfortunately there was one final complication before we close the book on this portion of the build. For a reason unknown at the time when
auth02 was rebooted, the kpropd service would start in a failed state. It would try and start up, but end up in a failed stated. It took a bit of digging but I was able to narrow it down.
What ended up happening was kpropd attempted to start before the network and services were fully ready. I tried a few variations under Unit->After for the systemctl configuration, however defaulting back to tried and true ExecStartPre=sleep ended up working.
And if we check systemctl we see that that the service did wait the 10 seconds and that it ultimately started successfully. Problem solved.
Summary & Next Steps
There we go. Kerberos primary and secondaries, with replication, and NFSv4 leveraging krb5p done and dusted. Next step is to get LDAP and DBIS setup which will remove our manual user creation and propagation requirements.
As always thanks folks, until next time!