Posts on the crosseroads

Creating a Two-Tier CA using Yubikeys

Thu, 30 Jun 2022 00:00:00 +0000

So you have more Yubikeys than sense? Great, me too! Let’s make a multi-tier certificate authority!

By the end of this article, we’ll have a fully-functioning two-tier CA where the private keys for the CAs are stored on Yubikeys. In addition, we’ll make sure that we generate the private keys directly on the Yubikeys for zero chance of key compromise. All keys will be elliptic-curve (ECC), and–barring mistakes on my part–the CA should fully adhere to RFC 5759, NSA’s Suite B Certificate and Certificate Revocation List (CRL) Profile.

ButWhy.gif

For small- or low-“traffic” CAs, two Yubikeys at a cost of $45 each could be much more feasible than two YubiHSM 2 devices at $650 each, or two Nitrokey HSM for €99 each.
I’ve read other articles about using a single Yubikey as a root CA, but all of them seem to want to generate the private key using OpenSSL and then import it to the Yubikey. Because of this, they usually come with dire warnings about ensuring you generate the keys using a LiveCD distribution of Linux, etc., etc. By generating the key directly on the Yubikey, you can do all this in an OS environment you’re comfortable with and eschew all the LiveCD contortions, plus have confidence that they private keys cannot accidentally leak at any time.
I wanted the challenge!

Note: I am not a security engineer. If someone who is a security engineer says this is not a smart idea for your use case, then please listen to them, not me. ;-) My ultimate goal with this was to use it to secure a handful of home-lab servers and have fun doing it.

Prerequisites

I probably shouldn’t have to say this, but following these instructions will wipe out whatever you might have in the PIV slot(s) of your Yubikey(s). Please only follow these instructions on devices you are okay with overwriting.

I’m using macOS, but as long as you have the various packages and commands installed, you should be able to do this on most any platform. (Be aware that I haven’t actually tested that statement, but I believe it to be true!)

Things you’ll need:

Two Yubikeys
OpenSSL
libp11 (for the pkcs11 OpenSSL engine)
opensc (for the pkcs11-tool1 command)
gnutls (for the p11tool command)
yubico-piv-tool and libykcs11, which comes with it
ykman (optional)

You’ll want to know the location where your package manager installed the libykcs11.{so,dylib,dll} shared library. On my macOS laptop, I’ve installed the prebuilt package from Yubico and it installed the file at /usr/local/lib/libykcs11.dylib.

Unless you’re using a quite old version of OpenSSL, you should not need the path to the OpenSSL pkcs11 engine from libp11. You can test to see if you do need it with the following command:

$ openssl engine -t pkcs11
(pkcs11) pkcs11 engine
     [ available ]

If this command fails, you’ll need to tell OpenSSL where the engine is with the dynamic_path statement in the OpenSSL config files below.

Find your Yubikeys

All of the commands we use will want to know which “slot” your Yubikey is in. After plugging one or more in, find them like so:

$ pkcs11-tool --module /usr/local/lib/libykcs11.dylib -T
Available slots:
Slot 0 (0x0): Yubico Yubikey NEO OTP+U2F+CCID
  token label        : YubiKey PIV #0
  token manufacturer : Yubico (www.yubico.com)
  token model        : YubiKey NEO
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 1.0
  firmware version   : 0.13
  serial num         : 0
  pin min/max        : 6/64
Slot 1 (0x1): Yubico YubiKey CCID
  token label        : YubiKey PIV #5212376
  token manufacturer : Yubico (www.yubico.com)
  token model        : YubiKey YK4
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 1.0
  firmware version   : 4.33
  serial num         : 5212376
  pin min/max        : 6/64

For illustration purposes I’ve plugged both cards into my machine. I do not recommend keeping both plugged in when you start on the instructions below–you will get mixed up and do something silly like overwrite the root CA’s key with a new key because you thought you specified the intermediate CA card. (Ask me how I know.)

For this run-through, we’ll use the card shown here in slot 1–a Yubikey 4–for the root, and the card in slot 0–a Yubikey NEO–for the intermediate. I chose these particular cards out of my fleet because I wanted the root CA to have a 384-bit ECC key (secp384r1), which the Yubikey 4 supports. The intermediate will use a 256-bit key (secp256r1), for no other reason than it is the largest ECC key the NEO supports.

To make things easier, I suggest exporting a few environment variables with some relevant information. I’ll use these in the instructions below so that they’re easier to read.

$ export KEY_SLOT=0
$ export PIN=123456
$ export MGMT_KEY=010203040506070801020304050607080102030405060708
$ export LIBYKCS11=/usr/local/lib/libykcs11.dylib

(You may be thinking, “why are the PIN and management keys the defaults?!?” Well, for one it makes writing these instructions easier; secondly, it means that your super-secret PIN and management keys are not hanging about in memory. You can either change the PIN, PUK, and management key before following these instructions and update the environment variables above, or you can use the defaults and immediately change the values once you’re done with everything. Either way, I definitely recommend changing the values from the defaults as soon as you’re able.)

Okay, so let’s test one of the keys:

$ pkcs11-tool --module "$LIBYKCS11" \
    --login --login-type so         \
    --pin $PIN --so-pin $MGMT_KEY   \
    --slot $KEY_SLOT --id 2         \
    --key-type EC:secp256r1         \
    --test-ec

This should run a bunch of stuff and report back that everything looks good. If it does, let’s move on! If not, fix those errors before continuing.

Reset the Yubikeys

This step is optional, but highly encouraged so that you start from a clean slate. You can use the ykman command line utility or the Yubikey Manager GUI app to reset the PIV application on both Yubikeys. With ykman that will look like this:

$ ykman piv reset
WARNING! This will delete all stored PIV data and restore factory settings. Proceed? [y/N]: y
Resetting PIV data...
Success! All PIV data have been cleared from the YubiKey.
Your YubiKey now has the default PIN, PUK and Management Key:
        PIN:    123456
        PUK:    12345678
        Management Key: 010203040506070801020304050607080102030405060708

(Note that it will complain and error out if you have more than one Yubikey connected.)

Create the CA Structure and the Root CA

For the OpenSSL part of this guide, I will rely heavily on a series of articles on building an OpenSSL CA that is compliant with RFC 5759, the NSA’s Suite B Certificate and Certificate Revocation List (CRL) Profile. Those articles, in order, are:

I won’t be focusing much on the OpenSSL “infrastructure” side of things–this guide is about how to bring the Yubikeys into that process–so if you have questions or want to know why I make a certain decision regarding the CA itself, have a look at those articles.

Since we’re creating an OpenSSL CA, we need to create a directory structure for it.

$ cd /somewhere/important
$ mkdir -p ca/{private,certs,crl}
$ cd ca/
$ touch index.txt
$ echo 1000 > serial

We also need to create some configuration files for OpenSSL. First, create pkcs11.cnf, which we’ll use to tell OpenSSL where the libykcs11 library is. We will import this into the subsequent config files so that we don’t have to repeat ourselves too much.

openssl_conf = openssl_def

[openssl_def]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id    = pkcs11
MODULE_PATH  = /usr/local/lib/libykcs11.dylib

Second, create openssl_root.cnf, which will be the config used when creating the root CA:

.include pkcs11.cnf

[ ca ]
default_ca = CA_default

[ CA_default ]
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/certs
database          = $dir/index.txt
serial            = $dir/serial
rand_serial	  = no
RANDFILE          = $dir/private/.rand

# Our private key will reside on a Yubikey.
#private_key       =
certificate       = $dir/certs/ca.example.crt.pem

crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.example.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 3650

# Set this to match the root CA's key length.
default_md        = sha384

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 3650
preserve          = no
policy            = policy_strict
email_in_dn	  = no

[ policy_strict ]
# See the "Building a CA" articles above about the implications of these lines.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# This doesn't matter to us; we're using ECC, not RSA.
default_bits        = 4096

distinguished_name  = req_distinguished_name
string_mask         = utf8only
# Set this to match the root CA's key length.
default_md          = sha384

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Update these to suit your environment
countryName_default             = US
stateOrProvinceName_default     = WA
localityName_default            = Seattle
0.organizationName_default      = Example Enterprises
organizationalUnitName_default  = Example Enterprises Root CA

[ v3_ca ]
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always,issuer
basicConstraints        = critical, CA:true, pathlen:1
keyUsage                = critical, digitalSignature, cRLSign, keyCertSign

Create the Root CA Key

Create the Root CA’s private key on the Yubikey. This will overwrite any already-existing key in slot 9c–that’s the --id 2 part of the command–on the card. (Slot 9c holds certificates used for digital signatures. Read more about Yubikey certificate slots here.)

$ pkcs11-tool --module "$LIBYKCS11" \
    --login --login-type so         \
    --pin $PIN --so-pin $MGMT_KEY   \
    --slot $KEY_SLOT --id 2         \
    --key-type EC:secp384r1         \
    --keypairgen

Key pair generated:
Private Key Object; EC
  label:      Private key for Digital Signature
  ID:         02
  Usage:      decrypt, sign
  Access:     always authenticate, sensitive, always sensitive, never extractable, local
Public Key Object; EC  EC_POINT 384 bits
  EC_POINT:   046104295c28f6bca2b544cb2b858ad4c408562d2cf1b801464e3864107df1361a4d05e96c9e4cf8290ba6bd837862094f448c1caa7185068d01c691eab42a1142747ae43cfb04f1aab687efdac1a8d0b4cb0ab9851e03d91b91c0c71a7d2ecdec2f43
  EC_PARAMS:  06052b81040022
  label:      Public key for Digital Signature
  ID:         02
  Usage:      encrypt, verify
  Access:     local

After creating the private key, we need to discover its “URI”–yes, even private keys on a hardware authentication device have URIs–and add it as the private_key in openssl_root.cnf (the openssl ca command will require this later on when we’re signing the intermediate certificate). For this we’ll lean on the p11tool command. What we need to do is tell it to list all private key objects on the card. We then get to sort through the output until we find the object that corresponds with the private key we just created. Ready? Me, neither. Let’s go!

$ p11tool --provider=$LIBYKCS11 --list-keys --login
Token 'YubiKey PIV #5212376' with URL 'pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20%28www.yubico.com%29;serial=5212376;token=YubiKey%20PIV%20%235212376' requires user PIN
Enter PIN:
Object 0:
        URL: pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20%28www.yubico.com%29;serial=5212376;token=YubiKey%20PIV%20%235212376;id=%02;object=Private%20key%20for%20Digital%20Signature;type=private
        Type: Private key (EC/ECDSA)
        Label: Private key for Digital Signature
        Flags: CKA_PRIVATE; CKA_ALWAYS_AUTH; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE;
        ID: 02

Object 1:
        URL: pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20%28www.yubico.com%29;serial=5212376;token=YubiKey%20PIV%20%235212376;id=%19;object=Private%20key%20for%20PIV%20Attestation;type=private
        Type: Private key (RSA-2048)
        Label: Private key for PIV Attestation
        Flags: CKA_PRIVATE; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE;
        ID: 19

Well those look…exciting. (If you want more excitement, use --list-all instead of --list-keys in the command.) You’ll notice the key with ID 02 is an ECDSA key, just like we created. (You may also notice that “ID 02” aligns with how we identified the certificate slot in the pkcs11-tool command.) That’s our key, so copy the value of the “URL” field, then paste it into openssl_root.cnf as the value for the private_key field. The relevant section should look something like this now:

[ CA_default ]
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/certs
database          = $dir/index.txt
serial            = $dir/serial
rand_serial	  = no
RANDFILE          = $dir/private/.rand

# Our private key will reside on a Yubikey.
private_key       = pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20%28www.yubico.com%29;serial=5212376;token=YubiKey%20PIV%20%235212376;id=%02;object=Private%20key%20for%20Digital%20Signature;type=private
certificate       = $dir/certs/ca.example.crt.pem

Generate the CSR and Create a Self-Signed Certificate

With the housekeeping out of the way, create a certificate-signing request (CSR). We will immediately sign it using OpenSSL, creating a self-signed cert that will be the root certificate.

$ OPENSSL_CONF=openssl_root.cnf openssl req \
    -new -x509 -sha384 -extensions v3_ca    \
    -days 3650                              \
    -engine pkcs11 -keyform engine          \
    -key slot_$KEY_SLOT-id_02               \
    -out certs/ca.example.crt.pem
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiKey PIV #5212376:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name [WA]:
Locality Name [Seattle]:
Organization Name [Example Enterprises]:
Organizational Unit Name [Example Enterprises Root CA]:
Common Name []:
Email Address []:
Enter PKCS#11 key PIN for Private key for Digital Signature: ******

I like to have the text representation of the certificate alongside the public key, which we can accomplish like so:

$ openssl x509 -text             \
    -in certs/ca.example.crt.pem \
    -out certs/ca.example.crt.pem.new
$ mv certs/ca.example.crt.pem{.new,}

And now you can see what the certificate “looks” like by peeking in certs/ca.example.crt.pem:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            71:f3:9f:7e:6c:d5:50:b8:60:53:3b:ae:75:90:b5:b4:e5:14:c2:a5
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = US, ST = WA, L = Seattle, O = Example Enterprises, OU = Example Enterprises Root CA
        Validity
            Not Before: Jun 29 20:39:58 2022 GMT
            Not After : Jun 26 20:39:58 2032 GMT
        Subject: C = US, ST = WA, L = Seattle, O = Example Enterprises, OU = Example Enterprises Root CA
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:43:0e:41:cc:5f:3f:3a:36:53:e4:22:08:8b:11:
                    f2:8d:65:65:b3:5b:e0:3c:83:d1:38:f8:b8:80:b0:
                    0b:9c:c5:39:22:47:8b:b0:70:50:01:f2:a1:e3:96:
                    f5:a6:fd:87:d4:c6:08:18:3d:50:fc:29:35:51:71:
                    ec:33:4b:af:98:5f:e3:3a:b7:d9:c6:11:67:1f:2d:
                    ca:cb:b2:9f:e5:1c:54:80:0b:ba:40:f3:9b:41:34:
                    a4:af:87:83:1e:35:f4
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                CE:02:71:A8:84:7A:10:D5:C4:80:E5:23:5A:90:70:67:BC:97:B5:E0
            X509v3 Authority Key Identifier: 
                keyid:CE:02:71:A8:84:7A:10:D5:C4:80:E5:23:5A:90:70:67:BC:97:B5:E0

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: ecdsa-with-SHA384
         30:65:02:30:0a:a6:a4:dd:84:1b:b4:3b:70:a7:82:32:c5:ba:
         f5:e6:69:93:2e:f8:3e:2a:9b:f0:ac:74:75:77:95:6a:42:24:
         27:ec:63:97:1e:8c:79:7f:88:c2:f8:dd:e3:3c:54:43:02:31:
         00:9d:f2:aa:de:37:93:67:81:e9:75:66:40:59:a2:75:5b:7a:
         2d:67:2e:6c:e9:24:99:9d:6d:ca:66:8c:f3:97:80:1a:93:d7:
         42:a9:f4:08:96:e1:5f:44:0f:c9:02:4f:0d
-----BEGIN CERTIFICATE-----
MIIChTCCAgugAwIBAgIUcfOffmzVULhgUzuudZC1tOUUwqUwCgYIKoZIzj0EAwMw
cDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRAwDgYDVQQHDAdTZWF0dGxlMRww
GgYDVQQKDBNFeGFtcGxlIEVudGVycHJpc2VzMSQwIgYDVQQLDBtFeGFtcGxlIEVu
dGVycHJpc2VzIFJvb3QgQ0EwHhcNMjIwNjI5MjAzOTU4WhcNMzIwNjI2MjAzOTU4
WjBwMQswCQYDVQQGEwJVUzELMAkGA1UECAwCV0ExEDAOBgNVBAcMB1NlYXR0bGUx
HDAaBgNVBAoME0V4YW1wbGUgRW50ZXJwcmlzZXMxJDAiBgNVBAsMG0V4YW1wbGUg
RW50ZXJwcmlzZXMgUm9vdCBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABEMOQcxf
Pzo2U+QiCIsR8o1lZbNb4DyD0Tj4uICwC5zFOSJHi7BwUAHyoeOW9ab9h9TGCBg9
UPwpNVFx7DNLr5hf4zq32cYRZx8tysuyn+UcVIALukDzm0E0pK+Hgx419KNmMGQw
HQYDVR0OBBYEFM4CcaiEehDVxIDlI1qQcGe8l7XgMB8GA1UdIwQYMBaAFM4CcaiE
ehDVxIDlI1qQcGe8l7XgMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYDVR0PAQH/BAQD
AgGGMAoGCCqGSM49BAMDA2gAMGUCMAqmpN2EG7Q7cKeCMsW69eZpky74Piqb8Kx0
dXeVakIkJ+xjlx6MeX+Iwvjd4zxUQwIxAJ3yqt43k2eB6XVmQFmidVt6LWcubOkk
mZ1tymaM85eAGpPXQqn0CJbhX0QPyQJPDQ==
-----END CERTIFICATE-----

You’ll want to import the completed certificate to the Yubikey. Do so with the following command:

$ pkcs11-tool --module "$LIBYKCS11"         \
    --login --login-type so                 \
    --pin $PIN --so-pin $MGMT_KEY           \
    --slot $KEY_SLOT --id 2                 \
    --write-object certs/ca.example.crt.pem \
    --type cert
Created certificate:
Certificate Object; type = X.509 cert
  label:      X.509 Certificate for Digital Signature
  subject:    DN: C=US, ST=WA, L=Seattle, O=Example Enterprises, OU=Example Enterprises Root CA
  ID:         02

Now, if you use Yubikey Manager or ykman piv info, you should see the Subject and Issuer DNs (which should be the same), etc. For example:

$ ykman piv info
PIV version: 4.3.3
PIN tries remaining: 3
Management key algorithm: TDES
CHUID:  No data available.
CCC:    No data available.
Slot 9c:
        Algorithm:      ECCP384
        Subject DN:     OU=Example Enterprises Root CA,O=Example Enterprises,L=Seattle,ST=WA,C=US
        Issuer DN:      OU=Example Enterprises Root CA,O=Example Enterprises,L=Seattle,ST=WA,C=US
        Serial:         650548932060042409555401858257111553448854471333
        Fingerprint:    e65811081b499751c6b1a384cfb4d2c3e2b5f0bb07bfeda3155f45c63c362bba
        Not before:     2022-06-29 20:39:58
        Not after:      2032-06-26 20:39:58

You now have a root CA on a Yubikey!

Well, you have a self-signed cert on a Yubikey that we’ve endowed with certain properties. To make it useful as a root CA, we’ll need to create an intermediate, or subordinate CA. Get your next Yubikey ready to go!

Create the Intermediate CA

Just like for the root CA, we need to create a directory structure for the intermediate CA:

$ cd /somewhere/important/ca
$ mkdir -p intermediate/{certs,crl,csr,private}
$ cd intermediate/
$ touch index.txt
$ echo 1000 > serial
$ echo 1000 > crlnumber

We need another OpenSSL config file. This should look similar to the root’s config.

.include ../pkcs11.cnf

[ ca ]
default_ca = CA_default

[ CA_default ]
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/certs
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# Our private key will reside on a Yubikey.
#private_key       =
certificate       = $dir/certs/int.example.crt.pem

crlnumber         = $dir/crlnumber
crl               = $dir/crl/int.example.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 3650

# Set this to match the intermediate CA's key length.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 3650
preserve          = no
policy            = policy_loose

[ policy_loose ]
countryName             = optional
stateOrProvinceName     = optional
localityName     	= optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# This doesn't matter to us; we're using ECC, not RSA.
default_bits        = 4096

distinguished_name  = req_distinguished_name
string_mask         = utf8only
# Set this to match the intermediate CA's key length.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_intermediate_ca

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Update these to suit your environment
countryName_default             = US
stateOrProvinceName_default     = WA
localityName_default            = Seattle
0.organizationName_default      = Example Enterprises
organizationalUnitName_default  = Example Enterprises Intermediate CA
commonName_default 		= Example Enterprises Intermediate Certificate Authority

[ v3_intermediate_ca ]
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always,issuer
basicConstraints        = critical, CA:true, pathlen:0
keyUsage                = critical, digitalSignature, cRLSign, keyCertSign
crlDistributionPoints   = @crl_info
authorityInfoAccess 	= @ocsp_info

[ crl_ext ]
authorityKeyIdentifier = keyid:always

[ ocsp ]
basicConstraints	 = CA:FALSE
subjectKeyIdentifier	 = hash
authorityKeyIdentifier	 = keyid:always,issuer
keyUsage		 = critical, digitalSignature
extendedKeyUsage	 = critical, OCSPSigning

[ crl_info ]
URI.0 = http://crl.example.com/intermediate.crl.pem

[ ocsp_info ]
caIssuers;URI.0 = http://ocsp.example.com/example-root.crt
OCSP;URI.0 = http://ocsp.example.com/

Create the Private Key

Now we can create a private key, just like for the root CA. (Note that I depart from the CA guide a bit here and use a 256-bit key; there is no reason for this other than I don’t have another Yubikey that does P-384 that I want to sacrifice for this example! As you read the rest of the document, keep in mind that NSA Suite B requires signatures to match the signing key’s length, so if you’re using a 384-bit key for the intermediate, make sure you modify any commands accordingly to also use 384-bit signatures.)

$ pkcs11-tool --module "$LIBYKCS11" \
    --login --login-type so         \
    --pin $PIN --so-pin $MGMT_KEY   \
    --slot $KEY_SLOT --id 2         \
    --key-type EC:secp256r1         \
    --keypairgen
Key pair generated:
Private Key Object; EC
  label:      Private key for Digital Signature
  ID:         02
  Usage:      decrypt, sign
  Access:     always authenticate, sensitive, always sensitive, never extractable, local
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   0441042adfd3269920a167450663357b355a3a2e709279b062482a226ad7d4ecd412f22e6060e5ceb7febc20ea646ca3a498908e2082bec5211c883ef1d676fd092f14
  EC_PARAMS:  06082a8648ce3d030107
  label:      Public key for Digital Signature
  ID:         02
  Usage:      encrypt, verify
  Access:     local

We should make note of this key’s URI, just like we had to do for the root key.

$ p11tool --provider=$LIBYKCS11 --list-keys --login
Token 'YubiKey PIV #0' with URL 'pkcs11:model=YubiKey%20NEO;manufacturer=Yubico%20%28www.yubico.com%29;serial=0;token=YubiKey%20PIV%20%230' requires user PIN
Enter PIN:
Object 0:
        URL: pkcs11:model=YubiKey%20NEO;manufacturer=Yubico%20%28www.yubico.com%29;serial=0;token=YubiKey%20PIV%20%230;id=%02;object=Private%20key%20for%20Digital%20Signature;type=private
        Type: Private key (EC/ECDSA)
        Label: Private key for Digital Signature
        Flags: CKA_PRIVATE; CKA_ALWAYS_AUTH; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE;
        ID: 02

Take that URI and add it as the private key value in intermediate/openssl_intermediate.cnf. It should now look like this:

[ CA_default ]
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/certs
database          = $dir/index
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# Our private key will reside on a Yubikey.
private_key       = pkcs11:model=YubiKey%20NEO;manufacturer=Yubico%20%28www.yubico.com%29;serial=0;token=YubiKey%20PIV%20%230;id=%02;object=Private%20key%20for%20Digital%20Signature;type=private
certificate       = $dir/certs/int.example.crt.pem

Generate a CSR

Create a CSR for the root CA to sign. This is slightly different from what we did for the root CA since we’re not trying to self-sign this one.

$ OPENSSL_CONF=openssl_intermediate.cnf openssl req \
    -new -sha384 -engine pkcs11                     \
    -keyform engine -key slot_$KEY_SLOT-id_02       \
    -out csr/int.example.csr
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiKey PIV #5212376:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name [WA]:
Locality Name [Seattle]:
Organization Name [Example Enterprises]:
Organizational Unit Name [Example Enterprises Intermediate CA]:
Common Name [Example Enterprises Intermediate Certificate Authority]:
Email Address []:
Enter PKCS#11 key PIN for Private key for Digital Signature: ******

You can view the certificate signing request using OpenSSL, just like you can a certificate. Examine the CSR in csr/int.example.csr and make sure everything looks okay:

$ openssl req -noout -text -in csr/int.example.csr
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C = US, ST = WA, L = Seattle, O = Example Enterprises, OU = Example Enterprises Intermediate CA, CN = Example Enterprises Intermediate Certificate Authority
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:2a:df:d3:26:99:20:a1:67:45:06:63:35:7b:35:
                    5a:3a:2e:70:92:79:b0:62:48:2a:22:6a:d7:d4:ec:
                    d4:12:f2:2e:60:60:e5:ce:b7:fe:bc:20:ea:64:6c:
                    a3:a4:98:90:8e:20:82:be:c5:21:1c:88:3e:f1:d6:
                    76:fd:09:2f:14
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        Attributes:
            a0:00
    Signature Algorithm: ecdsa-with-SHA384
         30:45:02:20:5e:70:7e:ec:e0:65:5f:a6:3c:f5:d6:6b:89:d1:
         af:e9:d5:61:7e:b1:89:a0:a0:80:f1:25:87:fc:c6:67:41:de:
         02:21:00:e9:fd:8a:cb:d5:df:f9:86:73:78:8d:55:f4:80:c9:
         95:fe:de:8a:b9:59:c6:ad:65:17:e2:70:07:26:73:d1:40

Complete the CSR and Sign the Certificate

We now need to complete (sign) this CSR using the root key. Remove the “intermediate” Yubikey and plug in the “root” Yubikey for the next step.

# do this in the root CA's directory, not the intermediate.
$ cd ..
$ OPENSSL_CONF=openssl_root.cnf openssl ca         \
    -extensions v3_intermediate_ca                 \
    -extfile intermediate/openssl_intermediate.cnf \
    -days 3600                                     \
    -engine pkcs11 -keyform engine                 \
    -in intermediate/csr/int.example.csr           \
    -out intermediate/certs/int.example.crt.pem
Using configuration from openssl_root.cnf
Enter PKCS#11 token PIN for YubiKey PIV #5212376:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number:
            6d:d2:2d:17:5e:b1:43:56:1e:23:f2:82:0a:ab:25:68:59:26:51:dc
        Validity
            Not Before: Jun 30 17:37:23 2022 GMT
            Not After : May  8 17:37:23 2032 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = WA
            organizationName          = Example Enterprises
            organizationalUnitName    = Example Enterprises Intermediate CA
            commonName                = Example Enterprises Intermediate Certificate Authority
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11
            X509v3 Authority Key Identifier:
                keyid:CE:02:71:A8:84:7A:10:D5:C4:80:E5:23:5A:90:70:67:BC:97:B5:E0

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.example.com/intermediate.crl.pem

            Authority Information Access:
                CA Issuers - URI:http://ocsp.example.com/example-root.crt
                OCSP - URI:http://ocsp.example.com/

Certificate is to be certified until May  8 17:37:23 2032 GMT (3600 days)
Sign the certificate? [y/n]:y
Enter PKCS#11 key PIN for Private key for Digital Signature: ******


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

If all looks good, remove the root CA Yubikey from the computer, store it somewhere safe, and only bring it out when you need to sign another intermediate CA’s certificate.

If you wish, you can inspect the new certificate (intermediate/certs/int.example.crt.pem) one more time:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6d:d2:2d:17:5e:b1:43:56:1e:23:f2:82:0a:ab:25:68:59:26:51:dc
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C=US, ST=WA, L=Seattle, O=Example Enterprises, OU=Example Enterprises Root CA
        Validity
            Not Before: Jun 30 17:37:23 2022 GMT
            Not After : May  8 17:37:23 2032 GMT
        Subject: C=US, ST=WA, O=Example Enterprises, OU=Example Enterprises Intermediate CA, CN=Example Enterprises Intermediate Certificate Authority
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:2a:df:d3:26:99:20:a1:67:45:06:63:35:7b:35:
                    5a:3a:2e:70:92:79:b0:62:48:2a:22:6a:d7:d4:ec:
                    d4:12:f2:2e:60:60:e5:ce:b7:fe:bc:20:ea:64:6c:
                    a3:a4:98:90:8e:20:82:be:c5:21:1c:88:3e:f1:d6:
                    76:fd:09:2f:14
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11
            X509v3 Authority Key Identifier: 
                keyid:CE:02:71:A8:84:7A:10:D5:C4:80:E5:23:5A:90:70:67:BC:97:B5:E0

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.example.com/intermediate.crl.pem

            Authority Information Access: 
                CA Issuers - URI:http://ocsp.example.com/example-root.crt
                OCSP - URI:http://ocsp.example.com/

    Signature Algorithm: ecdsa-with-SHA384
         30:65:02:31:00:82:d2:25:f3:f5:df:c8:09:f4:59:cf:d5:52:
         c9:67:5d:d3:d1:73:6d:90:69:4d:c5:43:1a:9b:d9:78:42:30:
         22:1c:39:ce:60:8a:9a:a7:a0:d6:d5:6c:4e:6c:61:51:1d:02:
         30:63:40:dd:c5:03:c5:d3:f9:6e:34:fb:f1:ba:8a:91:f9:69:
         24:56:a3:ee:10:4e:85:cb:4b:43:b0:1c:4d:0b:3a:13:59:86:
         71:42:9e:f8:50:fa:f5:e0:7a:fb:19:78:85
-----BEGIN CERTIFICATE-----
MIIDTjCCAtSgAwIBAgIUbdItF16xQ1YeI/KCCqslaFkmUdwwCgYIKoZIzj0EAwMw
cDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRAwDgYDVQQHDAdTZWF0dGxlMRww
GgYDVQQKDBNFeGFtcGxlIEVudGVycHJpc2VzMSQwIgYDVQQLDBtFeGFtcGxlIEVu
dGVycHJpc2VzIFJvb3QgQ0EwHhcNMjIwNjMwMTczNzIzWhcNMzIwNTA4MTczNzIz
WjCBpzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRwwGgYDVQQKDBNFeGFtcGxl
IEVudGVycHJpc2VzMSwwKgYDVQQLDCNFeGFtcGxlIEVudGVycHJpc2VzIEludGVy
bWVkaWF0ZSBDQTE/MD0GA1UEAww2RXhhbXBsZSBFbnRlcnByaXNlcyBJbnRlcm1l
ZGlhdGUgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAEKt/TJpkgoWdFBmM1ezVaOi5wknmwYkgqImrX1OzUEvIuYGDlzrf+vCDq
ZGyjpJiQjiCCvsUhHIg+8dZ2/QkvFKOCARIwggEOMB0GA1UdDgQWBBSvYO//Bx5L
zQjreOlzwcten8l+ETAfBgNVHSMEGDAWgBTOAnGohHoQ1cSA5SNakHBnvJe14DAS
BgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjA8BgNVHR8ENTAzMDGg
L6AthitodHRwOi8vY3JsLmV4YW1wbGUuY29tL2ludGVybWVkaWF0ZS5jcmwucGVt
MGoGCCsGAQUFBwEBBF4wXDA0BggrBgEFBQcwAoYoaHR0cDovL29jc3AuZXhhbXBs
ZS5jb20vZXhhbXBsZS1yb290LmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Au
ZXhhbXBsZS5jb20vMAoGCCqGSM49BAMDA2gAMGUCMQCC0iXz9d/ICfRZz9VSyWdd
09FzbZBpTcVDGpvZeEIwIhw5zmCKmqeg1tVsTmxhUR0CMGNA3cUDxdP5bjT78bqK
kflpJFaj7hBOhctLQ7AcTQs6E1mGcUKe+FD69eB6+xl4hQ==
-----END CERTIFICATE-----

Now to finish the signing process, plug the intermediate CA Yubikey back into your computer. Even though the intermediate Yubikey has some of the values for the certificate correctly filled out, you should write the new certificate data to the card just like we did with the root card.

$ pkcs11-tool --module "$LIBYKCS11"        \
    --login --login-type so                \
    --pin $PIN --so-pin $MGMT_KEY          \
    --slot $KEY_SLOT --id 2                \
    --write-object                         \
    intermediate/certs/int.example.crt.pem \
    --type cert
Created certificate:
Certificate Object; type = X.509 cert
  label:      X.509 Certificate for Digital Signature
  subject:    DN: C=US, ST=WA, O=Example Enterprises, OU=Example Enterprises Intermediate CA, CN=Example Enterprises Intermediate Certificate Authority
  ID:         02

If you want, check that things look okay using ykman or the Yubikey Manager:

$ ykman piv info
PIV version: 0.1.3
PIN tries remaining: 3
Management key algorithm: TDES
CHUID:	No data available.
CCC: 	No data available.
Slot 9c:
	Algorithm:	ECCP256
	Subject DN:	CN=Example Enterprises Intermediate Certificate Authority,OU=Example Enterprises Intermediate CA,O=Example Enterprises,ST=WA,C=US
	Issuer DN:	OU=Example Enterprises Root CA,O=Example Enterprises,L=Seattle,ST=WA,C=US
	Serial:		626967078516719141285955126592050016717579375068
	Fingerprint:	9d3c1806eed095b4a561e8ee9521650ea1b4ff9b0a2d1439d2da3c14ab1de4c8
	Not before:	2022-06-30 17:37:23
	Not after:	2032-05-08 17:37:23

Create the CRL and OCSP Certificate

Following the CA guide, next we’ll create a CRL and sign it with the intermediate CA:

# Do basically everything from here out in the intermediate directory
$ cd intermediate/
$ OPENSSL_CONF=openssl_intermediate.cnf openssl ca \
    -engine pkcs11 -keyform engine                 \
    -gencrl -out crl/intermediate.crl.pem
engine "pkcs11" set.
Using configuration from openssl_intermediate.cnf
Enter PKCS#11 token PIN for YubiKey PIV #0:
Enter PKCS#11 key PIN for Private key for Digital Signature: ******
Enter PKCS#11 key PIN for Private key for Digital Signature: ******

You can verify the CRL using OpenSSL:

$ openssl crl -in crl/intermediate.crl.pem -noout -text
Certificate Revocation List (CRL):
        Version 2 (0x1)
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = US, ST = WA, O = Example Enterprises, OU = Example Enterprises Intermediate CA, CN = Example Enterprises Intermediate Certificate Authority
        Last Update: Jun 30 18:18:54 2022 GMT
        Next Update: Jun 27 18:18:54 2032 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11

            X509v3 CRL Number:
                4097
No Revoked Certificates.
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:e7:2f:5e:17:a6:1c:9d:d7:c0:b2:48:23:9a:
         2e:32:9e:85:d6:47:b7:25:c6:db:b5:bc:85:4b:c4:67:9e:78:
         69:02:20:5b:bb:29:1e:86:d1:52:25:2c:b6:8e:20:f0:12:e7:
         a3:0c:d4:34:e5:d4:df:c2:8a:ba:38:99:f3:e4:2a:53:6d

Let’s also create an OCSP certificate. For this, we will generate a private key using OpenSSL instead of it being on a Yubikey, since it is going to be like a server certificate.

$ OPENSSL_CONF=openssl_server.cnf openssl req           \
    -new -newkey ec:<(openssl ecparam -name prime256v1) \
    -keyout private/ocsp.example.key.pem                \
    -out csr/ocsp.example.csr.pem                       \
    -extensions server_cert
Generating an EC private key
writing new private key to 'private/ocsp.example.key.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name [WA]:
Locality Name [Seattle]:
Organization Name [Example Enterprises]:
Organizational Unit Name [Information Technology]:
Common Name []:ocsp.example.com
Email Address []:ocsp@example.com

That command created the CSR; now we need to sign it using the intermediate CA key.

$ OPENSSL_CONF=openssl_intermediate.cnf openssl ca \
    -engine pkcs11 -keyform engine                 \
    -extensions ocsp -days 365                     \
    -in csr/ocsp.example.csr.pem                   \
    -out certs/ocsp.example.crt.pem
engine "pkcs11" set.
Using configuration from openssl_intermediate.cnf
Enter PKCS#11 token PIN for YubiKey PIV #0:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Jun 30 18:58:51 2022 GMT
            Not After : Jun 30 18:58:51 2023 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = WA
            localityName              = Seattle
            organizationName          = Example Enterprises
            organizationalUnitName    = Information Technology
            commonName                = ocsp.example.com
            emailAddress              = ocsp@example.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Subject Key Identifier:
                C1:C1:7D:A3:02:DE:B7:B2:F6:85:BE:24:7A:C9:BE:B2:39:E0:F1:A0
            X509v3 Authority Key Identifier:
                keyid:AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11

            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: critical
                OCSP Signing
Certificate is to be certified until Jun 30 18:58:51 2023 GMT (365 days)
Sign the certificate? [y/n]:y
Enter PKCS#11 key PIN for Private key for Digital Signature: ******


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Cool, now you have a certificate you can use to set up an OCSP responder.

Create and Sign a Web Server Certificate

Alright, let’s create some end-entity certificates. Let’s say you have a web server that requires a certificate; your new Yubikey PKI should do nicely!

What follows will be a simplified version of a generic create-request-sign process for getting a signed certificate for a web server. If you’re reading this, most probably you already have a process that does everything up to sending the CSR somewhere to get it signed. It’s at that point where we’ll pick up in more detail.

For this example, we’ll create a generic openssl_server.cnf file to hold defaults that apply to all server certificates. Then, we’ll create a config file for a mythical server named “web01.example.com” that includes the generic server config. (If you’re wondering why we need a config file per entity at all instead of specifying everything on the command line, it’s because we want to assign some subject alternative names to the certificate. Currently the only way to set subject alternative names is via a config file.)

First create intermediate/openssl_server.cnf:

[ req ]
# This doesn't matter to us; we're using ECC, not RSA.
default_bits        = 4096

distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha256

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Update these to suit your environment
countryName_default             = US
stateOrProvinceName_default     = WA
localityName_default            = Seattle
0.organizationName_default      = Example Enterprises
organizationalUnitName_default  = Information Technology
commonName_default 		=

[ server_cert ]
basicConstraints	 = CA:FALSE
nsCertType		 = server
nsComment		 = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier	 = hash
authorityKeyIdentifier	 = keyid,issuer:always
keyUsage		 = critical, digitalSignature, keyEncipherment
extendedKeyUsage	 = serverAuth
crlDistributionPoints	 = @crl_info
authorityInfoAccess 	 = @ocsp_info

[ crl_info ]
URI.0 = http://crl.example.com/intermediate.crl.pem

[ ocsp_info ]
caIssuers;URI.0 = http://ocsp.example.com/example-root.crt
OCSP;URI.0 = http://ocsp.example.com/

…and then create intermediate/openssl_web01.cnf:

.include openssl_server.cnf

[ req_distinguished_name ]
commonName_default = web01.example.com

[ server_cert ]
subjectAltName = @alt_names

[ alt_names ]
DNS.0 = web01.example.com
DNS.1 = web.example.com

Okay, now we can get to creating the certificate itself.

Create the CSR

In the real world, you’d probably generate the private key and CSR on the server and send the CSR elsewhere to get it signed. For this example we’ll pretend that these steps happen in the CA. How this step happens isn’t that important for this article.

$ OPENSSL_CONF=openssl_web01.cnf openssl req            \
    -new -newkey ec:<(openssl ecparam -name prime256v1) \
    -keyout private/web01.example.key.pem               \
    -out csr/web01.example.csr
Generating an EC private key
writing new private key to 'private/web01.example.key.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name [WA]:
Locality Name [Seattle]:
Organization Name [Example Enterprises]:
Organizational Unit Name [Information Technology]:
Common Name [web01.example.com]:
Email Address []:

Sign the Certificate

Maybe you generated the CSR in the previous step, or maybe a CSR magically fell into your lap for you to sign. Let’s do that. Notice that the certificate has the common name and subject alternative names we set in the config:

$ OPENSSL_CONF=openssl_intermediate.cnf openssl ca \
    -engine pkcs11 -keyform engine                 \
    -extfile openssl_web01.cnf                     \
    -extensions server_cert -days 730              \
    -in csr/web01.example.csr                      \
    -out certs/web01.example.crt.pem
engine "pkcs11" set.
Using configuration from openssl_intermediate.cnf
Enter PKCS#11 token PIN for YubiKey PIV #0:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Jun 30 19:20:08 2022 GMT
            Not After : Jun 29 19:20:08 2024 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = WA
            localityName              = Seattle
            organizationName          = Example Enterprises
            organizationalUnitName    = Information Technology
            commonName                = web01.example.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                8F:BE:E6:5B:41:4F:D5:A3:36:09:BE:58:A7:DD:AE:FD:B5:09:31:A9
            X509v3 Authority Key Identifier:
                keyid:AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11
                DirName:/C=US/ST=WA/L=Seattle/O=Example Enterprises/OU=Example Enterprises Root CA
                serial:6D:D2:2D:17:5E:B1:43:56:1E:23:F2:82:0A:AB:25:68:59:26:51:DC

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.example.com/intermediate.crl.pem

            Authority Information Access:
                CA Issuers - URI:http://ocsp.example.com/example-root.crt
                OCSP - URI:http://ocsp.example.com/

            X509v3 Subject Alternative Name:
                DNS:web01.example.com, DNS:web.example.com
Certificate is to be certified until Jun 29 19:20:08 2024 GMT (730 days)
Sign the certificate? [y/n]:y
Enter PKCS#11 key PIN for Private key for Digital Signature: ******


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Review the actual certificate and ensure the common name and subject alternative names are correct:

$ openssl x509 -noout -text -in certs/web01.example.crt.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4097 (0x1001)
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = US, ST = WA, O = Example Enterprises, OU = Example Enterprises Intermediate CA, CN = Example Enterprises Intermediate Certificate Authority
        Validity
            Not Before: Jun 30 19:20:08 2022 GMT
            Not After : Jun 29 19:20:08 2024 GMT
        Subject: C = US, ST = WA, L = Seattle, O = Example Enterprises, OU = Information Technology, CN = web01.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:66:8e:38:8a:e4:0b:c0:69:df:9e:e3:76:91:f3:
                    2f:38:73:67:2d:a5:85:0a:06:d5:83:7c:34:e9:3c:
                    90:a4:3b:2d:a3:5d:ce:c3:e9:9e:b4:5e:84:f8:9a:
                    8b:a1:f4:0a:79:e7:a0:6a:9f:bd:b1:f2:4e:65:a2:
                    1b:03:79:7d:ba
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                8F:BE:E6:5B:41:4F:D5:A3:36:09:BE:58:A7:DD:AE:FD:B5:09:31:A9
            X509v3 Authority Key Identifier:
                keyid:AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11
                DirName:/C=US/ST=WA/L=Seattle/O=Example Enterprises/OU=Example Enterprises Root CA
                serial:6D:D2:2D:17:5E:B1:43:56:1E:23:F2:82:0A:AB:25:68:59:26:51:DC

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.example.com/intermediate.crl.pem

            Authority Information Access:
                CA Issuers - URI:http://ocsp.example.com/example-root.crt
                OCSP - URI:http://ocsp.example.com/

            X509v3 Subject Alternative Name:
                DNS:web01.example.com, DNS:web.example.com
    Signature Algorithm: ecdsa-with-SHA256
         30:46:02:21:00:92:22:ab:ec:2d:c8:ab:89:11:81:ac:7a:33:
         46:63:7d:29:b1:c6:93:37:13:57:ad:ec:78:43:f1:f5:fb:1a:
         30:02:21:00:ba:63:f5:2a:fa:9c:99:51:d1:5a:f8:79:20:c0:
         87:3f:7d:64:89:e6:55:fb:f5:b2:03:f2:17:ab:5d:86:e8:d4

If everything looks good, you can send that certificate back to the requester and let them install it on web01.example.com!

Sign a Yubikey User Certificate

What’s that? You have yet another Yubikey just sitting around, gathering dust? (Or is that just me?) Let’s see what it would look like to go through this process using a physical key, shall we?

You’ve probably figured it out, but we’ll need another config file just for user certs. It’s called intermediate/openssl_user.cnf, and it looks like this:

.include ../pkcs11.cnf

[ req ]
# This doesn't matter to us; we're using ECC, not RSA.
default_bits        = 4096

distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha256

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Update these to suit your environment
countryName_default             = US
stateOrProvinceName_default     = WA
localityName_default            = Seattle
0.organizationName_default      = Example Enterprises
organizationalUnitName_default  = Information Technology
commonName_default 		=

[ usr_cert ]
basicConstraints	 = CA:FALSE
nsCertType		 = client, email
nsComment		 = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier	 = hash
authorityKeyIdentifier	 = keyid,issuer
keyUsage		 = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage	 = clientAuth, emailProtection
crlDistributionPoints	 = @crl_info
authorityInfoAccess 	 = @ocsp_info

[ crl_info ]
URI.0 = http://crl.example.com/intermediate.crl.pem

[ ocsp_info ]
caIssuers;URI.0 = http://ocsp.example.com/example-root.crt
OCSP;URI.0 = http://ocsp.example.com/

Note specifically that, since we’re dealing with a Yubikey, we need to include the pkcs11.cnf file so OpenSSL can figure out how to interact with it.

Generate the Private Key on the Yubikey

As we have done for both CA certs, we’ll need to generate the private key on the end-user’s Yubikey, but this time we’ll use slot 9a ("--id 1") instead of 9c. Make sure this is the only Yubikey plugged in, or else you may risk overwriting the intermediate CA’s private key. Or, you know, just do this on a machine far away from the intermediate Yubikey.

$ pkcs11-tool --module "$LIBYKCS11" \
    --login --login-type so         \
    --pin $PIN --so-pin $MGMT_KEY   \
    --slot $KEY_SLOT --id 1         \
    --key-type EC:secp256r1 --keypairgen
Key pair generated:
Private Key Object; EC
  label:      Private key for PIV Authentication
  ID:         01
  Usage:      decrypt, sign
  Access:     sensitive, always sensitive, never extractable, local
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   0441040296db53c7dca7f6019624da1419923b32603672652e64643d34d23f40d6f4dc946a753d93860707032db7c64002e6300086f44725591707dc06c141bf5a7904
  EC_PARAMS:  06082a8648ce3d030107
  label:      Public key for PIV Authentication
  ID:         01
  Usage:      encrypt, verify
  Access:     local

Generate the CSR

This should look a lot like generating every other CSR we’ve generated; not a lot of new ground here.

$ OPENSSL_CONF=openssl_user.cnf openssl req   \
    -new -engine pkcs11                       \
    -keyform engine -key slot_$KEY_SLOT-id_01 \
    -out csr/user.seth.csr
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiKey PIV #19171898:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name [WA]:
Locality Name [Seattle]:
Organization Name [Example Enterprises]:
Organizational Unit Name [Information Technology]:
Common Name []:Seth
Email Address []:seth@example.com

Walk the CSR over to the computer with the CA, ensure that the intermediate CA Yubikey is plugged in, and proceed to signing the CSR.

$ OPENSSL_CONF=openssl_intermediate.cnf openssl ca \
    -engine pkcs11 -keyform engine                 \
    -extfile openssl_user.cnf                      \
    -extensions usr_cert -days 365                 \
    -in csr/user.seth.csr                          \
    -out certs/user.seth.crt.pem
engine "pkcs11" set.
Using configuration from openssl_intermediate.cnf
Enter PKCS#11 token PIN for YubiKey PIV #0:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4098 (0x1002)
        Validity
            Not Before: Jun 30 19:51:07 2022 GMT
            Not After : Jun 30 19:51:07 2023 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = WA
            localityName              = Seattle
            organizationName          = Example Enterprises
            organizationalUnitName    = Information Technology
            commonName                = Seth
            emailAddress              = seth@example.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME
            Netscape Comment:
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier:
                4A:3F:C1:54:2F:A3:4C:6B:E7:7A:DC:6E:3A:D1:1B:A7:35:AA:A0:C0
            X509v3 Authority Key Identifier:
                keyid:AF:60:EF:FF:07:1E:4B:CD:08:EB:78:E9:73:C1:CB:5E:9F:C9:7E:11

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.example.com/intermediate.crl.pem

            Authority Information Access:
                CA Issuers - URI:http://ocsp.example.com/example-root.crt
                OCSP - URI:http://ocsp.example.com/

Certificate is to be certified until Jun 30 19:51:07 2023 GMT (365 days)
Sign the certificate? [y/n]:y
Enter PKCS#11 key PIN for Private key for Digital Signature: ******


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Take the intermediate/certs/user.seth.crt.pem (or, uh, whatever name you gave it) file over to the computer where the user’s Yubikey is plugged in, and import the signed certificate:

$ pkcs11-tool --module "$LIBYKCS11" \
    --login --login-type so         \
    --pin $PIN --so-pin $MGMT_KEY   \
    --slot $KEY_SLOT --id 1         \
    --write-object                  \
    certs/user.seth.crt.pem         \
    --type cert
Created certificate:
Certificate Object; type = X.509 cert
  label:      X.509 Certificate for PIV Authentication
  subject:    DN: C=US, ST=WA, L=Seattle, O=Example Enterprises, OU=Information Technology, CN=Seth/emailAddress=seth@example.com
  ID:         01

Verify it if you wish, using ykman or the Yubikey Manager app:

$ ykman piv info
PIV version: 5.4.3
WARNING: Using default PIN!
PIN tries remaining: 3/3
WARNING: Using default Management key!
Management key algorithm: TDES
CHUID:	No data available.
CCC: 	No data available.
Slot 9a:
	Algorithm:	ECCP256
	Subject DN:	1.2.840.113549.1.9.1=seth@example.com,CN=Seth,OU=Information Technology,O=Example Enterprises,L=Seattle,ST=WA,C=US
	Issuer DN:	CN=Example Enterprises Intermediate Certificate Authority,OU=Example Enterprises Intermediate CA,O=Example Enterprises,ST=WA,C=US
	Serial:		4098
	Fingerprint:	50ed7684a902c9cebb35ced0693c1b5d347a77f4cf54141fe254354e5fce06c3
	Not before:	2022-06-30 19:51:07
	Not after:	2023-06-30 19:51:07

Final Details

In order for any entity to be able to validate literally any of this, you’ll have to give each client the root and intermediate CAs' public keys (ca/certs/ca.example.crt.pem and ca/intermediate/int.example.crt.pem). I won’t go into detail on how to do this for every platform and browser, but generally you’ll need to either create a PFX certificate bundle or PEM bundle containing both certificates. Generally speaking:

$ cat intermediate/certs/int.example.crt.pem certs/ca.example.crt.pem > cert-bundle.pem

should get you working for macOS’s Keychain.app and *NIX-y things, but not Windows or Firefox. These last two require PKCS#12 PFX files. Create one something like this:

$ openssl pkcs12 -export -nokeys \
    -passout pass:password       \
    -in cert-bundle.pem          \
    -out cert-bundle.pfx

(The password doesn’t really do anything here since the only things in the file are public certificates.)

In addition, any server that uses certificates signed by your new CA must have the intermediate certificate available to send to clients along with its own certificate. It’s up to you to figure out what exactly your software’s needs are. :-)

Conclusion

And there you have it. If you followed the steps in this article, you should now have the following:

One Yubikey–in a safe place–that contains the private key for the root CA.
One Yubikey that contains the private key for the intermediate CA.
A full-blown OpenSSL CA, complete with config files for creating CAs, server certificates, and user certificates
One web server private key and certificate, signed by the intermediate CA, that you can load on some web server somewhere
One Yubikey that contains a user certificate, signed by the intermediate CA

Not bad for *checks notes* 6900 words/33 minutes!

Sources

Install Missing Docs on Alpine Linux

Thu, 09 Jun 2022 15:31:11 -0400

So maybe you’re not using Alpine Linux solely as a Docker container, and you want to have a useful set of documentation installed. First things first: when you install packages, make sure you grab the -doc package as well (duh). So for instance, if you want to install rsync, make sure you run apk add rsync rsync-doc. (Alternately, using shell expansion: apk add rsync{,-doc}.)

But how to install any missing documentation packages for packages that have already been installed? When trying to figure this out for myself, I found this article, and it did a good job of doing what I wanted, but–as mentioned in the article–it can be slow. Even the solution at the end of the article took 55 seconds to enumerate 45 packages. (I think this is due to running apk info on each and every package to see whether it has a -doc package or not, which takes just takes time.)

I worked on it a bit and came up with a quicker solution, which uses a single invocation of apk search with the full list of potential -doc packages:

$ apk list -I |    # get all installed packages...
    # for all non-doc packages, strip off the version and append -doc...
    awk '!/(-doc)/ { print gensub(/^(.*)-[[:digit:]].*/, "\\1-doc", "g", $1); }' |
    # ...get apk to tell us what is real and what isn't...
    xargs apk search |
    # ...strip off the version yet again (apk add doesn't like it)...
    awk '{ print gensub(/^(.*)-[[:digit:]].*/, "\\1", "g", $1); }' |
    # ...and then actually install them.
    xargs -rt sudo apk add -i

Doing it this way took only four seconds (down from 55!) and actually caught eleven more packages that had docs than the other method. Not bad!

Minecraft on Apple Silicon

Sun, 26 Dec 2021 15:29:23 -0500

Can I run Minecraft on my new M1/Apple Silicon Mac?

Sure! Let’s start with the simplest scenario and go get the official Minecraft: Java Edition client for macOS. Open the DMG and drag the Minecraft app to the Applications shortcut:

(Note that this requires you to install Rosetta 2, which is foreshadowing for the rest of this post.)

Log in with your Microsoft or Mojang account. Unless you have already played Minecraft through the official launcher, the launcher should default to giving you a new 1.18.1 instance. Click “Play” and let the launcher download all the assets it needs. Once it launches you’ll be greeted by the usual Minecraft title screen:

Minecraft title screen

Create a new world just to make sure everything works, and hit F3 to take a look at your frame rate (upper left-hand corner). It will probably be pretty low right when you start the world simply because Minecraft is still generating chunks and whatnot, but after a few seconds it should stabilize. On my M1 Max, I saw 21 frames per second as soon as I entered the world, but it quickly ramped up to about 60-70 fps, with dips into the 40s and spikes into the low 100s.

Just after starting the new world:

26fps: not great

…and after a few minutes:

60fps after chunk generation settled

Not bad. Let’s quit the game and see if we can do better.

This did not go according to plan

…Uh, wait a minute. The game crashed! I don’t remember that happening on my Intel-based MacBook Pro. We should probably investigate that.

(Note: maybe don’t immediately start doing all the things I do in this article, as you read it. I wrote it in a stream-of-consciousness style, walking through various reasonable steps a reasonably technical person might do, so you can better understand the why of the solution I present here. When doing my own research to get Minecraft working as well as I could on my M1 Max, I found a number of GitHub gists and blog posts that essentially boiled down to, “just go download these random binaries someone built and run this wrapper script and everything will be alright!” There’s nothing inherently wrong with that, and if you’d rather just get things working and don’t care about seeing me blather on, I encourage you to seek those resources out. But I needed to understand why those steps are necessary and wanted to use binaries I built instead of trusting someone else’s year-old builds, and wanted to pass on that knowledge to others who might find it useful.)

Intel, Arm, and Universal Binaries, Oh My!

If you’ve had your Apple Silicon-based Mac for a bit, you may have come across software that is not optimized for the M1 processor. Instead, the software is built for Intel-based Macs (which have been around since 2006), and this software will not run on an M1-based Mac without Rosetta 2 installed. Rosetta 2 is a layer that interprets the Intel machine instructions and translates them to the native Arm instruction set that the M1 processor can understand. As you can imagine, this translation does not come for free, and running Intel-only apps on an M1 Mac can be much slower than if it was natively built for the M1.

You may have never (or very rarely) had to actively decide whether to download and install an Intel (also called “amd64” or “x86-64”) versus an Apple Silicon (“M1”, “arm64”, or “aarch64”) app. This is because Apple has what they call “Universal Binaries”, which just means that one executable file can contain a version of the code compiled for Intel as well as another version compiled for the M1. This is what most developers target nowadays, so that their app will run natively regardless of whether it runs on an Intel or Apple Silicon processor.

With the technicalities dispensed with, we can easily see whether an app was compiled for the M1 or whether it’s running under Rosetta 2. Open Activity Monitor (check in the Utilities folder under Applications, or use Spotlight to find it) and check the “Kind” column. If it says “Apple”, then that software has been compiled for the M1. If it says “Intel”, though, then you can be sure that macOS is using Rosetta 2 to run the app, and performance may not be on-par with what you might expect from your fancy (and expensive) new machine. Let’s take a look at the java process when Minecraft starts up:

Hm, Intel.

“But wait,” you say, “I don’t remember installing Java!” True: Minecraft bundles its own JRE (“Java Runtime Environment”) so that you don’t have to bring one to the party. It also lets them update it so you don’t have to remember to do so.

So what we’ve discovered here is that the Java runtime that comes with Minecraft is not optimized for the M1 processor, so Rosetta 2 has to translate the entire Minecraft experience from Intel instructions to M1’s “Arm” instructions. That can’t be good for performance. Can we fix that, maybe?

BYO JRE

You can set a different Java executable if you need to

It turns out, maybe we can! The Minecraft launcher lets you bring your own JRE to the party if you really want to. What if we got a native M1 “arm64” version of Java 17 and told Minecraft to use that?

You’ll need to get yourself a Java runtime built for Arm instead of Intel. The easiest way to do this is to head to Adoptium or Azul, both of which offer OpenJDK builds for a number of different platforms. (I used the Azul JVMs for this post.) I suggest getting the JDK (Java Development Kit) over the JRE because it comes with niceties like the Java compiler.

What was that? Why do you need a Java compiler if we’re not compiling anything? *looks at the rest of this post* Uh, no reason…

Go to the Azul link above and click the “.dmg” link. Once downloaded, open the DMG and follow the instructions to install Java 17 where macOS likes them to be (/Library/Java/JavaVirtualMachines/). The next thing you need to do is tell Minecraft to use this instead of its bundled version. In the Minecraft launcher, click the “Installations” tab at the top of the window, then click on your 1.18 instance. In the Edit screen that comes up, choose “More Options”; on the “Java Executable” line, either click Browse to find the java command you want to use or, if you’re following along with me, just right-click and paste /Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home/bin/java into the box.

Java executable has been set to our newly-downloaded JVM

Click Save, and then click Play! Congratulations, you’re done!–uh, wait a minute.

Well this was un^H^Hexpected

(This is the part where I hope you weren’t actually following along with me, because at this point you’re probably not terribly happy with me.)

Well That Didn’t Work

What do we do now? We didn’t even get a crash report like our last crash to figure out what might have happened. Fortunately there are still log files we can look at. The logs are in ~/Library/ApplicationSupport/minecraft, and the summary is that one of the libraries that Minecraft uses for graphics (LWJGL) failed to load. I guess we’ll need to investigate that, now.

At this point, though, we’ve gone about as far as we can (should?) with the official Minecraft launcher. What we need is something that will make it easier to tweak our Minecraft instance’s various Java settings.

Enter MultiMC

From their website, “MultiMC is an alternative launcher for Minecraft. It allows you to have multiple, cleanly separated instances of Minecraft (each with their own mods, resource packs, saves, etc) and helps you manage them and their associated options with a simple and powerful interface.” On the left-hand side of the MultiMC main page, find the “Download & Install” link, then choose to download the “Mac OS X” version (direct link). Double-click the file and it will unpack the MultiMC app. You can drag that into your Applications folder (or wherever else you want to put it). Once you have it installed, open the app and create a new Minecraft instance by clicking the “Add instance” button. Choose “1.18.1” as the version, give it a name if you like and click OK.

Now we can start customizing our new install. First we need to ensure the instance runs with the proper Java version–which, in the case of 1.18, is Java 17. (Other Minecraft versions require different versions of Java, which is beyond the scope of this post.) Since we downloaded one for our previous (failed) attempt using the Minecraft launcher, we’ll use that one.

If you only have a single JVM installed, MultiMC will probably do the right thing and automatically use it. If you have multiple JVMs installed–say, you want to play vanilla Minecraft 1.18 as well as an older 1.16 or 1.12 world–then you’ll need to tell MultiMC which JVM to use for each instance.

Select the instance icon in the middle pane and click “Edit instance” in the menu on the right-hand side. In the window that appears, click Settings on the left.

Select the checkbox next to “Java installation”. If the path that appears isn’t correct, you can click “Auto-detect” and choose the correct one. (Remember: for 1.18 we’re looking for Java 17.)

At this point, we’ve set up MultiMC the same way as our failed official Minecraft launcher experiment…which means that if you started your new instance now, it would most likely result in the same failure we saw earlier. We now have to tackle that LWJGL error.

That LWJGL Error

To make a long story, uh, less long, the LWJGL Java library also includes “native” code (i.e., not Java bytecode) that needs to match the architecture of the JVM that loads it. In other words, if you want to load the LWJGL library into a JVM compiled for Intel Macs, you will not only need the (processor-agnostic) Java components of the library, but also some library files compiled for Intel processors; similarly, if you run the Arm-compiled JVM, you’ll need LWJGL libraries compiled for the M1 (Arm-based) processor.

Can you guess where our crash is coming from? If you guessed “Minecraft bundles Intel-only versions of the LWJGL natives and we’re trying to run an Arm JVM to avoid the Rosetta 2 tax,” you guessed right!

So, where do we get these native libraries? From the official LWJGL website, of course. They provide a neat way to download a customized version of the LWJGL library files. To make it easy, you can load this config and then click “Download”. After a bit, you’ll get a ZIP file of all the resources you requested. Find this file in Finder and double-click it to unzip it into a folder named “lwjgl-release-3.3.0-custom”.

At this point, you’ll need to open Terminal.app (or your favorite terminal app) so you can run some things on the command line. What we’re going to do is take all of the JAR files in that ZIP file we downloaded, extract them all into a single base directory, and repack them into a single JAR file. (Why do this? To make it easier for us, as we only have to specify a single JAR file to add to the Java classpath.)

$ pwd
/Users/seth/Downloads/lwjgl-release-3.3.0-custom
$ mkdir -p extracted
$ cd extracted/
$ for f in ../*.jar; do jar -xf "$f"; done
$ jar cf ../../lwjgl-release-3.3.0.jar .
$ ls -lah ../../lwjgl-release-3.3.0.jar
-rw-r--r--  1 seth  staff    18M Dec 26 19:19 ../../lwjgl-release-3.3.0.jar

At the end of this you will have a file called lwjgl-release-3.3.0.jar in the same directory where the downloaded ZIP file exists. The last step of this process is to move that file into a directory where MultiMC can find it, and then tell MultiMC to use it.

Back in MultiMC, select your 1.18 instance again and click “Edit instance” just as before. This time click the “Version” tab on the left.

See that line for LWJGL 3? We need to edit that. Click the “Customize” button on the right side, then click “Edit”. This should bring up a TextEdit window with some default JSON stuff in it. Select everything, delete it, and then paste the following into the window instead:

{
    "formatVersion": 1,
    "libraries": [
        {
            "name": "lwjgl:lwjgl-release-3.3.0:1",
            "MMC-hint": "local",
            "MMC-filename": "lwjgl-release-3.3.0.jar"
        }
    ],
    "name": "LWJGL 3",
    "releaseTime": "2019-06-24T12:52:52+00:00",
    "type": "release",
    "uid": "org.lwjgl3",
    "version": "3.3.0"
}

That tells MultiMC to use the JAR file we created instead of its defaults. With that done, we now need to move our new JAR file into the right place. Close the TextEdit window and, back in MultiMC, click the “Open libraries” button on the right. It will open a Finder window. Move (or copy and paste) the lwjgl-release-3.3.0.jar we created earlier into this directory. Close the Finder window.

Let’s go ahead and click the “Launch” button in the lower right and see what happens. Cross our fingers, and…

100fps with no other changes!

Whoa, success! And we’ve gained about 40 frames per second after all this for an average of around 100fps. Not bad! Let’s go into our Minecraft video settings and make this window full-screen so we can see it in all its glory–

Uh, wait a minute.

pls to stahp with all the crashing

*ahem*

Okay. So at least we’ve made progress, right? That’s a thing that’s happened.

Break Out Your Compiler

In for a penny, in for a pound, right?

We’ve stumbled on another Java library that requires native code; one that unfortunately is not shipped as a Universal binary, but containing only Intel-specific instructions. This time it’s the Java Objective-C Bridge. Thankfully, this library added support for M1 Macs about a month prior to when I wrote this post (but hasn’t yet pushed a new version to Maven Central, I guess), so we “just” have to compile it on our own.

This is the part where you’ll be glad you listened to me earlier and downloaded a JDK (Java Development Kit) instead of just the JRE.

You’ll also need Apache Maven, a tool used to build Java projects. I’m not going to go into how to download and install it (although it’s really as simple as “download and make sure the bin/ directory is in your $PATH"); you can find those resources elsewhere.

And, uh, not to bury the lede here, but you will also need Xcode–or at least the command-line tools, which I think (think) you can install by running xcode-select --install in your terminal. (Unfortunately I did not realize this was a requirement until after I wrote the whole post and was proofreading it–I already had Xcode installed–or I would have better guidance here.) Since Xcode is such a large download, this is really a bit out of the way just to play Minecraft if you don’t already have it installed.

But if you’re all-in on this little experiment and you have all the tools, you can follow the README and run mvn clean install and watch it go. At the end you’ll have a JAR in the target/ directory:

$ ls -lah target/java-objc-bridge-1.1-SNAPSHOT.jar
-rw-r--r--  1 seth  staff    91K Dec 27 00:20 target/java-objc-bridge-1.1-SNAPSHOT.jar

Just like we did before, we need to copy this JAR into a place that MultiMC can find it. This time, though, we need to create our own “custom package”. Start by navigating to the same “Version” configuration window where we edited the LWJGL config and click “Add Empty”. You can make the name “java-objc-bridge” and the uid “ca.weblite.java-objc-bridge”, then click Edit and paste the following JSON into the editor window:

{
    "formatVersion": 1,
    "libraries": [
        {
            "MMC-filename": "java-objc-bridge-1.1-SNAPSHOT.jar",
            "MMC-hint": "local",
            "name": "java-objc-bridge:java-objc-bridge-1.1-SNAPSHOT:1"
        }
    ],
    "name": "java-objc-bridge",
    "type": "snapshot",
    "uid": "ca.weblite.java-objc-bridge",
    "version": "1.1"
}

Close the window, then click the “Move up” button until our new custom package is at the top of the list. Now click “Open libraries” and copy the java-objc-bridge-1.1-SNAPSHOT.jar file you just built into the libraries folder, right next to the LWJGL one we added earlier.

Phew! Let’s try this again!

This has been a long post coming

FINALLY.

Let’s Wrap This Up

Well that was a lot of fun, wasn’t it? (You can stop laughing now.)

This will get you working for Minecraft 1.18. Other (older) versions of Minecraft use a different Java version and rely on different libraries (or different versions of the same libraries we’ve already seen), and so each may pose their own unique challenges to getting a fully-native Minecraft experience set up. Hopefully Mojang will support the M1 natively in the near future, at which point none of this post will be relevant any longer.

At the time of this writing, a fork of MultiMC (called “ManyMC") exists that supposedly solves all of these problems for you, for many versions of Minecraft and Minecraft Forge. Right now it has a large disclaimer on the top of the README, cautioning that it is an unofficial project, so its state of support may change at any time. I like how that project handles transparently serving corrected natives. One drawback for me is that in my limited poking at it and its various supporting repos, all I found were compiled JARs and not the patches required–if any–to get them working on the M1. I would approach it with the same hesitancy as I do the other “Minecraft on M1” hacks that ask you to download precompiled JARs that come without source code. No, the authors are not required to provide the sources (unless the code’s license requires it), especially if all they’re doing is recompiling the library to get arm64 natives. And I’m sure that many people who just want Minecraft to work well on their new Apple Silicon won’t care about this aspect of it, and that’s perfectly okay. Heck, I’ve been known to pipe curl to sh a few times, myself, so maybe I’m being hypocritical here. I guess in this instance it would just be nice if I could both trust and verify.

So, almost 3,000 words later: do what you want, play Minecraft the way you want, and have some fun!

Maybe in a part 2 we can see what it will take to get a Minecraft Forge-based client up and running. (Spoiler alert: more of the same, really!)

Pocket, Please Let Me Rename Things.

Tue, 29 Dec 2020 15:26:19 -0500

Sometimes I just want to name things the way I deem fit. Pocket, back when they were called “Read It Later”, used to include this functionality. Somewhere along the way, Pocket decided to remove that feature in preference to augmenting their product to do a really good job of figuring out what an article’s title should be. However, there are still many times that it would be helpful to have the ability to manually rename an article. I’ll detail some problematic articles below, with screenshots.

But First, History

I’ve been using Pocket since it was named Read It Later; Wikipedia tells me (in a roundabout fashion) that that means I’ve most likely used the service for over a decade. I’ve used it on Android phones, iOS devices, even Windows phones. I’ve used the actual website. I keep “Recommended by Pocket” as the only thing on my new tab/window page in Firefox.

Wikipedia also tells me that Pocket Premium has been A Thing™ since May 2014; I’ve been a subscriber since June of the same year. Needless to say, I like this product, and I will continue to use it for quite some time, I’m sure.

I can’t find it any longer, but there used to be a support forum for Pocket, with a number of threads asking for rename (or title editing) support. I marked a few as helpful (or upvoted, or whatever you were supposed to do to them to register relevance without “me too”‘ing the post), and even (if I recall correctly) wrote some comments expressing my desire for the feature. The feature requests were denied back then for the same reason the feature request isn’t considered today:

While we don’t currently offer title editing functionality, we encourage you to report any incorrect titles so we can try to get them fixed!

(But I will note that that feature seems to be under consideration now, so perhaps this post is timely!)

Why Allow Title Editing?

So, if Pocket does a really good job of extracting titles from articles most of the time, why do I even need the ability to rename articles myself? Well, as with everything in life, nothing is perfect. Pocket recognizes this, and does give you a way to report items with incorrect titles. While that’s a great option to have, it doesn’t help me in the moment. Another option that the help article suggests is that users use Tags “…as a way to help identify and organize items that might not have a title that helps you identify them easily.” That sounds okay on the surface, but I humbly suggest that the best tag to help me identify an article easily is the title of the article. I use Tags as just that: tags, or categories, to help me “group” articles together. I have tags for computer-related articles, science, programming, health, etc. If I had to guess, this is probably how most people use tags. Using a tag to help identify a badly-named article seems antithetical to what I see as the main use of tags.

But How Bad is it, Really?

Not that bad. Really! As I’ve said, most of the time Pocket gets it right. But I think that there will always be some web pages that Pocket can never name properly:

These are only three examples, but I’m sure you can think of a thousand more, given the size of the Internet.

Yes, But [Citation Needed]

You’re right, so here you go. There’s no way this list will look good, because it’s just a bunch of screenshots and I’m not a designer. Clicking on the screenshots will show you how the article looks in Pocket.

Teaching the Unfortunate Parts
You’re Allowed to Write Slow Rust Code
My evolution writing JSON-REST APIs
~~We Hacked Apple for 3 Months: Here’s What We Found~~

(This one was wrong, but sometime between when I saved the article and when I wrote this post, they corrected it! Sadly, the only way I knew about the update was because the web view showed the title correctly, leading me to refresh the reading list on my iPhone and get the updated title there. I’m not sure how I would have discovered this otherwise, which is a different issue, really; probably dealing with the Permanent Library feature of Pocket Premium…maybe?)
Inside the Hidden World of Legacy IT Systems
FPGA Design for Software Engineers
Those random PDFs I mentioned earlier (mmm, bacon; sadly, that paper is titled, “A Unified Theory of Garbage Collection”, and—probably—has nothing to do with actual bacon.)
Simple Made Easy - Talk Transcript
Punching Holes
WRITING FOR THE INTERNET ACROSS A HUMAN LIFETIME
You Don’t Need Reproducible Builds
C++ Creator Bjarne Stroustrup Weighs in on Distributed Systems, Type Safety and Rust

(This one is interesting, because the article view seems to have figured out the title, but the title shown in the list view is incorrect.)
The Success and Failure of Ninja
Making Python Programs Blazingly Fast
Re: Tom Critchlow’s proposal for a decentralised Goodreads-like system, how about using RSS?
And here’s an example of something that has a well-extracted title, but I still wish I could rename:

Conclusion

So there you have it. This is my plea to Pocket to please consider adding support for editing article titles. The sixteen examples above took me back in my archive until about April, 2020; I could have kept going, but I got bored of taking screenshots. I really love that, 95% of the time, I don’t have to think about (re-)naming an article. At the same time, I think that the list above shows a diverse-enough sampling that we can all agree that attempting to get the titles right on all of them is simply infeasible. And even when a title can be extracted properly, there are still times when the user may want or need to edit the title to make it shorter, simpler, more exact, etc. Finally, using Tags to get around this limitation is, in my opinion, simply using a different feature to get around a limitation in an adjacent feature.

I want to finish this by reiterating that I love the Pocket service, and it has only gotten better over the years. Thanks, Pocket, for keeping my “read it later” list organized for over a decade!

Booting a ZFS Root via UEFI on Debian

Tue, 09 Feb 2016 12:56:39 -0500

I’m playing with ZFS on Linux using Debian jessie (and eventually stretch). I decided I really wanted to use ZFS for everything, including /boot/grub and swap space. In addition, I wanted to boot using UEFI. This is how I did it.

The official ZFS Root on Debian HOWTO is much more cautious and wants you to create an ext4 partition for /boot/grub and a separate partition for swap. Being the idiot^Wdaredevil I am, I decided to flout that advice and go all-in on ZFS. The rest of this article details the steps involved in making this happen. (You really want to read the ZFS Root on Debian guide first, and then come back here to see where I deviated. Also, standard disclaimers apply: I’m doing this on a laptop I don’t care about and won’t use for any production purpose. If following this guide creates a monster that eats your disk and calls your mother names, I take no responsibility. It worked for me, but might not work for you. Phew.)

Get Your Install Environment Ready

This part is going to smell a lot like the official guide. I’ll make a note of where I deviate from it, though.

Download the rEFInd Boot Manager. You’ll use it to reboot into your new Debian install the first time.
Download and boot a Live CD, such as debian-live-8.3.0-amd64-standard.iso. The official guide specifically requires using a Live CD and not just an installer CD, so I used the “standard” non-graphical Live CD. Use whatever Live CD you are comfortable with, as long as it’s Debian 8. (Of course, I’m doing all this with an “iso-hybrid” image that I’ve dd‘ed to a USB stick.)

Once booted, download and install the zfsonlinux package. This simply adds the ZoL package repository to your APT sources.

$ sudo -i
# wget -q http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb
# dpkg -i zfsonlinux_6_all.deb
Selecting previously unselected package zfsonlinux.
(Reading database ... 44337 files and directories currently installed.)
Preparing to unpack zfsonlinux_6_all.deb ...
Unpacking zfsonlinux (6) ...
Setting up zfsonlinux (6) ...

Install the ZFS packages in the Live CD environment. This will compile the SPL and ZFS dynamic kernel modules, which will take some time.

# apt-get update
Hit http://archive.zfsonlinux.org jessie InRelease
Hit http://archive.zfsonlinux.org jessie/main amd64 Packages
Hit http://security.debian.org jessie/updates InRelease
Ign http://archive.zfsonlinux.org jessie/main Translation-en_US
Ign http://archive.zfsonlinux.org jessie/main Translation-en
Hit http://security.debian.org jessie/updates/main Sources
Hit http://security.debian.org jessie/updates/main amd64 Packages
Ign http://http.debian.net jessie InRelease
Hit http://security.debian.org jessie/updates/main Translation-en
Hit http://http.debian.net jessie Release.gpg
Hit http://http.debian.net jessie Release
Hit http://http.debian.net jessie/main Sources
Hit http://http.debian.net jessie/main amd64 Packages
Hit http://http.debian.net jessie/main Translation-en
Reading package lists... Done
# apt-get install linux-image-amd64 debian-zfs
[...]
Loading new spl-0.6.5 DKMS files...
First Installation: checking all kernels...
Building only for 3.16.0-4-amd64
Building initial module for 3.16.0-4-amd64
Done.
[...]
Loading new zfs-0.6.5.2 DKMS files..
First Installation: checking all kernels...
Building only for 3.16.0-4-amd64
Building initial module for 3.16.0-4-amd64
Done.
[...]

Load the kernel module and verify that it loaded properly:

# modprobe zfs
# dmesg | grep ZFS
[ 7446.660850] ZFS: Loaded module v0.6.5.2-2, ZFS pool version 5000, ZFS filesystem version 5

Partition Your Disks

I am performing these steps on a laptop with a single drive; as such my setup will not uncover any issues that may arise from using multiple drives in a mirror or RAIDZ configuration. (Just FYI.)

I want everything on ZFS, including /boot/grub and my swap partition. So I will create one large zpool using the entire disk:
```
# zpool create -o ashift=12 -o altroot=/mnt -m none rpool /dev/disk/by-id/<disk_name>
# zfs set atime=off rpool
# zfs set relatime=on rpool
# zfs set compression=lz4 rpool
```
zpool create will label the disk using GPT and actually create two partitions; partition 1 is the ZFS pool, and partition 9 is for the EFI System Partition (ESP). We’ll use this later. I also go ahead and disable atime, enable relatime, and enable lz4 compression on the root dataset so that these are inherited all the way down. I can always override them in a child dataset if necessary. “Wait, so handing zpool(1m) a whole disk still creates two partitions?”, you ask? Yes. Yes it does. So then why do I care so much about not creating a third or fourth partition for /boot/grub or swap? Principles, my friend. And blatant disregard for rules. Moving on…

Use zfs(1m) to create a zvol for swap space:

# zfs create -o compression=off -V 4G rpool/swap
# mkswap -L swap /dev/zvol/rpool/swap
Setting up swapspace version 1, size = 4194300 KiB
LABEL=swap, UUID=354f7cba-fa33-45bb-97c7-7c6aa6109b22

I’ll make sure to add an entry into /etc/fstab later, after that file gets created.

Create the ROOT filesystem. (This is not the same as /; this will be the parent dataset for all future boot environments.)
```
# zfs create -o mountpoint=none rpool/ROOT
```
Create the initial “boot environment” and mount rpool at /rpool:
```
# zfs create -o mountpoint=/ rpool/ROOT/debian-1
# zfs set mountpoint=/rpool rpool
```
Setting the mountpoint for the rpool filesystem deviates from the ZFS on Debian guide, but seems in-line with what OpenIndiana does by default. I do this now instead of when I created the pool, because otherwise the rpool/ROOT/debian-1 filesystem cannot mount properly.
Set the bootfs property on the pool to the boot environment you just created:
```
# zpool set bootfs=rpool/ROOT/debian-1 rpool
```
Create any other filesystems you want. If the filesystems should be managed as a unit (i.e., all part of the same boot environment), then make sure to create them under rpool/ROOT/debian-1. (While there’s not a beadm(1m)-type command for Linux yet, I still want to set up the system up as if there were. That way if/when beadm gets ported, the system will be ready to handle it.)
```
# zfs create -o mountpoint=/home rpool/home
# zfs create -o mountpoint=/usr rpool/ROOT/debian-1/usr
# zfs create -o mountpoint=/var rpool/ROOT/debian-1/var
# zfs create -o mountpoint=/var/tmp -o setuid=off rpool/ROOT/debian-1/var/tmp
# zfs create -o mountpoint=/tmp -o setuid=off rpool/tmp
```
The ZFS on Debian guide mentions setting exec=off on the two tmp filesystems they create. When I followed those instructions, I got errors during DKMS compilation, I believe, so I left it enabled.
Finally, export the zpool.
```
# zpool export rpool
```

Once all the above steps are done, you should now have the following layout (you might want to check this before exporting the pool):

# zpool get all rpool
NAME   PROPERTY                    VALUE                       SOURCE
rpool  size                        238G                        -
rpool  capacity                    0%                          -
rpool  altroot                     -                           default
rpool  health                      ONLINE                      -
rpool  guid                        14661786247243457179        default
rpool  version                     -                           default
rpool  bootfs                      rpool/ROOT/debian-1         local
rpool  delegation                  on                          default
rpool  autoreplace                 off                         default
rpool  cachefile                   -                           default
rpool  failmode                    wait                        default
rpool  listsnapshots               off                         default
rpool  autoexpand                  off                         default
rpool  dedupditto                  0                           default
rpool  dedupratio                  1.00x                       -
rpool  free                        237G                        -
rpool  allocated                   1.12G                       -
rpool  readonly                    off                         -
rpool  ashift                      12                          local
rpool  comment                     -                           default
rpool  expandsize                  -                           -
rpool  freeing                     0                           default
rpool  fragmentation               0%                          -
rpool  leaked                      0                           default
rpool  feature@async_destroy       enabled                     local
rpool  feature@empty_bpobj         active                      local
rpool  feature@lz4_compress        active                      local
rpool  feature@spacemap_histogram  active                      local
rpool  feature@enabled_txg         active                      local
rpool  feature@hole_birth          active                      local
rpool  feature@extensible_dataset  enabled                     local
rpool  feature@embedded_data       active                      local
rpool  feature@bookmarks           enabled                     local
rpool  feature@filesystem_limits   enabled                     local
rpool  feature@large_blocks        enabled                     local

# zfs list -t all -o name,type,mountpoint,compress,exec,setuid,atime,relatime
NAME                         TYPE        MOUNTPOINT  COMPRESS  EXEC  SETUID  ATIME  RELATIME
rpool                        filesystem  /rpool           lz4    on      on    off        on
rpool/ROOT                   filesystem  none             lz4    on      on    off        on
rpool/ROOT/debian-1          filesystem  /                lz4    on      on    off        on
rpool/ROOT/debian-1/usr      filesystem  /usr             lz4    on      on    off        on
rpool/ROOT/debian-1/var      filesystem  /var             lz4    on      on    off        on
rpool/ROOT/debian-1/var/tmp  filesystem  /var/tmp         lz4   off     off    off        on
rpool/home                   filesystem  /home            lz4    on      on    off        on
rpool/swap                   volume      -                off     -       -      -         -
rpool/tmp                    filesystem  /tmp             lz4   off     off    off        on

Install Debian

Reimport the pool and create the cache file:
```
# zpool import -d /dev/disk/by-id -R /mnt rpool
# mkdir -p /mnt/etc/zfs
# zpool set cachefile=/mnt/etc/zfs/zpool.cache rpool
```
In case you’re wondering about the options passed to zpool import (and don’t have a copy of the zpool(1m) man page), the -d option tells zpool to search the /dev/disk/by-id directory for importable pools, and the -R option tells it to use /mnt as the altroot and set the cachefile property to none. The latter gives us the chance to create the directory for the cache file first, then set the option appropriately.
Since I didn’t create a separate partition for /boot/grub, I can skip some things in the official guide and move directly to using debootstrap(8) to install the base Debian system.
```
# apt-get install debootstrap
# debootstrap --arch=amd64 jessie /mnt http://httpredir.debian.org/debian/
[...]
I: Base system installed successfully.
```

Configure Your New Debian System

Create /etc/{hosts,hostname}. Substitute whatever you’re going to name your machine for “debzfs” in the command below.

# echo debzfs > /mnt/etc/hostname
# sed -i -Ee "s#(127.+)#\1 debzfs#" /mnt/etc/hosts
# cat /mnt/etc/hosts
127.0.0.1	localhost debzfs
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

Create /mnt/etc/fstab. It should be empty (except for a comment, maybe), but a line needs to be added for the swap device. Here I use printf(1) for terseness, but you can use your favorite editor to do this as well.
```
# printf "/dev/zvol/rpool/swap\tnone\t\tswap\tdefaults\t0 0\n" >> /mnt/etc/fstab
# cat /mnt/etc/fstab
/dev/zvol/rpool/swap	none		swap	defaults	0 0
```
If you created a dataset for /tmp not under the boot environment (like I did), you might notice at some point that systemd won’t mount /tmp before things want to use it. If that is the case, you can attempt to try the workaround documented here, or you can do what I did and just set the mountpoint on rpool/tmp to legacy and add a line to fstab, like this:
```
# zfs set mountpoint=legacy rpool/tmp
# printf "rpool/tmp\t\t/tmp\t\tzfs\tdefaults\t0 0\n" >> /mnt/etc/fstab
# cat /mnt/etc/fstab
/dev/zvol/rpool/swap	none		swap	defaults	0 0
rpool/tmp		/tmp		zfs	defaults	0 0
```
I’d much rather figure out how to tell systemd to mount /tmp earlier, but until then this method certainly works.
Do some minor network configuration. The ZFS on Debian page just puts all interface configuration into /mnt/etc/network/interfaces, but here I’ve chosen to create a separate file per-interface in /mnt/etc/network/interfaces.d/. It’s your choice.
```
# ls /mnt/etc/network/interfaces.d/
eth0  lo
# cat /mnt/etc/network/interfaces.d/eth0
auto eth0
iface eth0 inet dhcp
# cat /mnt/etc/network/interfaces.d/lo
auto lo
iface lo inet loopback
```

Bind-mount some useful filesystems into the new system.

# for f in dev dev/pts proc sys; do mount -v --bind {,/mnt}/$f; done
mount: /dev bound on /mnt/dev.
mount: /dev/pts bound on /mnt/dev/pts.
mount: /dev/proc bound on /mnt/proc.
mount: /dev/sys bound on /mnt/sys.

Go ahead and copy the zfsonlinux package you downloaded earlier into /mnt/root so you don’t have to download it again, then chroot(1) into your new system.
```
# cp zfsonlinux_6_all.deb /mnt/root/
# chroot /mnt /bin/bash --login
```

Now that you’re in the new environment, you have to setup ZoL all over again…. (It seems that there is a documentation error in the official guide when running the locale-gen command. Use what I’ve got below instead.)

# apt-get install locales
[...]
# sed -i -Ee 's/# (en_US.UTF+)/\1/' /etc/locale.gen
# locale-gen
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
# apt-get install lsb-release
[...]
# dpkg -i /root/zfsonlinux_6_all.deb
Selecting previous unselected package zfsonlinux.
(Reading database ... 11210 files and directories currently installed.)
Preparing to unpack zfsonlinux_6_all.deb ...
Unpacking zfsonlinux (6) ...
Setting up zfsonlinux (6) ...
# apt-get update
[...]

Okay, hold on. I’m going to stop here and deviate once again from the ZFS on Debian guide.

APT-Pin ZoL Packages

In the realm of “that thing that happened that one time”, the first time I tried this setup I got into a situation after updating to Debian’s testing channel (stretch) where the spl package from Debian updated before the spl and zfs packages from ZoL, which broke ZFS for a while until I could uninstall the Debian version and reinstall the ZoL version. Since then I’ve created an preferences file to pin ZoL packages at a higher priority than the Debian ones. (I’m sure this will never come back to bite me, ever.) If you want to do this, create the file /etc/apt/preferences.d/50-zol-packages with the following content. If you’d rather not mess with the Natural Order of Things, then feel free to skip this part.

# cat <<EOF > /etc/apt/preferences.d/50-zol-packages
> Package: *
> Pin: origin archive.zfsonlinux.org
> Pin-Priority: 900
> EOF
# apt-cache policy spl
spl:
  Installed: 0.6.5-1
  Candidate: 0.6.5-1
  Version table:
 *** 0.6.5-1 0
        900 http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
# apt-cache policy grub2-common
grub2-common:
  Installed: 2.02-beta2.9-ZOL11-7aa9f6
  Candidate: 2.02-beta2.9-ZOL11-7aa9f6
  Version table:
 *** 2.02-beta2.9-ZOL11-7aa9f6 0
        900 http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
     2.02~beta2-22+deb8u1 0
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

(Debian jessie doesn’t have an spl package, but stretch (testing) does, so I’m showing the output for grub2-common as well to show you the difference.)

Install ZoL (Again) and GRUB2 EFI Support

And now we continue on. Installing the debian-zfs metapackage will install the spl, spl-dkms, zfs-dkms, and zfsutils packages. Note that the DKMS modules will get compiled (again) at this time.

Once more I’ve deviated from the ZFS on Debian instructions. Instead of installing grub-pc, I’m installing the grub-efi package because I want to boot via UEFI. Exciting times.

# apt-get install linux-image-amd64 debian-zfs
[...]
# apt-get install grub2-common grub-efi zfs-initramfs
[...]
# apt-get dist-upgrade
[...]

Digression: Get Ready for UEFI Boot

Keep in mind that the Debian live-installer you booted from didn’t boot in (U)EFI mode, so configuring GRUB2 will only halfway work. What we need to do is reboot into UEFI mode so that grub-install can access and set EFI variables properly. However, before we do that we can finish out the installation process and prepare partition 9 to be the ESP. We need to install the dosfstools package to be able to format part9 as VFAT. Note that my disk is /dev/sda; yours might be different. You can also use the /dev/disk/by-id/identifier-part9 path instead if you want to stay consistent with the rest of this process.

# apt-get install dosfstools
# mkdir /boot/efi
# mkfs.vfat /dev/sda9
mkfs.fat 3.0.27 (2014-11-12)
# mount /dev/sda9 /boot/efi
# grub-probe -d /dev/sda9
fat
# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.16.0-4-amd64
Found initrd image: /boot/initrd.img-3.16.0-4-amd64
done
# grub-install -d /usr/lib/grub/x86_64-efi /dev/sda # Note: actually unsuccessful!
Installing for x86_64-efi platform.
efibootmgr: EFI variables are not supported on this system.
efibootmgr: EFI variables are not supported on this system.
Installation finished. No error reported
# find /boot/efi -type f
/boot/efi/EFI/debian/grubx64.efi

We’re going to want to mount the ESP to /boot/efi, so add a line to /etc/fstab for it. (If you want, you can mount it read-only instead, or not automatically mount it, etc.)

# printf "/dev/sda9\t\t/boot/efi\tvfat\tdefaults\t0 1\n" >> /etc/fstab
# cat /etc/fstab
/dev/zvol/rpool/swap	none		swap	defaults	0 0
rpool/tmp		/tmp		zfs	defaults	0 0
/dev/sda9		/boot/efi	vfat	defaults	0 1

At this point, you will not be able to boot your install without using rEFInd. We’ll fix that in a minute. Let’s go ahead and finish out the instructions in the official guide.

# passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
# umount /boot/efi
# exit
logout
# for f in dev/pts dev proc sys boot/efi; do umount /mnt/$f; done
# zfs umount -a
# zpool export rpool

If umount(8) complains about not being able to unmount /sys, find out what other random things got mounted there while you weren’t looking (mount|grep /mnt/sys) and unmount those first, then try to unmount /sys again.

Reboot into rEFInd

Download the rEFInd Boot Manager USB image (if you haven’t already) and dd(1) it to a USB stick. (If you have a second thumb drive, use that so that you still have the Debian Live environment on the first stick in case you need it.) Once you’ve done that, reboot your machine. Now would be a good time to make sure your “BIOS” is set to boot into UEFI mode. After verifying that, insert the rEFInd stick and tell your system to boot to it.

Even though grub-install failed to write to the EFI variables, it still put all of the boot files where they needed to go, so rEFInd should be able to identify your Debian installation. Select it from the boot manager screen (if it’s not already selected) and boot.

Log in as root (because everything worked swimmingly and it actually booted, right? Right? Whew!) and use grub-install to fix the UEFI variables it couldn’t set last time.

# grub-install /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.

Before doing anything else, let’s reboot without rEFInd and make sure that all the UEFI stuff is set properly and that your machine can boot under its own power. If anything goes wrong, go back over this article and make sure that you performed all the steps (and that none of them failed, horribly or otherwise). You should also consult the ZFS Root on Debian documentation as well.

If you rebooted and everything worked, you should be able to log back in as root and start installing whatever other things you want! Before you get too far down the road, though, it’d probably be a good idea to snapshot your boot environment just in case.

# zfs snapshot -r rpool/ROOT/debian-1@2016020401-pristine
# zfs list -rt all -o name,type,used,refer,mountpoint rpool/ROOT
NAME                                             TYPE         USED  REFER  MOUNTPOINT
rpool/ROOT                                       filesystem  1.12G    96K  none
rpool/ROOT/debian-1                              filesystem  1.12G   554M  /
rpool/ROOT/debian-1@2016020401-pristine          snapshot        0   554M  -
rpool/ROOT/debian-1/usr                          filesystem   337M   337M  /usr
rpool/ROOT/debian-1/usr@2016020401-pristine      snapshot        0   337M  -
rpool/ROOT/debian-1/var                          filesystem   252M   252M  /var
rpool/ROOT/debian-1/var@2016020401-pristine      snapshot        0   252M  -
rpool/ROOT/debian-1/var/tmp                      filesystem    96K    96K  /var/tmp
rpool/ROOT/debian-1/var/tmp@2016020401-pristine  snapshot        0    96K  -

Conclusion

And this is where I leave you, with Debian installed on a ZFS root and booting from UEFI. Where you go from here is up to you. Install a desktop environment or two, all the utilities you want, create yourself a regular user, and enjoy the benefits of using ZFS.

Crashplan on SmartOS

Fri, 04 Dec 2015 13:03:25 -0500

It’s taken quite a while, but I have replaced my home server’s OS with SmartOS. (It was previously running Windows with Hyper-V. Look for a future blog post about the migration in general.) With the previous layout, I had a single Linux guest called “nas” that was my file server, but did triple-duty running Plex and the CrashPlan client as well. After migrating most of my other Hyper-V guests to OS zones where possible, it was time to tackle getting a CrashPlan zone going. Unfortunately, Code42 retired Solaris as a supported platform a while back. However, since it’s “just Java”, I figured the Linux package might work in an LX-brand zone. Turns out, it works quite well–with one minor caveat.

(I won’t rehash what an LX zone is; instead I will point you towards Bryan Cantrill’s excellent talk, The Dream is Alive! Running Linux containers on an illumos kernel.)

Why?

I use CrashPlan as a local backup target for various clients on my home network. Some of these clients (like my desktop and laptop) backup to both this internal host and to the CrashPlan service, and some only backup to the internal host, depending on how important I deem their data. For months I did not have this local backup target available, and it was time to change that.

Zone Creation

So, first I needed an LX zone. Using imgadm avail I found a few different images that I could potentially use. I decided to go for an Ubuntu zone:

root@vm ~ $ imgadm avail | grep lx-ubuntu
818cc79e-ceb3-11e4-99ee-7bc8c674e754  lx-ubuntu-14.04         20150320    other    2015-03-20T03:45:09Z
430da066-e3a7-11e4-9657-332a2dbdf565  lx-ubuntu-14.04         20150415    other    2015-04-15T19:40:25Z
a21a64a0-0809-11e5-a64f-ff80e8e8086f  lx-ubuntu-14.04         20150601    other    2015-06-01T02:55:17Z

I simply chose the most recently-released lx-ubuntu image I could find at the time and constructed the following zone configuration file:

{
  "alias": "crashplan",
  "hostname": "crashplan",
  "brand": "lx",
  "kernel_version": "3.13.0",
  "quota": 10,
  "max_physical_memory": 2048,
  "image_uuid": "a21a64a0-0809-11e5-a64f-ff80e8e8086f",
  "resolvers": ["192.168.2.1"],
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.2.20",
      "netmask": "255.255.255.0",
      "gateway": "192.168.2.1",
      "primary": true
    }
  ],
  "filesystems": [
    {
      "type": "lofs",
      "source": "zones/crashplan",
      "target": "/opt/crashplan"
    }
  ]
}

Note the filesystems section. Here I am telling vmadm(1M) that I want to mount the zones/crashplan dataset into the zone at /opt/crashplan. This will become Crashplan’s default backup archive location, where all backup data from other computers will be stored. Before creating the zone, I had to create a new ZFS filesystem called “zones/crashplan”, like so:

root@vm ~ $ zfs create zones/crashplan
root@vm ~ $ zfs set compression=off dedup=off zones/crashplan
root@vm ~ $ zfs get compression,dedup zones/crashplan
NAME    	PROPERTY	VALUE		SOURCE
zones/crashplan	compression	off		local
zones/crashplan	dedup		off		local

root@vm ~ $ zfs list zones/crashplan
NAME		USED	AVAIL	REFER	MOUNTPOINT
zones/crashplan	19K	855G	19K	/zones/crashplan

Here, I’ve created the ZFS dataset and turned off compression and deduplication. Since CrashPlan does its own compression (I think) and deduplication (for sure), I didn’t see a need to make ZFS try to do the same things over again.

To make sure I have all syntax right (because that’s rarely a thing on the first shot), I typically check the JSON file with vmadm validate and then create the zone if it is valid:

root@vm /opt/vmdefs $ vmadm validate create -f crashplan_lx.json 
VALID 'create' payload for lx brand VMs.
root@vm /opt/vmdefs $ vmadm create -f crashplan_lx.json 
Successfully created VM f5b988e9-260b-4103-8bab-441c8e0df0fe
root@vm /opt/vmdefs $ vmadm list | grep crashplan
f5b988e9-260b-4103-8bab-441c8e0df0fe  LX    2048     running           crashplan

CrashPlan Installation

At this point, I could log in to the zone and treat it just like a normal Ubuntu host, meaning that everything from here on is the same as when I installed CrashPlan on the previous, real Linux guest. After updating the zone, I downloaded the Linux CrashPlan client and copied it over, then installed the app as per the documentation (starting at “Extract the TGZ…"). Since this install was going to be headless, I then switched over to the headless usage guide and followed the instructions there.

root@crashplan:~# tar xzf CrashPlan_4.4.1_Linux.tgz
root@crashplan:~# cd crashplan-install/
root@crashplan:~/crashplan-install# ./install.sh 
[...]
root@crashplan:~/crashplan-install# service crashplan status
CrashPlan Engine (pid 75632) is running.

I used my laptop’s CrashPlan client to configure the host (as per the headless guide). Once I was done configuring and had restored “normal operation”, I enabled my new host as a backup target on my laptop and watched it merrily start backing up!

Caveat

I was so happy about having CrashPlan working under an LX zone that I tweeted about it…and as sometimes happens, I was informed of a potential issue:

@Crosse3 @crashplan @opensolaris however note this bug might affect you: https://t.co/KiTdM1ssRf
— Chris Ridd (@chrisridd) November 19, 2015

The bug relates to this troubleshooting guide and seems to be specific to using the CrashPlan client to watch for changes on the local filesystem and sync them in real-time to a remote backup target. So far I have not been bitten by this particular issue and, since I am not backing up my local backup server to another backup server (wat) I don’t think I will run into it any time soon. But if this is something you want to do, you might want to watch that Github issue.

Conclusion

Since standing up the zone I’ve pointed my desktop and laptop at it, and they’ve been backing up perfectly fine ever since.

root@vm ~ $ zfs list zones/crashplan
NAME		USED	AVAIL	REFER	MOUNTPOINT
zones/crashplan	230G	763G	230G	/zones/crashplan

And here’s one of my favorite parts: instead of wasting an entire 2+GB of memory had I built this as a KVM guest, the zone is using less than a quarter of that!

root@vm ~ $ zonememstat -az f5b988e9-260b-4103-8bab-441c8e0df0fe
                                 ZONE         ALIAS  RSS(MB)  CAP(MB)    NOVER  POUT(MB)
 f5b988e9-260b-4103-8bab-441c8e0df0fe     crashplan      352     2048        0         0

I’ve been very happy with SmartOS in general and the flexibility of OS vs. LX vs. KVM zones in particular. If you’re looking for a nice hypervisor-type OS, SmartOS deserves some serious consideration.

Blog Cleanup

Sun, 15 Nov 2015 09:26:19 -0500

Anything older than 2014 got moved to an “archive” section. You probably don’t want to go there, but if you do, the link is on the left. I also migrated the site to Hugo. Had I done that recently, it might have made a good blog post. Alas, it is not to be.

Blog moved to Google App Engine

Sat, 31 May 2014 22:38:20 -0400

I’ve been playing with Google App Engine for a project lately, and I decided to give it a shot for hosting this blog. Since Octopress is just a bunch of static files, there isn’t much fiddling to do at all to make it work. In fact, simply copying the app.yaml file from this Github repository was all that was needed.

Neat!