docs/tls_docs/ssh_ca/ssh_certs.md

# Configuring and Using SSH certificates

While not a silver bullet, SSH certificates provide a better mechanism for SSH authentication than the normal SSH key public key authentication. While management is not as robust as a full PKI solution, SSH certificates provide a solution to the TOFU (Trust On First Use) and identity management for both users and machines easier using a lighterweight (albeit incomplete)

## Server side configuration

### Create the directory structure and the ssh ca for hosts and users

It's best practice to create a separate CA for users and systems (hosts). The following commands will create the directories and the key pairs for both the host and user SSH CAs

```
mkdir {host_ca,user_ca}
ssh-keygen -t ed25519 -b 4096 -f host_ca/host_ca -C host_ca
ssh-keygen -t ed25519 -b 4096 -f user_ca/user_ca -C user_ca
```

The data proceeding the ***-C*** is just a metadata (a comment) to identify the files create. These are not used by SSH so name them clearly to identify what they are. Change the paths/file names as necessary.

### Generate host keys and set the permissions for private key

```
ssh-keygen -f hosts/host -N "" -b 4096 -t ed25519 -C <name-of-host>
chmod 400 hosts/host
```

Where ***host*** is the name of the host key being created. Like creating normal SSH keys, it'll create a ***private*** and ***public*** key

```
host
host.pub
```

### Sign host keys

The following command will sign the public key and create the certificate:

`ssh-keygen -s host_ca -I <fqdn> -h -n <fqdn> -V +<#-of-weeks>w hosts/host_key.pub`

***NOTE: When passing the -n flag, it is important to list ALL names that the host will be identified by using a comma-separated list. Usually the FQDN and IP address is sufficient but the names/IPs much be present in order for SSH to properly identify the host. It is also perfectly acceptable to not set the length of time the certificate is valid using the -V flag here if there is no requirement to do so.***

***NOTE: When Linux hosts are created, ssh host key pairs are generated as well by default. Those public keys (usually they are named with the algorthim with the file type of .pub) can be used instead for signing instead of generating a new key pair for a host.***

### Copy key files to host (along with the user_signing_ca public key) and edit sshd_config

The following 3 files should now be present in the ***hosts*** directory:

```
host
host.pub
host-cert.pub
```

The other file that needs to exist on the server host is the ***public key*** for the ***user_ca*** (***user_ca.pub***) in order for the host to know what user identities to trust. This will be sent along with the host keys and certificates.

Copy keys and certificate to the ***/etc/ssh/*** (This is the Linux default. Other OS's may use different directories so refer to the OS documentation and copy them to the appropriate directory) and ensure that permissions for the private key are set to ***user read*** using `chmod 400 /etc/ssh/host` (the restart of the ssh server will fail if the system deems the permissions are too open), then append the following lines to the ***/etc/ssh/sshd_config***:

```
HostKey /etc/ssh/host
HostCertificate /etc/ssh/host-cert.pub
TrustedUserCAKeys /etc/ssh/user_ca.pub
```

Where ***HostKey*** is the ***host private key***, ***HostCertificate*** is the ***host-cert.pub***, and the ***TrustedUserCAKeys*** is the ***user_ca.pub*** key.

Restart the ssh server

```
systemctl restart sshd
```

## Client side configuration

### Create user keys

***NOTE: Just like host keys, existing user keys can be used as well so it might be preferrable to have the user run the following command on their host and have them provide the public key for signing. The process for user key sigining is the same as it is for host keys.***

```
ssh-keygen -f username -b 4096 -t ed25519 -C <username-or-email-for-identification>
```

### Sign the user key using the user_signing_ca certificate

Use the following command below to sign the certificate, where ***-s*** is the private key of the user CA and the ***-n*** is the principals (or usernames) associated with the certificate. Any and all principals ***NEED*** to be listed here (comma-separated) that the user will attempt to access a host as using SSH.

```
ssh-keygen -s user_ca -I <username-info> -n <username> -V +<#-of-weeks>w <user-name>.pub
```

### Copy the user key files to the user client machine

Once the public key is signed, 3 files will be available (or just the ***username-cert.pub*** if the public key was provided by the user) where ***username*** is the name of the key and certificate files:

```
username
username.pub
username-cert.pub
```

Copy these to the client's ***~/.ssh*** directory. Then create or edit the ***~/.ssh/config*** as follows:

```
Host *.domain.com 192.168.1.* 192.168.2.* 192.168.3.*
    IdentityFile ~/.ssh/username
    IdentitiesOnly yes
```

Edit the ***Host*** line to reflect the domain, hosts, subnets, etc desired. The ***IdentityFile*** line should point to the ***private key*** associated with the cert.pub. The ***IdentitiesOnly*** set to ***yes*** ensures the certificate identities are the only identities used when connecting via ssh. This is necessary when the ssh-agent has multiple identities (this ***WILL*** cause confusion if you have more than 1 ssh key pair in the ***~/.ssh*** directory). A list of users can also be supplied here as well. The config can have any number of sections like the above dedicated to a set of hosts or sections for individual hosts.

### Edit the .ssh/known_hosts file

Edit the ***~/.ssh/known_hosts*** file and append/prepend the following line:

```
@cert-authority *.example.com,192.168.1.1,192.168.1.2 ssh-ed25519 <AAAAC.....> host_ca
```

The ***@cert-authority*** tells the SSH agent what certificate authority to use. Following that, the comma-separated list containing ___*.example.com,192.168.1.1,192.168.1.2___ is the domain (here a wildcard) and a list of hosts to be identified by IP. The final part is the ***host ca public key***. This is ***NOT*** same public key used in the ***TrustedUserCAKeys*** directive in the ***sshd_config***, but the public key of the ***host ca*** that was used to sign those host keys. This tells the client the which hosts are trusted to connect to. Unlike SSH using just private and public keys, there will be no need to add the private key to the ***ssh-agent*** as long as the ***~/.ssh/config*** and the certificates exist.

Any entries in the ***~/.ssh/known_hosts*** associated with any configured hosts can be removed. Once the environment is configured, there should be no reason to have static host entries here. Any custom configuration should be in the client's  ***~/.ssh/config*** file or host aliases as listed in the ***~/.ssh/known_hosts*** file.

Once the connection is tested as successful. The ***~/.ssh/authorized_keys*** file can be deleted from the server host. Using SSH certificates will no longer require this file, as the mechanism for handling authorized identities is noted in the ***TrustedUserCAKeys*** line in the ***/etc/sshd_config***.

***NOTE: The best way to test this is to remain logged into an established SSH session on the server (or have console access) and copy the authorized_keys file from the ~/.ssh directory to another directory as a backup (usually just copying it to ~/ will suffice), then deleting the key. Another ssh session can be attempted from another terminal window/tab and if that's successful then the backup file can be removed. It isn't sufficient enough to remove the line from the file, as trying to ssh into the server with a blank file will cause a publickey denied error on the client.***

## SSH Certificate Script

The provided script will automate *some* of the tedious portions of the process. Edit to best fit the environment workflow. Take care to read the ***NOTE*** proceeding, as there are some inherent security risk that will need to be taken into account.

```
#!/bin/bash
HOST_CA=/path/to/ssh_ca/host_ca
USER_CA=/path/to/ssh_ca/user_ca
HOST_CA_KEY=host_ca
USER_CA_KEY=user_ca
DOMAIN="example.com"


echo "SSH Certificate Authority Manager"
echo "---------------------------------"
echo "1) Generate Host Certificate"
echo "2) Generate User Certificate"
read -p "Selection (1 or 2): " CHOICE

case $CHOICE in
    1)
        read -p "Name of host: " NAME
        echo "Enter space-separated list of additional host principals:"
        read -r -a input_array
        PRINCIPALS=$(IFS=,; echo "${input_array[*]}")

        ssh-keygen -f "${HOST_CA}/hosts/${NAME}" -N "" -b 4096 -t ed25519 -C "$NAME"
        chmod 400 "${HOST_CA}/hosts/${NAME}"

        ssh-keygen -s "${HOST_CA}/${HOST_CA_KEY}" -I "${NAME}" -h \
        -n "${NAME},${NAME}.${DOMAIN}${PRINCIPALS:+,${PRINCIPALS}}" \
        "${HOST_CA}/hosts/${NAME}.pub"

        clear
        
        cat << EOF
****************************************************************************
************************* Copy to Remote Host ******************************
****************************************************************************
Host Certificate for ${NAME}
-------------------------------------
$(ssh-keygen -L -f "${HOST_CA}/hosts/${NAME}-cert.pub")
------------------------------------------

${NAME} Private Key
$(cat ${HOST_CA}/hosts/${NAME})
chmod 400 /etc/ssh/${NAME}
--

${NAME} Public Key and Certificate
----------------------------------
echo "$(cat "${HOST_CA}/hosts/${NAME}.pub")" > /etc/ssh/${NAME}.pub
echo "$(cat "${HOST_CA}/hosts/${NAME}-cert.pub")" > /etc/ssh/${NAME}-cert.pub
--

Trusted User CA
---------------
echo "$(cat "${USER_CA}/${USER_CA_KEY}.pub")" > /etc/ssh/${USER_CA_KEY}.pub
--

Edit ${NAME} sshd_config
------------------------
echo "HostKey /etc/ssh/${NAME}" >> /etc/ssh/sshd_config
echo "HostCertificate /etc/ssh/${NAME}-cert.pub" >> /etc/ssh/sshd_config
echo "TrustedUserCAKeys /etc/ssh/${USER_CA_KEY}.pub" >> /etc/ssh/sshd_config
--
systemctl restart sshd && systemctl status sshd
--
Make sure to remove the ~/.ssh/authorized_keys file
****************************************************************************
EOF
        ;;

    2)
        read -p "Name of user: " NAME
        echo "Enter space-separated list of additional user principals:"
        read -r -a input_array
        PRINCIPALS=$(IFS=,; echo "${input_array[*]}")

        ssh-keygen -f "${USER_CA}/users/${NAME}" -N "" -b 4096 -t ed25519 -C "$NAME"
        chmod 400 "${USER_CA}/users/${NAME}"

        ssh-keygen -s "${USER_CA}/${USER_CA_KEY}" -I "${NAME}" \
        -n "${NAME},${NAME}.${DOMAIN}${PRINCIPALS:+,${PRINCIPALS}}" \
        "${USER_CA}/users/${NAME}.pub"

        clear

        cat << EOF
****************************************************************************
************************* Copy to client ***********************************
****************************************************************************
"${NAME} Client Certificate:"
$(ssh-keygen -L -f "${USER_CA}/users/${NAME}-cert.pub")
--------------------------

${NAME} Private key
-------------------
cat ${USER_CA}/users/${NAME}
chmod 400 ~/.ssh/${NAME}
--

${NAME} Public Key and Certificate
----------------------------------
echo "$(cat "${USER_CA}/users/${NAME}.pub")" > ~/.ssh/${NAME}.pub
echo "$(cat "${USER_CA}/users/${NAME}-cert.pub")" > ~/.ssh/${NAME}-cert.pub
--

Add Trusted host CA to ~/.ssh/known_hosts (edit domain wildcard and hostnames/IPs)
----------------------------------------------------------------------------------
echo "@cert-authority * $(cat "${HOST_CA}/${HOST_CA_KEY}.pub")" >> ~/.ssh/known_hosts
--

~/.ssh/config template
----------------------
Host *.domain.com 192.168.1.* 192.168.2.* 192.168.3.*
    IdentityFile ~/.ssh/username
    IdentitiesOnly yes
****************************************************************************
EOF
        ;;

    *)
        echo "Invalid selection. Exiting."
        exit 1
        ;;
esac
```

***NOTE: While this script can be used "as is" (save for some variable editing), the echo commands seen here output the private keys for the user and host to the console, which is an inherent security risk. The command is for ease of getting the key to the remote host or user via simple copy/paste, however in production these keys should be transported to the host or user in much more secure manor, such as rsync or scp or some other secure workflow. As stated above, the system generated host public key from the /etc/ssh directory can be used signing and then the signed certificate can be added to the host without exposing the private key. This does create a management issue in the case of redeployment of a new host system a new certificate will need signed using the new system generated keys. It may be perferable to generate the certificates on the SSH CA and deploy the certificates via secure transport in the case of system redeployment. This way there is no need to re-sign a new certificate. User client certificates are less problematic, since best practice should be to have users generate the key pair and provide to the CA for signing.***