OpenSSH with X509 certificates HOW TO

In this post I will explain how to test a connection with OpenSSH using PKIXSSH fork from Roumen Petrov.

The full process followed to test a SSH connection between a client and a server machine using X509 certificates will be detailed. To make the test we will use a third machine, that we will call control machine, machine that will act as a "Certification Authority", which is the entity that will validate the authenticity of the certificates presented by the user who wants to make a connection and by the destination server.

We will use a custom compiled version of PKIXSSH, as our client demands. So the directories mentioned here will not be the standard.

To understand how it works I have read the following documents:

In a quick summary, and if I have correctly understood, this is how it works

  • X509 is a standard to sign public keys. Signed public keys are considered valid if the Certification Authority is known.
  • We can sign public keys for hosts and users
  • With X509 certificates we can sign in a OpenSSH server without using passwords and without using the traditional OpenSSH private-public key authentication. This means that no public keys must be distributed.

We are going to make two tests

  • Test the connection for an user from the client machine to the server using a X509 certificate
  • In a second step add authentication for the server host

The next steps must be followed

  • In the control machine:

    • Creation of CA Certificate
    • Deploy of CA Certificate in certificate signers directory of OpenSSH server and client machines
  • In the client machine:

    • Creation of keys and certificate for the user in the client machine
  • In the server machine:

    • Configuration of the server to accept X509 certificates for the user
    • Creation of a X09 certificate for the host
  • Again in the client machine:

    • Configuration of the client to accept X509 certificates from the server

So a bidirectional authentication will be made: the user is going to be verified by the server, and the server host is going to be verified by the client.


Creation of CA Certificate

In the control server we run the following commands:

  • The first one creates a key for the CA
  • Then we create Certificate Signature Request for this key
  • And then we create a self-signed certificate, valid for 10 years, for this key
openssl genrsa -des3 -out ca.key 2048
openssl req -new -key ca.key -out ca.csr
openssl x509 -req -days 3650 -in ca.csr -signkey ca.key -out ca.crt

Some info is requested. The important is the "Common Name". Set as the server's hostname. Notice also the option -days 3650 that set the expire time of this certificate to be in 10 years.

The results files are:

  • ca.key: private key for this "fake" certification authority
  • ca.crt: certificate we will use to sign user and host certificates

Deploy of CA Certificate in client and server machines

Then we send the CA certificate to the OpenSSH on server and client machines, under the path signaled in CACertificatePath directive of OpenSSH configuration file sshd_config.

In the control machine:

scp ca.crt server:/opt/openssh/data/ca/crt/
scp ca.crt client:/opt/openssh/data/ca/crt/

We should also create a link with the form [HASH].[NUMBER]. The hash can be obtained with the command:

openssl x509 -in ca.crt -noout -hash

Then, in the server and client machines, we add the link with:

cd /opt/openssh/data/ca/crt/
ln -s ca.crt `openssl x509 -in ca.crt -noout -hash`.0

So, this CA will be recognized as a valid authority and the certificates signed by it seen as valid.

It is needed in both sides, server and client, as the user certificate will be verified by the server, an the server host will be verified by the client before opening a SSH session. The certificate must be also readable by every user.

If the CA certificate is not available the following warning will appear (in verbose mode)

cannot build certificate chain, code=20, msg='unable to get local issuer certificate'

Creation of keys and certificate for the user in the client machine

First, in the client machine:

  • create the private key for the user
ssh-keygen -t rsa -b 2048 -f .ssh/id_rsa -N ""
  • generate a signing request and send it to the control server to be signed
openssl req -new -key .ssh/id_rsa -out id_rsa.csr
scp id_rsa.csr control:/tmp

Now, in the control server, where the CA files are stored:

  • create a matching signed certificate for the user's private key
cd /tmp
openssl x509 -req -days 3650 -in id_rsa.csr -out id_rsa.crt -CA ca.crt -CAkey ca.key -CAcreateserial

The result file, id_rsa.crt is what we want

Again, in the client,

  • add the generated certificate to the client SSH private key and create also the public key
scp control:/tmp/id_rsa.crt .
cat id_rsa.crt >> .ssh/id_rsa
ssh-keygen -y -f .ssh/id_rsa > .ssh/id_rsa.pub

Comparing standard OpenSSH keys with X509 certificates keys

Here I show the keys created for the example user to show the differences between OpenSSH standard private/public key files and those created with X509 certificates

  • With X509 certificates the corresponding certificate for the private key is added to to private key file
  • With X509 there is no public key. The public key file is the same certificate and, as we will see, there is no need of this part to make the authentication work

Example standard RSA OpenSSH private key

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA9IhylUyhWTE5bsR9R8xAe4/b0PBnYiCikLXkQgBAxfWCaW3h
12/FY4YflHv7EHjG7Zfo6ryMvuEEW8lNqXQVMRcP56U4GUfUU3aqZG7FFlFbKIU0
n9y2JHWaeef/hlOgQaJ5goX4yg38C9S0XT1YFSGGei/ifnz57DSfUgSv02Az7LFs
3oQene/9Cp9+umOAOFJIzqIiwXSN7weqM4BJ7qvqRh9ib54yIuMO+D/VTI2LyZJy
h4dZJzyM/O8ovOIw5b5dVnIPQlEG+u/1IBsJcDEkc9zSpcHYu47/mL9JoVPj0DPt
MsnDimk7PEP1fomUYbEipUC3CH41CmOvTR2LKwIDAQABAoIBAGTAG0W8TvYqu9kR
h/SfCSpfuVe61T2u6IvrBWLUK9vkLZk2tshGuvMofbZ15pmAyaV243ZjvTGgEGWo
6SCaPWj+cYUlo5l26NqCl+3NXdK2LnLhfy8mhr063yy/E57KscqZIKtQe5L0dBaD
ytRnw/Tg3UFWnWE0KNzTSZlHaRITUOTdZf/bBNL3rOZrK+5be60ueLJt6gMery1d
6kRdpef3Qh0nOXVChB9cWH8aE5unk6vEi9khC4p5DtpOesAlQt9jlT6GFdZJhkZ9
dvZ+1i3aNa6B4fGWe53F0ecoIAU+FsWnan+bCOxql80+3a7WEVoslozihZ+PNC1Q
bPADioECgYEA+kStLNV4tDkTlQJdeQz8kiLeI2j7HnWMSWrWRSdXZUHe3+J3BbiJ
cjCGOBzIsbPjqnQzOXlZDnKsVG02CPV0Nv11j8PnYagRU7HNxEM448gCIZjoQhN4
IcA7jukoKSxI4a5eve5iQ7fGOHFAm7iQ2EA+MQIXD4aLtSKsNGk57EsCgYEA+iIl
LZyc3P9uTECv00NWL8eAVuqHesIu0h7tiOaxR/r36XL8IOfjauyc5I2CTPkXQcJc
ZrfIk17e0q/ei1Kzvr269u00ChMT82AnxVAwxjeHgliUbOQL0BWcrDwiFbsfttaX
bObLpjTJT6oqU6unxGohFHmILEG3dLGmz3k50KECgYEA7Rb+kAizzth7iqCw+Kqq
466Qjy83JwXpHuxNjTnV+6FJiQO8CflmjH0XyjTKlD59Ic/vbzVcful4BItps1cE
/8tiAg5vNv7HW2iILLQaQwAJtNZswZw8JI0+XwbW+xpu+Q39xyT8hnalHHd944gY
ACXsRPVb72NhGvp79TLyor0CgYEAyeP/gBY1okblHtTjVbC3Au+SzhSUb2gGZICk
Fuik1MVjjmDJ0kF4lJgQdoUlU72FoQUgkaPrV8+uJ/3dsTR6cg0vuBhy9WK6qqjE
0QTNqV+ul22pt05FnpmjEH2kwUd87JW+OR775tYaWputeEVHr0g+FQmW+Km+SokN
a86b9KECgYAdRGpvYsboviu1S4PdeOzz0VH21Nw3rPMO8Xumpr9P3gGbiY0GxOLg
Yy5McOLyGBDDSzvQU8plYGIWn6eGnsVEC2Lz3SIYaG5GQRZjMaCaCvfVMy59ipxW
VUpRDuaK58qGbkZZ5Cy0a5jqE2Mul+0a9iMn0bv3i0UQccZ+auUXLw==
-----END RSA PRIVATE KEY-----

Same OpenSSH private key with X509 certificate added

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA9IhylUyhWTE5bsR9R8xAe4/b0PBnYiCikLXkQgBAxfWCaW3h
12/FY4YflHv7EHjG7Zfo6ryMvuEEW8lNqXQVMRcP56U4GUfUU3aqZG7FFlFbKIU0
n9y2JHWaeef/hlOgQaJ5goX4yg38C9S0XT1YFSGGei/ifnz57DSfUgSv02Az7LFs
3oQene/9Cp9+umOAOFJIzqIiwXSN7weqM4BJ7qvqRh9ib54yIuMO+D/VTI2LyZJy
h4dZJzyM/O8ovOIw5b5dVnIPQlEG+u/1IBsJcDEkc9zSpcHYu47/mL9JoVPj0DPt
MsnDimk7PEP1fomUYbEipUC3CH41CmOvTR2LKwIDAQABAoIBAGTAG0W8TvYqu9kR
h/SfCSpfuVe61T2u6IvrBWLUK9vkLZk2tshGuvMofbZ15pmAyaV243ZjvTGgEGWo
6SCaPWj+cYUlo5l26NqCl+3NXdK2LnLhfy8mhr063yy/E57KscqZIKtQe5L0dBaD
ytRnw/Tg3UFWnWE0KNzTSZlHaRITUOTdZf/bBNL3rOZrK+5be60ueLJt6gMery1d
6kRdpef3Qh0nOXVChB9cWH8aE5unk6vEi9khC4p5DtpOesAlQt9jlT6GFdZJhkZ9
dvZ+1i3aNa6B4fGWe53F0ecoIAU+FsWnan+bCOxql80+3a7WEVoslozihZ+PNC1Q
bPADioECgYEA+kStLNV4tDkTlQJdeQz8kiLeI2j7HnWMSWrWRSdXZUHe3+J3BbiJ
cjCGOBzIsbPjqnQzOXlZDnKsVG02CPV0Nv11j8PnYagRU7HNxEM448gCIZjoQhN4
IcA7jukoKSxI4a5eve5iQ7fGOHFAm7iQ2EA+MQIXD4aLtSKsNGk57EsCgYEA+iIl
LZyc3P9uTECv00NWL8eAVuqHesIu0h7tiOaxR/r36XL8IOfjauyc5I2CTPkXQcJc
ZrfIk17e0q/ei1Kzvr269u00ChMT82AnxVAwxjeHgliUbOQL0BWcrDwiFbsfttaX
bObLpjTJT6oqU6unxGohFHmILEG3dLGmz3k50KECgYEA7Rb+kAizzth7iqCw+Kqq
466Qjy83JwXpHuxNjTnV+6FJiQO8CflmjH0XyjTKlD59Ic/vbzVcful4BItps1cE
/8tiAg5vNv7HW2iILLQaQwAJtNZswZw8JI0+XwbW+xpu+Q39xyT8hnalHHd944gY
ACXsRPVb72NhGvp79TLyor0CgYEAyeP/gBY1okblHtTjVbC3Au+SzhSUb2gGZICk
Fuik1MVjjmDJ0kF4lJgQdoUlU72FoQUgkaPrV8+uJ/3dsTR6cg0vuBhy9WK6qqjE
0QTNqV+ul22pt05FnpmjEH2kwUd87JW+OR775tYaWputeEVHr0g+FQmW+Km+SokN
a86b9KECgYAdRGpvYsboviu1S4PdeOzz0VH21Nw3rPMO8Xumpr9P3gGbiY0GxOLg
Yy5McOLyGBDDSzvQU8plYGIWn6eGnsVEC2Lz3SIYaG5GQRZjMaCaCvfVMy59ipxW
VUpRDuaK58qGbkZZ5Cy0a5jqE2Mul+0a9iMn0bv3i0UQccZ+auUXLw==
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDqTCCApECCQCozKP8XcNukDANBgkqhkiG9w0BAQUFADCBljELMAkGA1UEBhMC
RVMxEzARBgNVBAgMClBvbnRldmVkcmExDTALBgNVBAcMBFZpZ28xDDAKBgNVBAoM
A1BTQTETMBEGA1UECwwKQUlORlJBUEVSTDERMA8GA1UEAwwIeXZhbDV4YTAxLTAr
BgkqhkiG9w0BCQEWHmpvc2VtYW51ZWwuY2lnZXMxQGV4dC5tcHNhLmNvbTAeFw0y
MDA1MjYxMTAxNDNaFw0zMDA1MjQxMTAxNDNaMIGVMQswCQYDVQQGEwJFUzETMBEG
A1UECAwKUG9udGV2ZWRyYTENMAsGA1UEBwwEVmlnbzEMMAoGA1UECgwDUFNBMRMw
EQYDVQQLDApBSU5GUkFQRVJMMRAwDgYDVQQDDAd0ZXN0c3NoMS0wKwYJKoZIhvcN
AQkBFh5qb3NlbWFudWVsLmNpZ2VzMUBleHQubXBzYS5jb20wggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQD0iHKVTKFZMTluxH1HzEB7j9vQ8GdiIKKQteRC
AEDF9YJpbeHXb8Vjhh+Ue/sQeMbtl+jqvIy+4QRbyU2pdBUxFw/npTgZR9RTdqpk
bsUWUVsohTSf3LYkdZp55/+GU6BBonmChfjKDfwL1LRdPVgVIYZ6L+J+fPnsNJ9S
BK/TYDPssWzehB6d7/0Kn366Y4A4UkjOoiLBdI3vB6ozgEnuq+pGH2JvnjIi4w74
P9VMjYvJknKHh1knPIz87yi84jDlvl1Wcg9CUQb67/UgGwlwMSRz3NKlwdi7jv+Y
v0mhU+PQM+0yycOKaTs8Q/V+iZRhsSKlQLcIfjUKY69NHYsrAgMBAAEwDQYJKoZI
hvcNAQEFBQADggEBACo2AzZHcdvpEZuw9nqIlcrkAhGJV6b64TavSTR4/Itq7npV
ln+19wyOI1j4HpgKE41/XEm3kAlTdIcUa00e/yrcDuodrNCvhGcA0uvuunCO13Wq
XhDgk8CBL3NWn5jvDjWknEj7oDGDYmrBF9Dmbk3jlsp8Ptp1qATH1711dOynRSeT
0ajLMA96FZfUA6kk6tslBPKD1sjHekgEbsY/uht9R5yyDplrbw4dS2wddU585o/M
YOJ1JicYhEOjdNg2Ki30GORYVPp/rV0K8Hu+uU5iTSdMyYUx2KS96fk/62xSwZec
dWL8nLKKBz9XgX+UySBdJZFSFvr9ai0WZ/fdSV8=
-----END CERTIFICATE-----

And the same for the public keys ....

Standard RSA OpenSSH public key for the previous private example one

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD0iHKVTKFZMTluxH1HzEB7j9vQ8GdiIKKQteRCAEDF9YJpbeHXb8Vjhh+Ue/sQeMbtl+jqvIy+4QRbyU2pdBUxFw/npTgZR9RTdqpkbsUWUVsohTSf3LYkdZp55/+GU6BBonmChfjKDfwL1LRdPVgVIYZ6L+J+fPnsNJ9SBK/TYDPssWzehB6d7/0Kn366Y4A4UkjOoiLBdI3vB6ozgEnuq+pGH2JvnjIi4w74P9VMjYvJknKHh1knPIz87yi84jDlvl1Wcg9CUQb67/UgGwlwMSRz3NKlwdi7jv+Yv0mhU+PQM+0yycOKaTs8Q/V+iZRhsSKlQLcIfjUKY69NHYsr testssh@yval4cs0

OpenSSH public key for the previous private using X509 certificates

x509v3-sign-rsa MIIDqTCCApECCQCozKP8XcNukDANBgkqhkiG9w0BAQUFADCBljELMAkGA1UEBhMCRVMxEzARBgNVBAgMClBvbnRldmVkcmExDTALBgNVBAcMBFZpZ28xDDAKBgNVBAoMA1BTQTETMBEGA1UECwwKQUlORlJBUEVSTDERMA8GA1UEAwwIeXZhbDV4YTAxLTArBgkqhkiG9w0BCQEWHmpvc2VtYW51ZWwuY2lnZXMxQGV4dC5tcHNhLmNvbTAeFw0yMDA1MjYxMTAxNDNaFw0zMDA1MjQxMTAxNDNaMIGVMQswCQYDVQQGEwJFUzETMBEGA1UECAwKUG9udGV2ZWRyYTENMAsGA1UEBwwEVmlnbzEMMAoGA1UECgwDUFNBMRMwEQYDVQQLDApBSU5GUkFQRVJMMRAwDgYDVQQDDAd0ZXN0c3NoMS0wKwYJKoZIhvcNAQkBFh5qb3NlbWFudWVsLmNpZ2VzMUBleHQubXBzYS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQD0iHKVTKFZMTluxH1HzEB7j9vQ8GdiIKKQteRCAEDF9YJpbeHXb8Vjhh+Ue/sQeMbtl+jqvIy+4QRbyU2pdBUxFw/npTgZR9RTdqpkbsUWUVsohTSf3LYkdZp55/+GU6BBonmChfjKDfwL1LRdPVgVIYZ6L+J+fPnsNJ9SBK/TYDPssWzehB6d7/0Kn366Y4A4UkjOoiLBdI3vB6ozgEnuq+pGH2JvnjIi4w74P9VMjYvJknKHh1knPIz87yi84jDlvl1Wcg9CUQb67/UgGwlwMSRz3NKlwdi7jv+Yv0mhU+PQM+0yycOKaTs8Q/V+iZRhsSKlQLcIfjUKY69NHYsrAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBACo2AzZHcdvpEZuw9nqIlcrkAhGJV6b64TavSTR4/Itq7npVln+19wyOI1j4HpgKE41/XEm3kAlTdIcUa00e/yrcDuodrNCvhGcA0uvuunCO13WqXhDgk8CBL3NWn5jvDjWknEj7oDGDYmrBF9Dmbk3jlsp8Ptp1qATH1711dOynRSeT0ajLMA96FZfUA6kk6tslBPKD1sjHekgEbsY/uht9R5yyDplrbw4dS2wddU585o/MYOJ1JicYhEOjdNg2Ki30GORYVPp/rV0K8Hu+uU5iTSdMyYUx2KS96fk/62xSwZecdWL8nLKKBz9XgX+UySBdJZFSFvr9ai0WZ/fdSV8=

Configuring the server to accept X509 certificates for the user

Now came the really interesting part.

We don't need to copy the public key on server's SSH configuration for the user. Just add the "subject" information of x509 certificate to authorized_keys in destination server.

To get the subject run in the client

openssl x509 -noout -subject -in .ssh/id_rsa

On the server, add this line with the prefix x509v3-sign-rsa subject= to the server's .ssh/authorized_keys.
This line will have a content similar to this one:

x509v3-sign-rsa subject= /C=ES/ST=Pontevedra/L=Vigo/CN=testssh/emailAddress=josemanuel@ciges.net

As we can see, the authentication is really made trusting the CA for any valid x509 certificate from the user. We could generate a new certificate and it will be accepted with no intervention on server side.

Test the connection

Now we should be able to connect from client to server without a password. For example, to list the /home directory on server we could use

ssh server "ls -l /home"

If we run in with option -vvvv (yes, four) for verbose mode we could see info lines like this, telling that x509 certificates are being used:

Trying private key: /users/login/testssh/.ssh/id_rsa
debug3: sshkey_load_private_type() type=10, filename=/users/login/testssh/.ssh/id_rsa
debug3: ssh_x509_sign: key alg/type/name: x509v3-ssh-rsa/RSA+cert/x509v3-sign-rsa
debug3: ssh_x509_sign: alg=x509v3-ssh-rsa, md=rsa-sha1
debug3: ssh_x509_sign: signame=ssh-rsa

And it should work 🙂

Creation of certificate for the host in the server machine

The first time we try to connect to an OpenSSH server, the public key of the destination host is added to the client's known_hosts file. The user must accept it interactively of use the option "StrictHostKeyChecking no" to don't check remote host identity.

It will be more interesting if the server's identity could be verified by a external certification authority. With OpenSSH we can configure it the same way we have done with the user.

So, first in the server machine

  • generate a signing request for the host rsa key and send it to the control server to be signed.

The host RSA key is already present, we don't have to create it, as the OpenSSH daemon generates one when it's installed.

As "Common Name" we will use the host name with the domain

openssl req -new -key /opt/openssh/data/ssh_host_rsa_key -out ssh_host_rsa_key.csr
scp ssh_host_rsa_key.csr control:/tmp

Now, in the control server, where the CA files are stored, we create a signed certificate for this key

  • create a matching signed certificate for the host's private key
cd /tmp
openssl x509 -req -days 3650 -in ssh_host_rsa_key.csr -out ssh_host_rsa_key.crt -CA ca.crt -CAkey ca.key

The result file, ssh_host_rsa_key.crt is what we want

Again, in the server,

  • add the generated certificate to the server SSH private key and create also the public key
scp control:/tmp/ssh_host_rsa_key.crt .
cat ssh_host_rsa_key.crt >> /opt/openssh/data/ssh_host_rsa_key
ssh-keygen -y -f /opt/openssh/data/ssh_host_rsa_key > /opt/openssh/data/ssh_host_rsa_key.pub

Test the connection to the server

Now, in the client machine, we can delete the known_hosts file and try to make a connection to the server.

We will have a message similar to this one:

$ ssh server "ls -l /home"
The authenticity of host 'server (192.168.56.241)' can't be established.
RSA+cert key fingerprint is SHA256:u/tkQ1WQUqptnj8Fbyn1nHOPU+HgchxJ9a13MQKq7FQ.
Distinguished name is 'C=ES,ST=Pontevedra,L=Vigo,O=Internet Widgits Pty Ltd,CN=server.inetpsa.com,emailAddress=josemanuel@ciges.net'.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

After telling "yes", we will have the following line in known_hosts

server,192.168.56.241 x509v3-sign-rsa Subject:C=ES,ST=Pontevedra,L=Vigo,O=Internet Widgits Pty Ltd,CN=server.inetpsa.com,emailAddress=josemanuel@ciges.net

Once again, no public key is added to the file. With the host name, ip and certificate description OpenSSH has enough. Is the X509 certificate presented by the server which is used to validate the host as as legitimate one.

We could verify that the remote host X509 certificate is being used connecting with very verbose level information set

$ ssh server -vvv "ls -l /home"

and look for the lines similar to this:

debug1: Server host key: x509v3-sign-rsa SHA256:u/tkQ1WQUqptnj8Fbyn1nHOPU+HgchxJ9a13MQKq7FQ
debug1: Host 'server' is known and matches the RSA+cert host key.
debug1: Found key in /users/login/testssh/.ssh/known_hosts:1

Done 🙂