Compilation of OpenSSH 8.1 on AIX

Table of Contents

1 Preparing the compilation environment
2 Compilation of zlib
3 Compilation of OpenSSL
4 Compilation of PKIX-SSH
5 Daemon running
6 Before creating a package for distribution

In this post I will explain how to to compile OpenSSH 8.1 on AIX 7.1. The system has a 64 bits PowerPC architecture.

The installation will be made on a custom directory /soft/openssh-8.1 and the following software will be included

zlib 1.2.11 for compression support
OpenSSL 1.1.1d, for some system where LibreSSL did not compile, as in AIX
PKIX-SSH 12.3, a fork of OpenSSH from Roumen Petrov with support for X.509 v3 certificates.

The link to the source code of PKIX-SSH 12.3 is to my own Github repository with the modifications told in this post to successfully compile it on AIX

Compilation on AIX is not a trivial task, and for a guy coming from a Linux world, some steps not so evident, has been needed.

In my AIX system GNU compilation tools which are installed under /soft/freeware or /usr/ccs and present in the system's path

So the exact steps will be ...

Preparing the compilation environment

We are going to set the path for finding our binaries and the following variables used in compilation time: blibpath, LIBPATH, CFLAGS, CC, OBJECT_MODE, NM and AR.

The key for compiling on AIX is using the variable LIBPATH, equivalent to LD_LIBRARY_PATH on Linux.

In this case in particular the variable blibpath was set also, as is used in some of the configure scripts.

OPENSSH_PATH="/soft/openssh-8.1"
# Library path in linking time
export blibpath="$OPENSSH_PATH/lib64:/usr/lib64:/usr/lib:/lib:/opt/freeware/lib64:/opt/freeware/lib:/usr/ccs/lib"
# Library path in run time
export LIBPATH="$blibpath"
export LDFLAGS="-Wl,-blibpath:$blibpath,-brtl"
export CFLAGS="-g -O2"
export CPPFLAGS="-I$OPENSSH_PATH/include"
export CC="gcc -maix64"
export OBJECT_MODE=64
export NM="nm -X64"
export AR="ar -X64"

The architecture option for C compiler was set in CC variables, instead of in CFLAGS because de configure script didn't take it for all the compilation commands.

Compilation of zlib

gunzip zlib-1.2.11.tar.gz
tar xvf zlib-1.2.11.tar
cd zlib-1.2.11
./configure --prefix=$OPENSSH_PATH --libdir=$OPENSSH_PATH/lib64
make
make install

In AIX I have found that 32 bits library are trying to be used. That provoke errors because we are compiling binaries for 64 bits. So, in this case, a lib64 directory under destination path is created.

After installation we should rename libz.a to not use it (an error is thrown when the compiler tries to use it)

mv $OPENSSH_PATH/lib64/libz.a $OPENSSH_PATH/lib64/libz.a.out

Compilation of OpenSSL

Note here again the use of the blibpath variable in options for the compiler ...

gunzip openssl-1.1.1d.tar.gz
tar -xvf openssl-1.1.1d.tar
cd openssl-1.1.1d
./Configure aix64-gcc --prefix=$OPENSSH_PATH --openssldir=$OPENSSH_PATH --libdir=$OPENSSH_PATH/lib64 -lz zlib
make
make install

OpenSSL was used instead of LibreSSL, a fork from OpenSSL with more recent patches applied and enforced security, because when trying to compile it we were unable as it stopped with the following error:

CCLD     libcrypto.la
ld: 0711-317 ERROR: Undefined symbol: .vsyslog_r

And, well, we don't know how to solve it :-(.

Compilation of PKIX-SSH

tar xvf pkixssh-12.3.tar.gz
cd pkixssh-12.3

Before the config, make, make install phase we should modify the user_path variable in configure file to have the path of our custom installation used before the default's systems (if not the system's OpenSSH binaries could be used instead of our custom solution).

The line 19541 in configure should be established to (you can download configure file from here)

user_path=$t_bindir:$user_path

In AIX we should modify also the source code of session.c (functions read_environment_file and do_setup_env) because if not PATH environment variable is not set, and the binaries will not be found in our custom directory when a new session is opened.

--- session.c.original  2020-03-08 19:02:27.000000000 +0100
+++ session.c   2020-05-14 16:50:26.000000000 +0200
@@ -906,7 +906,9 @@
                if (whitelist != NULL &&
                    match_pattern_list(cp, whitelist, 0) != 1)
                        continue;
-               child_set_env(env, envsize, cp, value);
+         /* If environment variable is PATH then don't override it */
+         if (strcasecmp(cp, "PATH") != 0)
+             child_set_env(env, envsize, cp, value);
        }
        free(line);
        fclose(f);
@@ -1041,6 +1043,9 @@
        child_set_env(&env, &envsize, "LOGNAME", pw->pw_name);
  #ifdef _AIX
        child_set_env(&env, &envsize, "LOGIN", pw->pw_name);
+       /* In AIX PATH variable is not updated, so set here to _PATH_STDPATH */
+       child_set_env(&env, &envsize, "PATH", _PATH_STDPATH);
+     debug3("AIX: PATH is not updated in do_setup_env");
  #endif

You can download the new file session.c from here.

Now we can begin the compilation

./configure --prefix=$OPENSSH_PATH           \
        --sbindir=$OPENSSH_PATH/bin          \
        --libexecdir=$OPENSSH_PATH/bin       \
        --libdir=$OPENSSH_PATH/lib64         \
        --includedir=$OPENSSH_PATH/include   \
        --sysconfdir=$OPENSSH_PATH/data      \
        --datadir=$OPENSSH_PATH/data         \
        --with-pid-dir=$OPENSSH_PATH/data    \
        --without-stackprotect               \
        --with-zlib=$OPENSSH_PATH/lib64      \
        --with-pam                           \
        --with-md5-passwords                 \
        --with-privsep-path=/var/empty/sshd  
make
make install

The result binaries have the libraries correctly set :-). Notice that in this particular config all binaries will be installed under bin directory, also with the ones that are normally installed under sbin and libexec.

$ ldd /soft/openssh-8.1/bin/sshd
/soft/openssh-8.1/bin/sshd needs:
         /soft/openssh-8.1/lib/libssl.a(libssl64.so.1.1)
         /usr/lib/libpam.a(shr_64.o)
         /soft/openssh-8.1/lib/libcrypto.a(libcrypto64.so.1.1)
         /soft/openssh-8.1/lib/libz.so
         /usr/lib/libc.a(shr_64.o)
         /usr/lib/libpthreads.a(shr_xpg5_64.o)
         /usr/lib/libmls.a(shr_64.o)
         /usr/lib/libz.a(libz.so.1)
         /opt/freeware/lib64/libgcc_s.a(shr.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)
         /usr/lib/libmlsenc.a(shr_64.o)
         /usr/lib/libodm.a(shr_64.o)

$ dump -X64 -Hv /soft/openssh-8.1/bin/sshd

/soft/openssh-8.1/bin/sshd:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x0000024f       0x0000073e       0x000000cc

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000006       0x0000ab80       0x00002446       0x0000ac4c

                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER
0      /soft/openssh-8.1/lib:/usr/lib64:/usr/lib:/lib:/opt/freeware/lib64:/opt/freeware/lib:/usr/ccs/lib                         
1                                    libssl.a            libssl64.so.1.1
2                                    libpam.a            shr_64.o
3                                    libcrypto.a         libcrypto64.so.1.1
4                                    libz.so
5                                    libc.a              shr_64.o

The ldd command tells what libraries are used and the dump -X64 -Hv tells in what paths this libraries are going to be used.

Daemon running

If no specific configuration is needed them we could run our new daemon with:

/soft/openssh-8.1/bin/sshd -f /soft/openssh-8.1/data/sshd_config

Or, if we want to have a detailed log (in the example at /users/openssh-8.1/log/openssh.log), we could use:

/soft/openssh-8.1/bin/sshd -E /users/openssh-8.1/log/openssh.log -f /soft/openssh-8.1/data/sshd_config

Before creating a package for distribution

Make attention to delete hosts keys (and log files if present) created before creating a package for distribution. This keys are specific for each host and normally are generated in the installation phase or when the daemon is started

rm /soft/openssh-8.1/data/ssh_host_*

Done! 🙂