BACKWARDS BUILDS

A nice warm day today [yesterday, by the time I finished writing 
this], at least compared against the endless cold dreary winter's 
days that have set the mood for months. Great for working on one of 
my outdoor projects, but I was rather unfocused and distractable. 
Ended up on my chair in the nice warm sun room, took off all my 
clothes, and grabbed my laptop for some hardcore... Linux!

Ages ago I talked about battling with getting OpenSSH to build for 
the old version of Linux that I run on the old PCs like the one I'm 
posting from now. The problem was actually with the configure 
script because I wanted it to link against OpenSSL libraries in a 
custom directory to avoid messing with the old OpenSSL lib used by 
other programs, and the script just didn't want to play ball. I 
couldn't even manage to hack the configure script into submission, 
so I eventually gave up.

Then I found PuTTY, which turned out to support Linux as well as 
Windows, didn't require any extra libraries, and had all the extras 
that come with openSSH, particularly an SFTP client. But that ran 
afoul of Tilde.Club when I signed up earlier this year, because it 
didn't support the latest RSA key magic using SHA-2 instead of 
SHA-1. Yes, it seems SSH software might only last a couple of years 
these days before becoming obsolete. Bloody internet.

No problem though, PuTTY added support for that later, so I can 
just compile the latest version. But when I tried that about a week 
ago things took a turn full-circle. PuTTY has switched to using 
CMake as a build system. Whereas configure scripts and Makefiles at 
least try to run everywhere, CMake builds require a cmake program 
that's only a few years old at the latest. Building the latest 
CMake requires a C++11 compiler, but I found an earlier version 
that could still compile with old GCC, but the build failed early 
looking for a system header file that didn't exist. It didn't look 
hopeful. Like OpenSSH before, I've been roadblocked by the build 
system.

So I gave up and built the last version of PuTTY before it switched 
to CMake, PuTTY 0.76, which Tilde.Club does like, for now. But 
clearly that solution is only going to last so long as well. What I 
need is to build from a modern system, with a modern CMake. But 
Linux, frustratingly, isn't kind to those attempting binary 
compatibility between systems of different ages.

Actually that blame is largely misplaced. It's not actually Linux 
that's the problem, it's Glibc. Glibc, for the uninitiated, is the 
library that implements the standard C function routines (open, 
printf, etc.) of the C programming language. It talks between 
programs and the operating system kernel, and as a GNU project it's 
probably the overall most common GNU part of "GNU/Linux". 
Unfortunately programs compiled for new versions of Glibc generally 
won't run with older Glibc versions (although the Glibc developers 
do a great job at keeping old binaries working with newser Glibc 
versions). In theory static linking (GCC's "-static" option) solves 
this problem by bundling Glibc in with the program's binary, but in 
practice it doesn't because Glibc has some features which cause it 
to load extra files from the system. Glibc's not compatible with 
old versions of some of these files, particulary some which are 
actually object files (like mini libraries themselves) and don't 
get included in the static binary. Glibc's developers don't really 
like static linking, and they've basically willfully broken it, 
especially for building networking-related software like PuTTY.

AppImages give the impression of having solved this, but they 
haven't. I built an AppImage for a complicated program a while ago. 
While in the (very slow) process of creating a monsterously long 
command line of manually-added rules to make the AppImage work 
successfully, I discovered that the standard solution to the Glibc 
compatibility problem is that developers simply pick the oldest 
Linux distro that they expect anyone to use, and run that to build 
the AppImage. As such, most users of the AppImage run later Glibc 
versions, which are designed to be compatible with such 'older' 
binaries, but it doesn't solve anything for going the other way and 
running 'new' binaries on old Glibc. It also means developers are 
tempted to bundle old support libraries, from the old Linux distros 
that they use, into the AppImages, which isn't good (there are a 
few solutions for building compatible AppImages on newer systems, 
but they're quite complicated and still require picking a 
relatively-recent minimum Glibc version).

On top of that, Glibc since version 2.26 simply won't run on Linux 
kernel versions older than 3.2, which also rules it out for 
building a static binary to work for me on older Linux.

So today's "hardcore Linux" task was to try an alternative to 
Glibc: Musl. This is designed as a more lightweight C library than 
Glibc, and has been adoped by some Linux distros instead of Glibc. 
Alpine Linux is particularly popular amongst these, and has amassed 
a surprisingly large number of programs in their package repo 
apparantly built against Musl. Even PuTTY! The Musl docs also don't 
mention a Linux kernel requirement beyond version 2.4 for "simple 
single-threaded applications" which _might_ include PuTTY? Plus it 
doesn't do any 'tricks' with object files like Glibc, so static 
builds work like they should.

Musl _can_ be installed on Debian 10 Buster (or in my case the 
matching version of Devuan) too. Packages will still use Glibc 
because that's what they were compiled for, but you can compile 
your own software against Musl by setting the compiler to 
"musl-gcc" after installing "musl-tools".

My Devuan system is actually x86_64 though, and I want to build for 
x86. I _could_ set up a separate x86 build system, but in theory 
GCC's x86_64 build target also supports building x86 binaries, even 
when it's not running on x86. Debian also has "multiarch" for 
installing 32bit libraries on 64-bit installations, which I already 
set up for running Wine. In theory this all goes together to mean 
that I can compile static, 32bit, Musl libC, builds of software on 
x86_64 Devuan!

In practice everything fought against me on this, but after a 
couple of hours I'd worked out these steps to success (for C 
programs at least):

Install these packages with their dependencies on multiarch-enabled 
Debian/Devuan:
 gcc-multiarch musl-tools musl-dev:i386

You need a GCC "specs" file for building against the 32-bit Musl 
library, but the Debian package system doesn't want to install it 
from musl-tools:i386 without replacing GCC with its 32-bit 
equivalent (damn obstinate package systems!), so make it from the 
64-bit version (run as root):
  sed 's/x86_64/i386/g' /usr/lib/x86_64-linux-musl/musl-gcc.specs > \
  /usr/lib/i386-linux-musl/musl-gcc.specs

Set the CFLAGS environment variable with all the right magic words 
for CMake (or a 'configure' script) to whisper to GCC (Debian 
packages, presumably including Musl, are currently compiled for 
i686 arch.):
  export CFLAGS='-m32 -march=i686 -specs /usr/lib/i386-linux-musl/musl-gcc.specs -static -Xlinker -melf_i386'

Run CMake (or ./configure):
  cmake -DCMAKE_BUILD_TYPE=Release .

Edit the newly created CMakeCache.txt file as required (or run one 
of the GUI CMake interfaces).* I built just the command-line 
programs, not "putty" itself which uses GTK.

Build with CMake (or "make"):
  cmake --build .

Now you should have 32-bit i686 static binaries which will run on 
the x86_64 build system, and on older Linux distros, either x86_64 
or x86.

To be sure that it's really working, you can try building a test 
program described on this webpage (which otherwise stops a bit 
short of describing a complete solution for the task):
https://www.geeksforgeeks.org/compile-32-bit-program-64-bit-gcc-c-c/

------------------------------------------------------------------
// C program to demonstrate difference
// in output in 32-bit and 64-bit gcc
// File name: geek.c
 
#include<stdio.h>
int main()
{
  printf("Size = %lu\n", sizeof(size_t));
}
------------------------------------------------------------------

When built as a 64-bit binary it prints "Size = 8" to the terminal 
when executed, and as a 32-bit binary it prints "Size = 4".

Build it with dynamic linking first to confirm that the GCC command 
arguments are really causing it to link to Musl instead of Glibc:
  gcc -m32 -march=i686 \
  -specs /usr/lib/i386-linux-musl/musl-gcc.specs \
  -Xlinker -melf_i386 -o geek geek.c

Now check with "file" to see the arch and "interpreter" used by the 
created binary:
  file geek
geek: ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), 
dynamically linked, interpreter /lib/ld-musl-i386.so.1, with 
debug_info, not stripped

So it's an x86 binary, and running it will therefore print "Speed = 
4", plus "interpreter /lib/ld-musl-i386.so.1" means it's linked to 
Musl instead of Glibc!

Now for the static binary:
  gcc -m32 -march=i686 \
  -specs /usr/lib/i386-linux-musl/musl-gcc.specs -static \
  -Xlinker -melf_i386 -o geek geek.c
  file geek
geek: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 
statically linked, with debug_info, not stripped

So after all that, did it make a PuTTY binary that worked on my old 
Linux distros on my old PCs? No. They just give me "segmentation 
fault".

It does work on older 32-bit Linux distros. It worked on Debian 7 
"Wheezy". But it doesn't go as far back as I needed it to. Musl 
isn't as compatible with old Linux kernels as I'd hoped, or maybe 
the problem is something else. Anyway it turned out to be pretty 
much a waste of time for me, but the technique might come in handy 
later on.

I've also uploaded my i686 Musl static binaries of the PuTTY 
command-line tools here, for anyone who might want to give them a 
try. You're welcome to report back the earliest Linux that you can 
run them on. I'm too lazy to test with lots of old Linux kernels to 
find out where the cut-off point actually is:
gopher://aussies.space/5/~freet/cupboard/putty_0.78_musl_static.tar.bz2

The next step for me might be to try and chroot into a copy of the 
root filesystem from my target Linux distro on a modern distro with 
modern CMake (accessed via a bind mount to the 'real' root FS and 
run with LD_LIBRARY_PATH). Then in theory I could run the CMake 
command where it thinks it's running in the old distro and 
generates corresponding makefiles. After that I could copy that 
PuTTY build directory to the real copy of the old distro and run 
"make" in there.

Or maybe it's time to go back and have another go at OpenSSH? 
That's really getting desperate after the agony of last time though.

 - The Free Thinker

* I really can't believe that the CMake developers haven't 
implemented an equivalent to "./configure --help", which has always 
been my starting point for building anything with a configure 
script. CMakeCache.txt is full of jumbled up options which makes it 
very hard to read (bare hand-edited makefiles are more readable!). 
Then, if you're trying to create a repeatable command to build 
later versions, you've got to then delete the CMakeCache.txt file 
and run the "cmake" command again with the changed options set as 
"-D[option]=[value]" arguments. What an awkward design! Neither 
PuTTY, nor the one other program I've compiled using CMake, 
bothered to document custom build options in their documentation 
either.