How I compiled deark for 64bit AROS

Mantainer's diary. Some days ago I got a short interesting message from Bob Wilson: «Hello Paolone, I recently came across a command line program called deark. It's not a program that I would normally port, but you might find a good use for it. It should be easy to compile, if you want to try, add "CC= gcc" as the first line of the makefile». He was right: deark is one of those little big swiss army knives that most of us - dealing with PCs and the information technology every day - learn to love. It parses and extracts resources from an amazing variety of file formats, many of them almost gone forgotten these days. If you have a old file and nothing to process it, try deark: it might do something useful with it. Problem was, of course, porting it to AROS and, best of all, making it available for the 64bit version as well. Exactly like Bob said, compiling it for 32bit AROS had been straightforward: just adding the proper CC variable in the makefile and casting a 'make' did it. Another completely different story was, unluckily, the 64bit version.

There are two premises to be done: first of all, currently 64bit gcc for AROS doesn't work as expected, so - in order to compile sources for 64bit AROS - you should use either a crosscompiler or the AROS build system, which provides the crosscompiler by itself. Second, I am not a coder at all, which is limiting all my coding skills to shell and LUA scripts, and to the 'make' command: if you ask me to change a ./configure script, or a .c source file directly, I stop understanding almost at once. But, since I said I would have made a 64bit AROS distribution, I am now trying to compile everything I already need for it, or at least I suspect it may be useful in the future. And deark falls exactly in the latter family. So, I'll now explain what I did, in the hope it may be useful for others, too.

First of all, I set up a 64bit Ubuntu virtual machine and cloned the AROS git repository, installed all needed packages to compile AROS and made a initial build of the operating system. This operation is mandatory to download and install also all needed Linux tools and libraries, which are not included in the AROS sources, and all missing parts that were left outside for space reasons. Then I created a 'icaros' subdirectory in my local repo, entered it, and made also a new subfolder for every program I tried porting. I placed sources in these subfolder and, for every given program, created a mmakefile.src "recipe" that would tell the crosscompiler what to do. Please remember my disclaimer above: I am no coder at all, so I just have a vague idea of what these files do and how they work. If you can code in C or in other high level language, you should be far better than me in this. This is why I am not explaining how these .src files work: because I don't know this by myself. The magic is that... well, they work! And we should all be grateful to AROS coders for this!

A mmakefile.src file can be like this, for a very simple .c program. This is the one for poorx:

include $(SRCDIR)/config/aros.cfg
#MM local-icaros-poorx : includes linklibs
FILES := poorx
EXEDIR := $(AROS_CONTRIB)/$(AROS_DIR_DEVELOPER)/Build
NOWARN_FLAGS := $(NOWARN_FORMAT)
USER_CFLAGS := $(NOWARN_FLAGS)
%build_prog mmake=local-icaros-poorx \
progname=poorx targetdir=$(EXEDIR) \
files=$(FILES)
%common

the FILES variable lists all the .c files needed for building, while the progname one tells the name of the resulting binary file. EXEDIR variable is the path where the resulting program will be placed (according to AROS' filesystem) while NOWARN_FLAGS and USER_CFLAGS should be the place for all those nice options that coders know how to use, while I don't. Then there is the name for the recipe, which in our case is local-icaros-poorx. Once this has been written, and I have all needed files in-place, I can move to the root directory of my repo (in my case, ~/sources) and cast a command like:
make local-icaros-poorx
which will hopefully produce the poorx binary. Things get very complicated (for me, not for you expert coders!) when several .c files are needed for building, like in the case of deark. I had a very hard time creating a working one: I don't know yet how to manage prior building of dependencies, nor I know the right way to handle subdirectories, fact is that deark sources have the following structure:

With main sources included in the /src subfolder, all needed modules in the /modules directory and other needed components in the /foreign one. So I decided the 'best' way to start would be creating the mmakefile.src in the /src subfolder instead. This drawer looks like this:

a quite huge list of .c files, isn't this? well, so my initial FILES variable looked like:

FILES := deark-bitmap deark-char deark-data deark-dbuf deark-ucstring \
deark-fmtutil deark-font deark-liblzw deark-miniz \
deark-modules deark-tar deark-unix deark-user deark-util \
deark-win deark-cmd

I just took the precaution of placing the main program (deark-cmd.c) at the end. Remember, in the FILES list, names don't have the .c extension in the end. First compile trial showed me that I needed to build all .c files included in /modules as well. But when, exactly? Build error suggested me that they should have gone between deark-ucstring and deark-fmtutil. The /modules subdirectory includes 125 .c files, how could I include them?

The right "chicken tomorrow" answer would be: "bug someone on slack or on the AROS mailing list and find someone so patient to care about you, learn how to properly handle this and do it well" but, since I am a "egg today" kind-of guy, that would have been simply too wise. So I decided to include all .c modules here with the ../modules/name format. I'm sure bash has its own syntax to get the same result, but since I am more comfortable with Amiga shell, I used the List command instead (BTW: having hosted AROS running on Linux allows handling of Linux files with AROS shell. Cozy!):
LIST LFORMAT "../modules/%M \" >temp.txt
and copied/pasted/adapted the result in the FILES section:

FILES := deark-bitmap deark-char deark-data deark-dbuf deark-ucstring \
../modules/sunras \
../modules/cab \
../modules/binhex \
../modules/bsave \
.......................
.......................
../modules/alphabmp \
deark-fmtutil deark-font deark-liblzw deark-miniz \
deark-modules deark-tar deark-unix deark-user deark-util \
deark-win deark-cmd

Another problem I had, was the fact that all .c files from the /modules subdirectory included lines like these:

#include <deark-config.h>

#include <deark-private.h>

#include <deark-fmtutil.h>

that I had to turn into these:

#include </home/paolone/sources/icaros/deark/src/deark-config.h>
#include </home/paolone/sources/icaros/deark/src/deark-private.h>
#include </home/paolone/sources/icaros/deark/src/deark-fmtutil.h>

because, otherwise, they couldn't find header files and building stopped every time with a new "file not found" error. Once again, the right solution would have been different, but a faster hack woud have given a 64bit deark executable more quickly. So I ask you to forgive my rudeness, and warmly suggest to learn the right solution instead. If you're in a hurry, however, you can do like me.

How to modify the same three lines in 125 .c files? Linux surely had its own commands, and I even considered using sed, however I noticed that examples on line used a syntax like:
sed -i 's/old-string/new-string/g' filename
and, since my strings included slash characters as well, I simply didn't want to look for the smarter solution. Why bothering, when I still have both AROS' List command and AROS gsar port? Three commands like

LIST LFORMAT "gsar -o %N -s<deark-config.h> -r</home/paolone/sources/icaros/deark/src/deark-config.h>" >subst.s
LIST LFORMAT "gsar -o %N -s<deark-private.h> -r</home/paolone/sources/icaros/deark/src/deark-private.h>" >>subst.s
LIST LFORMAT "gsar -o %N -s<deark-fmtutil.h> -r</home/paolone/sources/icaros/deark/src/deark-fmtutil.h>" >>subst.s

would create a very nice subst.s script I could EXECUTE within AROS. There would be even a smarter "create the gsar based script with a single command" solution, but since I'd had to understand how to handle strings which include spaces AND double commas, I went as usual for the faster+ugly one. In the end, I ended up having all modules .c sources modified in a way I can't share them anymore with anyone, but that at least compile on my machine. In fact, when I finally gave the command
make local-icaros-deark-src
for the last time, it compiled without much issues. And we now have deark for 64bit AROS as well!