Skip to content

Design of Makefiles and Visual Studio project build

Jump to bottom
Robert Edgar edited this page Dec 23, 2021 · 9 revisions

How to build MUSCLE

This page is about design, if you just want to know how to build see Building MUSCLE.

Design objectives

  1. The Visual Studio project is the primary source, because this is my preferred development environment.

  2. Binary files such as object files and the executable must be written to a separate directory for each target platform (Linux, OSX, Windows) to avoid polluting the directory containing the source code and to avoid collisions where different object files have the same pathname (e.g., object files xyz.o for Linux and OSX).

  3. A version control commit identifier must be embedded in the source code so that it can be displayed at runtime. This embedding must be automated such that the identifier is written only if a source code file has been updated. The latter is to ensure that an "up to date" source code directory does not trigger a re-build. It seems that version embedding along these lines should be a simple and common requirement, but I found it very difficult to implement in a way that is compatible both with Visual Studio's proprietary build system and with a Makefile.

  4. A Makefile must not require manual updates when the list of source files and headers changes. This could be solved either (a) by generating a Makefile from the Visual Studio project files (as attempted in github.com/rcedgar/vcxproj2makefile), or (b) by using wildcards to include all source and header files. When I first committed MUSCLE to github I was using vcxproj2makefile.py, but this turned out to be rather fragile and didn't do a good job of accommodating tweaks needed to compile on different platforms. At the time of writing I am trying to implement strategy (b). My concern about (b) is that it will include "orphaned" source files that are no longer needed in the build. These are likely to arise for a couple of reasons: (1) because Visual Studio does not enforce git integration and ignores files that are not explicitly included in the project, and (2) by merging of different branches by git, though I don't understand the merge process very well at this point. Orphaned source files which break the build will hopefully be identified by a CI workflow which I'm also working on implementing.

  5. The following platforms must be continuously supported without manual intervention (e.g. editing Makefiles) when the project is updated: (a) the setup I use at home for on-going development, (b) Linux under WSL at home, (c) common Linux variants which a user might have, (d) ditto OSX, (e) github runners for Linux, OSX and Windows to enable CI/CD workflows.

  6. The source code and build mechanism must be broadly portable to compiler and operating systems in common use.

For me, it turned out to be very challenging to meet these objectives in practice.

Platform-specific build directory name

Visual Studio automatically constructs separate sub-directories for each distinct platform (e.g., x64/Release). In the Makefile, I use a shell escape to assign the output of uname to a variable named $(OS):

OS := $(shell uname)

A list of object filenames with the platform name directory prefix is constructed by a shell escape to sed:

OBJS := $(shell echo *.cpp | sed "-es/^/$(OS)\//" | sed "-es/ / $(OS)\//g" | sed "-es/\.cpp/.o/g")

Working around gcc problems under OSX

In recent releases of OSX, there are two big compatibility problems: gcc is an alias for clang, and OpenMP is not supported by the default installation of clang. This is solved by requiring that gcc-11 is installed, which can typically be done by brew install gcc or similar. The Makefile re-assigns $CXX on OSX (where uname returns Darwin) to ensure that genuine gcc is used:

CXX := g++
ifeq ($(OS),Darwin)
    CXX := g++-11
endif

Commit identifier string

This git command is used to generate a string which identifies the commit:

git describe --abbrev=6 --dirty --long --always

This returns something like cada96-dirty, where dirty indicates there are uncommitted changes. This string must be surrounded by quotes because I couldn't figure out a way to put quotes around an unquoted string taken from an #include file. Quotes are added using sed:

git describe --abbrev=6 --dirty --long --always | sed '-es/^\(.*\)$/"\1"/' > gitver.txt

In myutils.cpp the string is embedded using #include "gitver.txt". Probably the requirement for sed could be eliminated by using something like

echo \" $(git...) \"

but sed has worked fine on every platform where I have tested so far.

Triggering the commit identifier string under Visual Studio

A pre-build event runs gitver.bat, which checks whether Cygwin is installed by testing whether the file c:\cygwin64\bin\bash.exe exists. If Cygwin is not installed, nothing happens and the build will continue without updating gitver.txt. This ensures that the build will work on Windows platforms where Cygwin is not installed, e.g. a github workflow runner. If Cygwin is installed, a bash script is run which requires that the Cygwin version of git is installed. I prefer Cygwin to requiring a native Windows version of git because I don't know a robust solution to ensure that git is in the PATH when the pre-build event runs.

Triggering the commit identifier string in the Makefile

The Makefile has the following dependency:

gitver.txt : $(SRCS)
    bash ./gitver.bash

The $(SRCS) variable has the source filenames obtained by running ls in the shell: SRCS := $(shell ls *.cpp *.h). Probably there is a way to do this using make wildcards but I'm a make novice and the Makefile syntax is baffling; this seems good enough to me.

Clone this wiki locally