The Compiler Driver (1/5)

Let's start by clearing a common misconception. When talking about compilers, the names of clang, gcc, or, on Windows, CL.EXE will come to mind. These are the names of CLIs (Command-Line Interface) used to build executables but invoking them doesn't directly call the compilers. These CLIs are in fact compiler drivers.

Turning source code into an executable is a multiple step process. It can be viewed as a pipeline where stages communicate via artifact files. In the case of K&R, program hello.c requires three stages before the computer can greet you.

First the source file hello.c is preprocessed into a translation unit (TU) hello.tu. Then the TU is compiled into an object hello.o. Finally the linker turns the object into an executable. Since we did not give the driver a name for the output (-o), the file is named a.out.

Compiler drivers are a convenient way to invoke all the tools auto-magically with a single command. We can run the driver in verbose mode (-v) or in dry-run mode (-###) to see what is happening behind the scene.

$ clang -v hello.c
clang -cc1 -o hello.o hello.c
ld -o a.out hello.o

There are three important things to notice in this trace.

First, we see that clang is calling itself with the -cc1 flag. Because it is convenient for distribution, the CLI contains both the driver and the compiler. By default the executable behaves like a driver but if invoked with the flag -cc1, it behaves like a compiler. Note that the linker (ld) is its own executable and we will see why later.

Second, even though three stages were mentioned earlier, we only see two commands issued (one to the compiler clang -cc1 and one to the linker ld). The C Preprocessor (cpp) used to be a standalone program but it is no longer the case. Instead of loading a .c file to memory, write back a .tu to disk, only to load it to memory again, it is much more I/O efficient to pre-process inputs inside the compiler and compile right away.

Lastly, and perhaps most importantly, we see the linker ld invocation. Wouldn't it be more efficient to include the linker in the driver, the same way cpp is embedded? No, because of two reasons.

1. The compiler ingests one .c and outputs one .o. It has a low memory footprint. The linker on the other side, must use all the .o files at once to generate the executable. Keeping all these .o in memory would stress the system too much on big projects.
2. Compilers and linkers are complex machines. Keeping them separate adds an indirection layer so new versions of each stage can be deployed without impacting the other. It is thanks to this architecture that clang got a foot in the door by only providing compilation capabilities while leaving linking to GNU's ld. Likewise, the ELF specific gold linker and more recently LLVM's lld were drop-in replacement of GNU's ld.

Is cc a driver?

What is this cc CLI we saw in the introduction, mentioned in K&R? It was the name of the command to invoke the compiler driver back in the '70s. These days, cc is no more. But usage of cc was widespread and since it is a convenient indirection layer, it became part of POSIX and is still used nowadays.

$ which cc
/usr/bin/cc
$ readlink /usr/bin/cc
/etc/alternatives/cc
$ readlink /etc/alternatives/cc
/usr/bin/gcc

It's GNU gcc! What about the linker? What is /usr/bin/ld?

$ readlink /usr/bin/ld
x86_64-linux-gnu-ld
$ which x86_64-linux-gnu-ld
/usr/bin/x86_64-linux-gnu-ld

It is GNU's linker! Alternatively, we could have found out using the -v flag.

$ cc -v
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
$ ld -v
GNU ld (GNU Binutils for Ubuntu) 2.38

GNU Binary Utilities (binutils)

Throughout these articles, we will use tools to explore the inputs and outputs in the compiler driver pipeline. The set of CLIs we will rely on is called GNU's Binary Utilities, a.k.a binutils. This collection is the cornerstone of the free software world. It is used by countless platforms, from Linux to BSD, not forgetting Darwin, Playstation 1 Dev Kit, Playstation 3 and 4 OS.

// List of binutils tools

Name      |  Description
--------------------------------------------------------------
ld        |  The GNU linker.
as        |  The GNU assembler.
--------------------------------------------------------------
addr2line |  Converts addresses into filenames and line numbers.
ar        |  Creates, modifies and extracts from archives.
c++filt   |  Filter to demangle encoded C++ symbols.
dlltool   |  Creates files for building and using DLLs.
gold      |  New, faster, ELF only linker, 5x faster than ld.
gprof     |  Displays profiling information.
ldd       |  List libraries imported by object file.
nlmconv   |  Converts object code into an NLM.
nm        |  Lists symbols from object files.
objcopy   |  Copies and translates object files.
objdump   |  Displays information from object files.
ranlib    |  Generates an index to the contents of an archive.
readelf   |  Displays information from any ELF format object file.
size      |  Lists the section sizes of an object or archive file.
strings   |  Lists printable strings from files.
strip     |  Discards symbols.
windmc    |  Windows compatible message compiler.
windres   |  Compiler for Windows resource files. 

Mastering the usage of binutils is a wise investment of a programmer's time. Not only is the knowledge highly reusable across the aforementioned systems, these tools are often the building block of new languages, including recent ones like golang and rust.

Let's take the example of hello-world in golang.

package main
import "fmt"

func main() {
    fmt.Println("Hello world")
}

We can build it and find its entry address with readelf and its entry symbol with nm.

$ go build hello-world.go
$ ./hello-world 
hello world
$ readelf -h hello-world | grep Entry
  Entry point address:               0x6a680
$ nm hello-world | grep 6a680
000000000006a680 T _rt0_arm64_linux  	

We can also investigate at the dynamic libraries dependencies of hello-world and find out that go executable are statically linked.

$ ldd ./hello-world 
    not a dynamic executable

$ man 1 nm   // Show CLI     nm   documentation
$ man 2 read // Show syscall read documentation
$ man 3 getc // Show libc    getc documentation

$ man man
   1   Executable programs or shell commands
   2   System calls (functions provided by the kernel)
   3   Library calls (functions within program libraries)
   4   Special files (usually found in /dev)
   5   File formats and conventions, e.g. /etc/passwd
   6   Games

Driver flags vs Compiler flags vs Linker flags?

Since a programmer mostly interacts with the driver, flags and parameters must be routed to the appropriate component. It is a good idea to identify which elements are the intended target.

$ clang -v -lm -std=c89 hello.c
clang -cc1 -std=c89 -o hello.o hello.c
ld -lm -o a.out hello.o

In the previous trace, notice how -v is consumed by the compiler driver. It has no impact on the compiler or the linker. The driver detected that option -std=c89 was intended for the compiler and routed it automatically. Likewise, the driver forwarded -lm to the linker.

The driver will detect most commonly used parameters but you can also use a wrapper for a block to be blindly forwarded by the driver to the linker via -Wl.

$ clang -v -Wl,foo,bar hello.c
clang -cc1 -o hello.o hello.c
ld -o a.out hello.o foo bar

$ gcc -v hello.c
 cc1 hello.c -o /tmp/ccCKUy4c.s
 as -o /tmp/ccUkKMFH.o /tmp/ccCKUy4c.s
 collect2 /tmp/ccUkKMFH.o

Driving a multi-file project

Let's take a look at what happens when a project is made of multiple C files.

$ clang -v hello.c foo.cpp bar.m
clang -cc1 -c hello.c -o hello.o     // Compile
clang -cc1 -c foo.cpp -o foo.o       // Compile
clang -cc1 -c bar.m -o bar.o         // Compile

ld -o a.out hello.o foo.o bar.o    // Link

The driver turned three source files into three object files before linking them together into an executable. The verbose mode shows the compilation steps in sequence but it is important to understand they are in fact completely independent from each other. Looking at a dependency graph explains it better.

The driver ran all the steps sequentially but build systems leverage this translation unit isolation to drastically reduce their wall-time duration. They purposely avoid using the driver to spawn multiple compilers to turn source files into objects in parallel. These build systems also maintain a dependency graph to track which source files an object file depends on to re-compile only what has changed to dramatically speed up incremental compilation.

The linker bottleneck

The compilation stage scales linearly with the number of source files. The linking stage however does not. As you can see in the previous illustration, the linking stage depends on all object files which means that any file change, whether they are source or header will result in a full linking stage from scratch. Moreover, linking cannot be parallelized.

As a result, linkers are highly optimized. Apple, for example, leveraged multicores to improve the speed of their linker, ld64. Techniques such as Incremental linking, allowing to re-use the result from the previous linking operations, are supported by gold, lld, and Microsoft's linker LINK.EXE.

$ cat foo.h
int add(int x, int y);
int mul(int x, int y);	
$ clang foo.h -o foo.gch
$ file foo.gch
GCC precompiled header (version 013) for C 

clang++ and g++

Drivers are able to guess many things to make a programmer's life easier but they have their limits. Let's take a look at what happens when we build the C++ version of HelloWorld, hello.cc.

#include <iostream>

int main() {
    std::cout << "Hello World!";
    return 0;
}	

Let's compile it with the same compiler driver command we have used thus far.

$ clang -v hello.cc
clang -cc1 -o hello.o foo.cc
ld hello.o -o a.out
hello.cc:(.text+0x11): undefined reference to `std::cout'
.. 265 lines more lines of "Undefined symbols" 

It doesn't work. The compiler successfully generated an object but the linker failed to find the C++ symbols it needed to generate an executable. C++ source files require special care for which dedicated compiler drivers have been written. GNU GCC has g++ and LLVM has clang++.

$ clang++ -v hello.cc
clang -cc1 -I/usr/include/c++/10 -o hello.o foo.cc
ld hello.o -lc++ -o a.out
$ ./a.out
Hello World!

Language detection and name mangling

Typically, a driver sets up the header search path for the pre-processor and let the linker know where to find libraries. But there is much more. Let's take the example of a rainbow project using four languages (C, C++, Objective-C, and Objective-C++) and compile it. Since we are only interested in generating the object files, we request it to the driver via flag -c.

$ clang -v -c foo.c bar.cc toto.cpp baz.m qux.mm
clang -cc1 -o hello-f4.o -x c foo.c
clang -cc1 -o hello-ea.o -x c++ bar.cc
clang -cc1 -o hello-fa.o -x c++ too.cpp
clang -cc1 -o hello-a1.o -x objective-c baz.m
clang -cc1 -o hello-12.o -x objective-c++ qux.mm

Notice how the driver provides the language of the source file. Isn't it stating the obvious? Not really. Sometimes, a file must be compiled in a different language than its extension indicates. Compiling the same project with a C++ driver, clang++, shows the difference.

$$ clang++ -v -c foo.c bar.cc toto.cpp baz.m qux.mm
clang -cc1 -o hello-f4.o -x c++ foo.c
clang -cc1 -o hello-ea.o -x c++ bar.cc
clang -cc1 -o hello-fa.o -x c++ too.cpp
clang -cc1 -o hello-a1.o -x objective-c++ baz.m
clang -cc1 -o hello-12.o -x objective-c++ qux.mm

Notice how the driver requested from the compiler to treat the C source file (foo.c) and Objective-C file (baz.m) to be respectively interpreted as C++ and Objective-C++.

clang-cl.exe

Let's finish the part about compiler drivers with a clever one called clang-cl.exe.

In Microsoft world, Visual Studio IDE backend uses a compiler driver named CL.EXE which flags are incompatible with those used by LLVM's clang. Sometimes the flags differences are minimal e.g: To enable all warnings, /Wall is needed in CL.EXE while -Wall is used in clang. But most of the time, flags are completely different.

To allow Visual Studio to use clang as its backend, LLVM team created clang-cl.exe driver which converts Microsoft's CL.EXE flags into LLVM ones. In the following example, Visual Studio requested RTTI support (/GR-) and made char unsigned (/J). See how clang-cl.exe converted these flags into something clang -cc1 compiler could understand.

$ clang-cl.exe -v -c -o hello.o /GR- /J hello.c	
clang -frtti -funsigned-char -o hello.o  hello.c

Next

The Preprocessor (2/5)