RTB - Tracking compile time and compiler sizes for LLVM and GCC

Compile-time and compiler size for many compilers are growing at an alarming rate. The situation is particularly complicated for C++ compilers due to the rapid evolution and relative complexity of the language. As each C++ revision adds new features to both the language and the standard library, the compile time and compiler size increase. In addition, most (if not all) compilers are increasingly heterogeneous, supporting multiple target architectures; in many cases a single source file can use both CPUs and GPUs (e.g. OpenMP target, SYCL, etc.).

To quantify these performance trends, we have created a visualisation site called Race To Binary (RTB) The site downloads weekly snapshots of both GCC and LLVM compilers and compiles a series of test programs to track compile-time and compiler sizes over time:

https://uob-hpc.github.io/rtb/

The visualisation on the site is interactive: you can navigate the chart and select data points to find out which commits were made in that specific snapshot. RTB’s URLs are stable; for example, here are the compiler performance and size results for LLVM12-15 and GCC11-13 for miniBUDE OpenMP:

• https://uob-hpc.github.io/rtb/#/dataset/node33/perf?versions=gcc-11%5Egcc-12%5Egcc-13%5Ellvm-12%5Ellvm-13%5Ellvm-14%5Ellvm-15&jobs=bude-omp&scale=2.00&group=user
• https://uob-hpc.github.io/rtb/#/dataset/node33/size?versions=gcc-11%5Egcc-12%5Egcc-13%5Ellvm-12%5Ellvm-13%5Ellvm-14%5Ellvm-15&jobs=bude-omp&scale=2.00&group=user

This project is partially inspired by similar work in the LLVM community:

• https://www.duskborn.com/posts/llvm-15-speed/
• https://www.npopov.com/2020/05/10/Make-LLVM-fast-again.html

The style of visualisation is similar to the “Are we * yet” websites.

Snapshots

The RTB project requires pre-built compiler binaries, however we were unable to find existing resources that host old snapshots. To resolve this, we created a snapshot-building pipeline using only GitHub Actions and Pages:

https://uob-hpc.github.io/compiler-snapshots/

This site compiles weekly snapshots for both LLVM and GCC and publishes the binaries as GitHub releases. We then run a Page action to deploy a static site so that specific snapshots can be easily located.

Similar work

If you find this work interesting, we’ve done similar research on SYCL’s runtime performance, see https://dl.acm.org/doi/10.1145/3456669.3456701. For compile time, an interesting study from another group looked at SYCL’s compile time: https://dl.acm.org/doi/10.1145/3529538.3529548.