Software configuration tuning¶
LLVM · Compiler flags
In this tutorial, we’ll use GraphFLA to explore how software configuration choices affect performance. To work through a Boolean search space, we’ll build a landscape from measurements of LLVM compiler flags and examine how their combinations affect compilation time.
Download notebook Notebook + data
Run the notebook in Google Colab, or download it to run locally. It downloads its data when no data/ folder is beside it.
1. Setting up¶
On Google Colab, the cell below installs GraphFLA. In any environment, it also downloads the dataset into a data/ folder next to the notebook if the files are not already there.
We then import GraphFLA for landscape construction and analysis, and pandas for working with the data.
import sys
from pathlib import Path
from urllib.request import urlretrieve
if "google.colab" in sys.modules:
%pip install -q graphfla==0.4.0
DATA_URL = "https://raw.githubusercontent.com/COLA-Laboratory/GraphFLA/v0.4.0/tutorials/datasets/data/"
Path("data").mkdir(exist_ok=True)
for name in ["llvm.csv"]:
if not Path("data", name).exists():
urlretrieve(DATA_URL + name, Path("data", name))
from graphfla import analysis
from graphfla.landscape import BooleanLandscape
import pandas as pd
2. Loading the dataset¶
A compiler can perform different analysis and optimisation steps while processing a program. Its configuration determines which steps run and can affect how long compilation takes. To explore these interactions, we’ll use the LLVM dataset from Siegmund et al. (2012), who measured every combination of ten optional flags on a fixed test-suite workload.
This gives 2¹⁰ = 1,024 configurations. We’ll minimise the recorded compilation-time response, retaining its original scale because the released table does not establish the unit. Each value aggregates repeated measurements.
| Columns | Meaning |
|---|---|
| Ten flag columns | 0 disables a flag; 1 enables it. |
compile_time_raw |
Original performance response; smaller is better. |
Let’s load the table. The mandatory time_passes option stays fixed, so it is excluded from the ten variables.
df = pd.read_csv("data/llvm.csv")
df.head()
Output
| gvn | instcombine | inline | jump_threading | simplifycfg | sccp | print_used_types | ipsccp | iv_users | licm | compile_time_raw | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 210.440000 |
| 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 200.660000 |
| 2 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 213.603333 |
| 3 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 248.583333 |
| 4 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 251.803333 |
3. Preparing the inputs¶
To construct a landscape, GraphFLA needs two aligned inputs:
X: one row per configuration and one column per variable.f: one measured or calculated outcome for each row ofX.
The ten independent on/off choices form X, and the performance response forms f. We keep the feature order explicit so that each bit can be mapped back to its compiler flag.
flag_columns = [
"gvn", "instcombine", "inline", "jump_threading", "simplifycfg",
"sccp", "print_used_types", "ipsccp", "iv_users", "licm",
]
X = df[flag_columns]
f = df["compile_time_raw"]
X.head()
Output
| gvn | instcombine | inline | jump_threading | simplifycfg | sccp | print_used_types | ipsccp | iv_users | licm | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 |
| 2 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 |
| 3 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 |
| 4 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 |
4. Constructing the landscape¶
We can use BooleanLandscape because each flag has two states. Neighbours toggle exactly one flag while leaving the other nine unchanged.
We set maximize=False because shorter compilation time is better. Improving edges therefore point towards smaller responses; the measured values themselves retain their original sign and scale.
landscape = BooleanLandscape(maximize=False)
landscape.build_from_data(X, f, neighborhood_strategy="active", verbose=False)
Output
BooleanLandscape(maximize=False)
5. Inspecting the landscape¶
Let’s first look at what we’ve built. Printing the landscape shows its variables, configurations, improving edges and local optima:
print(landscape)
Output
BooleanLandscape(kind='boolean'): 10 variables, 1024 configurations, 5118 edges, 7 local optima
For a closer look at individual compiler configurations, we can call get_data(). The outcome appears as fitness; out_degree counts directly improving moves, and is_lo indicates local-optimum membership.
landscape.get_data().head()
Output
| bit_0 | bit_1 | bit_2 | bit_3 | bit_4 | bit_5 | bit_6 | bit_7 | bit_8 | bit_9 | fitness | plateau_id | plateau_size | in_degree | out_degree | is_lo | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 210.440000 | -1 | 1 | 3 | 7 | False |
| 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 200.660000 | -1 | 1 | 10 | 0 | True |
| 2 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 213.603333 | -1 | 1 | 5 | 5 | False |
| 3 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 248.583333 | -1 | 1 | 6 | 4 | False |
| 4 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 251.803333 | -1 | 1 | 8 | 2 | False |
We can also summarise the performance responses with pandas:
landscape.get_data()["fitness"].describe()
Output
count 1024.000000
mean 236.893249
std 17.140479
min 199.683333
25% 223.326667
50% 236.921667
75% 250.503333
max 269.516667
Name: fitness, dtype: float64
To inspect the configuration with the lowest recorded response, we can use the global-optimum node index: The displayed bit_0 to bit_9 follow the order in flag_columns.
landscape[landscape.go_index]
Output
{'bit_0': np.int64(0),
'bit_1': np.int64(0),
'bit_2': np.int64(0),
'bit_3': np.int64(1),
'bit_4': np.int64(0),
'bit_5': np.int64(0),
'bit_6': np.int64(0),
'bit_7': np.int64(0),
'bit_8': np.int64(1),
'bit_9': np.int64(0),
'fitness': np.float64(199.683333333333),
'plateau_id': -1,
'plateau_size': 1,
'in_degree': 10,
'out_degree': 0,
'is_lo': True}
6. Analysing the landscape¶
Next, we’ll count local optima, examine local fitness similarity and interaction orders, and assess the trend towards the best observed configuration.
6.1 Number of local optima¶
We can start with landscape.n_lo. Each connected neutral optimum plateau counts once. A local optimum has no improving move out of it, including through its neutral plateau. Multiple optima mean successive improvements can end at different configurations.
print(f"Number of local optima: {landscape.n_lo}")
Output
Number of local optima: 7
6.2 Fitness autocorrelation¶
How similar are responses at neighbouring steps? autocorrelation() measures persistence along sampled walks. Higher values indicate more persistence under the same walk settings.
We use 200 walks of up to 20 visited configurations, lag one and a fixed seed. Walks traverse stored edges in either direction; separately stored neutral pairs are excluded.
walk_autocorrelation = analysis.autocorrelation(
landscape, walk_length=20, walk_times=200, lag=1, seed=42,
)
print(f"Lag-1 fitness autocorrelation: {walk_autocorrelation:.4f}")
Output
Lag-1 fitness autocorrelation: 0.6840
6.3 Walsh–Hadamard decomposition¶
Enabling a flag may have different effects depending on the other settings. walsh_hadamard() fits individual-flag contributions and pairwise interactions. The delta_r2 column shows how much training variation each order adds; higher-order interactions remain outside this fit.
Coefficients describe the original time response, even though we minimise it. Their positions follow flag_columns; their signs depend on the encoded contrast. The model-variance spectrum and cumulative training R² answer different questions.
wh = analysis.walsh_hadamard(landscape, max_order=2)
wh["order_summary"]
Output
| order | r2 | delta_r2 | rmse | n_terms | rank | alpha | model_variance_fraction | n_nonzero | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0.000000 | 0.000000 | 17.132108 | 1 | 1 | NaN | 0.000000 | <NA> |
| 1 | 1 | 0.779552 | 0.779552 | 8.043848 | 11 | 11 | NaN | 0.874667 | <NA> |
| 2 | 2 | 0.891255 | 0.111703 | 5.649558 | 56 | 56 | NaN | 0.125333 | <NA> |
We can also inspect the largest fitted nonconstant coefficients. Their units follow the response, and their signs depend on the encoded contrasts:
wh["coefficients"].query("order > 0").sort_values(
"coefficient", key=abs, ascending=False,
).head(8)
Output
| order | positions | term | coefficient | |
|---|---|---|---|---|
| 1 | 1 | (10,) | 0_10_1 | 16.882122 |
| 2 | 1 | (1,) | 0_1_1 | 16.059870 |
| 4 | 1 | (3,) | 0_3_1 | 14.818958 |
| 11 | 2 | (1, 10) | 0_1_1-0_10_1 | -13.334635 |
| 3 | 1 | (2,) | 0_2_1 | 12.028620 |
| 28 | 2 | (3, 10) | 0_3_1-0_10_1 | 11.156042 |
| 21 | 2 | (2, 3) | 0_2_1-0_3_1 | -8.082266 |
| 13 | 2 | (1, 3) | 0_1_1-0_3_1 | -7.792995 |
6.4 Fitness-distance correlation¶
Distance is the number of flags that differ from the selected best configuration. It measures configuration edits under the fixed workload.
We use Spearman fdc to compare ranks. A positive value means lower responses tend to occur nearer the optimum. A value near zero indicates little monotonic association. This trend does not guarantee an improving path from every configuration.
fitness_distance_r = analysis.fdc(landscape, method="spearman")
print(f"Fitness-distance correlation: {fitness_distance_r:.4f}")
Output
Fitness-distance correlation: 0.5779
Pairwise flag terms raise training R² from 0.780 to 0.891. Seven local optima remain, while positive FDC (0.578) indicates that configurations closer to the selected optimum tend to have smaller time responses.
7. Running several analyses together¶
We can collect autocorrelation and FDC with analysis.profile(), keeping the same walk settings. landscape.n_lo supplies the count directly; the W–H result keeps its separate coefficient and order-summary tables.
analysis.profile(
landscape,
metrics=["autocorrelation", "fdc"],
params={"autocorrelation": {"walk_length": 20, "walk_times": 200, "lag": 1}},
seed=42, progress=False,
)
Output
autocorrelation 0.684013
fdc 0.577932
dtype: float64
8. Analysis reference¶
For further exploration, analysis.list_metrics() lists the metrics available to profile(). The worked W–H result also exposes coefficients and fit_info; its order summary separates cumulative training fit from the fitted model’s variance spectrum.
Data sources¶
Siegmund N. et al. (2012). Predicting Performance via Automated Feature-Interaction Detection. We use the LLVM model and measurement archives from the authors’ project page, converting decimal commas to numerical values without rescaling.