Methodology¶
Measurements are produced with ./bench, a small nanoBench-style tool for
LoongArch.
./bench assembles the given instruction sequence, unrolls it, warms it up,
and executes it in a tight loop, counting retired uops and core cycles with
perf_event_open(2).
Removing measurement overhead¶
Every measurement pays a fixed cost that does not belong to the instructions
under test: the call itself, the register-save prologue and epilogue, the
caller's loop branch, and the perf_event_open enable/disable and counter
reads. Subtracting a separately measured empty function is not a valid way
to remove this cost, because the empty function cannot overlap its prologue
with any work while a real body can, so the empty baseline always overstates
the real per-call cost.
./bench therefore defaults to the slope method: it compiles the same
sequence twice with different unroll factors and differences the two
measurements.
C(k) = O + k * c O = all per-call and per-measurement cost
c = (C(k2) - C(k1)) / (k2 - k1)
O cancels exactly, because both measurements execute the same number of
calls and the same number of perf enable/disable cycles. --unroll N sets the
smaller body and --unroll2 N2 the larger one (default 2N). The reported
value is normalised by iters * (N2 - N).
This matters most at small unroll factors. For a single dependent add.d
(1 cycle) the raw body measurement reads 0.89 at --unroll 100 while the slope
method reads 0.98; for a fadd.d chain (3 cycles) the raw value is 2.23 at
--unroll 100 and the slope method gives 3.00.
--method raw reports a single body measurement unchanged. It includes the
per-call overhead and is only useful as an upper bound or for comparing runs
made with identical settings. Earlier data in this repository was collected
with the old empty-function subtraction, so its recorded command lines do not
show --method; the values have since been normalised to nominal integers.
Latency is measured with dependency chains: the instruction under test is followed by chain instructions whose latency is known, and the chain latency is subtracted from the measured total. Throughput is measured with independent lanes.
Each raw measurement is stored as JSON under docs/data/<microarchitecture>/.
Every experiment records:
- the assembly sequence
- the init sequence
- the exact command line
- the measured counter and cycle counts
- latency operand pair, or throughput lanes
Latency entries use operand indices from the operands list. For example,
from_operands: [2] and to_operand: 1 means the latency from operand 2 to
operand 1.
The JSON format is defined by schema/instruction.schema.json.
Custom performance counters¶
./bench can also read raw LoongArch PMU counter configs with --counter.
The option is repeatable; at most two extra counters can be added because the
PMU has four counters total (instructions, cycles, and two extras).
./bench --unroll 1000 --iters 1000 --counter 0xfb --counter 0xa8 \
--init 'li.d $r4, 1; vreplgr2vr.b $vr0, $r4' \
--asm 'vadd.b $vr0, $vr0, $vr0'
Each extra counter is printed as:
counter 0xfb: 1.00
counter 0xa8: 0.00
Counts are normalised per execution of the unrolled sequence and have the
empty-function baseline subtracted, the same way as instructions retired and
cycles.
Common raw counters (LA464 and LA664)¶
These configs have the same meaning on both microarchitectures:
| Config | Meaning |
|---|---|
0x01 |
Retired uops (PERF_COUNT_HW_INSTRUCTIONS) |
0xa8 |
Load operations (not uops) |
0xb0 |
Store operations (not uops) |
0x01 is a uop count, not an architectural instruction count. On LA664, for
example, vpickve2gr.w retires as two uops and shows up as 2.00. To obtain
architectural instruction counts, count the assembly instructions statically
and compare them with 0x01.
0xa8 and 0xb0 are memory-operation counters, not uop counters. On LA664,
atomics make the distinction clear: amadd.w retires as two uops, but
contributes one load and one store. llacq.w contributes one load, and sc.q
contributes one store, even though each is a two-uop instruction.
LA664-specific raw counters¶
The following vendor events are specific to LA664 (Loongson-3C6000):
| Config | Meaning |
|---|---|
0xfb |
LSX vector uops |
0xfa |
LASX vector uops |
0xfc |
Scalar floating-point uops |
These are outside the public Linux event map. The meanings are inferred from controlled instruction-mix experiments.
LA464-specific raw counters¶
The LA464 (Loongson-3C5000) PMU uses a different raw event map. The LA664
configs 0xfb, 0xfa, and 0xfc read zero on LA464; use the following
mappings instead:
| Config | Meaning |
|---|---|
0x30 |
Scalar FP arithmetic/conversion uops |
0x25 |
Vector ALU/conversion instructions (LSX and LASX combined) |
0x31 |
Vector ALU/conversion uops (LSX and LASX combined) |
On LA464, 0x25 counts each LSX/LASX ALU or conversion instruction once,
while 0x31 counts the corresponding vector uops. For example, vadd.b,
vand.v, vpickve2gr.w, and xvpickve2gr.w give 0x25 = 1 and
0x31 = 1, but a two-uop conversion such as vfcvt.h.s, vftint.w.d,
vffint.s.l, or their LASX forms gives 0x25 = 1 and 0x31 = 2.
Vector loads/stores and some vector moves are not included: vld, vst,
xvld, xvst, and vreplgr2vr.w all read zero from 0x25 and 0x31.
One scalar exception was observed: movfr2cf also increments 0x25 and
0x31 by one. A scan of configs 0x00-0xff did not find separate LSX and
LASX uop counters on LA464, so 0x31 is the combined vector-uop counter.
Scalar floating-point examples for 0x30: fadd.d = 1, fdiv.d = 1,
fsel = 2, and fcmp.slt.d = 1. Most FP moves, FP CSR transfers, and FP
loads/stores do not count: movgr2fr.d, movfr2gr.d, movfrh2gr.s,
movcf2gr, movgr2cf, movfcsr2gr, movgr2fcsr, fld.d, and fst.d
all read zero. The exceptions are movgr2frh.w = 1 and movcf2fr = 1,
while movfr2cf = 0. Integer instructions count as zero.
Unaligned vector memory accesses¶
On LA464, both the load and store counters split when a vector access crosses a 64-byte cache line. With a 64-byte-aligned base:
- for a 16-byte LSX access (
vld/vst),0xa8/0xb0report 2 at offsets 49-63 and 113-127, and 1 otherwise; - for a 32-byte LASX access (
xvld/xvst),0xa8/0xb0report 2 at offsets 33-63 and 97-127, and 1 otherwise; 0x01stays at 1 per instruction even when the memory counter reports 2.
On LA664, the load/store counters were measured with a 64-byte-aligned base
and unaligned offsets. 0xa8 counts a load twice when the access crosses a
64-byte cache line: for a 16-byte LSX load, offsets such as 49, 56, 63, and
127; for a 32-byte LASX load, offsets such as 33 through 63 and 97 through
127. Loads entirely within one 64-byte line count once. 0xb0 counted one
store for every tested LSX and LASX store offset, including cache-line
crossings, so no store split was visible in that counter on LA664. The
instruction-retired count (0x01) remained one per instruction even when
0xa8 reported two load operations.
The load-splitting behavior therefore matches between LA464 and LA664, but the store-splitting behavior does not: the LA464 store counter splits on a 64-byte-line crossing, while no LA664 store split was observed.
Multi-uop instructions¶
On both microarchitectures, 0x01 counts retired uops, so a multi-uop
instruction has a counter value larger than its static instruction count. The
details of how to separate instructions and uops differ between LA464 and
LA664.
LA664¶
In the current LA664 data set, 112 of 1270 instructions with throughput data are two-uop instructions:
| Category | Count |
|---|---|
Atomic am* |
52 |
vfcvt / vffint / vftint conversions, LSX and LASX |
48 |
vpickve2gr.b/h/w/d, xvpickve2gr.w/d |
6 |
llacq.d, llacq.w, sc.q |
3 |
vreplgr2vr.b/h |
2 |
movgr2frh.w |
1 |
For these instructions the throughput experiment also records the relevant
extra counters in result.counters:
- atomics:
0xa8(one load) and0xb0(one store), even though the atomic retires as two uops; llacq.*:0xa8(one load);sc.q:0xb0(one store);vfcvt/vffint/vftintLSX:0xfb(two LSX uops);- the same LASX conversions:
0xfa(two LASX uops); vpickve2gr.*:0xfb(two LSX uops);xvpickve2gr.w/d:0xfa(two LASX uops);vreplgr2vr.b/h:0xfb(one LSX uop);movgr2frh.w:0xfc(one scalar floating-point uop).
The extra counter values are normalised per instruction (the measured per-sequence values divided by the eight lanes). They are shown in the per-instruction throughput table and in the interactive table details.
LA464¶
In the current LA464 data set, 41 of 348 instructions with throughput data are two-uop instructions:
| Category | Count |
|---|---|
Atomic am* |
36 |
fsel |
1 |
movgr2frh.w |
1 |
rdtime.d, rdtimeh.w, rdtimel.w |
3 |
Unlike LA664, LA464 has a vector-instruction counter (0x25) as well as a
vector-uop counter (0x31), so the vector relationship can be measured
directly:
vector uops per instruction = 0x31 / 0x25
For example, vadd.b gives 0x25 = 1 and 0x31 = 1, while a two-uop vector
conversion gives 0x25 = 1 and 0x31 = 2. LA464 does not expose separate LSX
and LASX uop counters; 0x31 is the combined LSX+LASX vector-uop count. On
LA464, the 0x31 / 0x25 ratio is therefore the direct way to identify
multi-uop vector instructions.
CPUCFG and trapped instructions¶
On LA664, cpucfg is a special case for the instruction-retired counter. It is
trapped and emulated: in a 10M-instruction cpucfg loop, instructions:u is only about
0.13M while the all-mode instructions count is about 31M, so roughly 30M
retired instructions happen in the kernel. bench excludes kernel events, so
cpucfg reports instructions retired: 0.00. The validator therefore uses a
retired multiplier of zero for cpucfg; its latency and throughput are still
meaningful from the cycle counter.