# Highway implementation details

## Introduction

This doc explains some of the Highway implementation details; understanding them
is mainly useful for extending the library. Bear in mind that Highway is a thin
wrapper over 'intrinsic functions' provided by the compiler.

## Vectors vs. tags

The key to understanding Highway is to differentiate between vectors and
zero-sized tag arguments. The former store actual data and are mapped by the
compiler to vector registers. The latter (`Simd<>` and `SizeTag<>`) are only
used to select among the various overloads of functions such as `Set`. This
allows Highway to use built in vector types without a class wrapper.

Class wrappers are problematic for SVE and RVV because LLVM (or at least Clang)
does not allow member variables whose type is 'sizeless' (in particular,
built-in vectors). To our knowledge, Highway is the only C++ vector library that
supports SVE and RISC-V without compiler flags that indicate what the runtime
vector length will be. Such flags allow the compiler to convert the previously
sizeless vectors to known-size vector types, which can then be wrapped in
classes, but this only makes sense for use-cases where the exact hardware is
known and rarely changes (e.g. supercomputers). By contrast, Highway can run on
unknown hardware such as heterogeneous clouds or client devices without
requiring a recompile, nor multiple binaries.

Note that Highway does use class wrappers where possible, in particular NEON,
WASM and x86. The wrappers (e.g. Vec128) are in fact required on some platforms
(x86 and perhaps WASM) because Highway assumes the vector arguments passed e.g.
to `Add` provide sufficient type information to identify the appropriate
intrinsic. By contrast, x86's loosely typed `__m128i` built-in type could
actually refer to any integer lane type. Because some targets use wrappers and
others do not, incorrect user code may compile on some platforms but not others.
This is because passing class wrappers as arguments triggers argument-dependent
lookup, which would find the `Add` function even without namespace qualifiers
because it resides in the same namespace as the wrapper. Correct user code
qualifies each call to a Highway op, e.g. with a namespace alias `hn`, so
`hn::Add`. This works for both wrappers and built-in vector types.

## Adding a new target

Adding a target requires updating about ten locations: adding a macro constant
to identify it, hooking it into static and dynamic dispatch, detecting support
at runtime, and identifying the target name. The easiest and safest way to do
this is to search for one of the target identifiers such as `HWY_AVX3_DL`, and
add corresponding logic for your new target. Note the upper limits on the number
of targets per platform imposed by `HWY_MAX_DYNAMIC_TARGETS`.

## When to use -inl.h

By convention, files whose name ends with `-inl.h` contain vector code in the
form of inlined function templates. In order to support the multiple compilation
required for dynamic dispatch on platforms which provide several targets, such
files generally begin with a 'per-target include guard' of the form:

```
#if defined(HWY_PATH_NAME_INL_H_) == defined(HWY_TARGET_TOGGLE)
#ifdef HWY_PATH_NAME_INL_H_
#undef HWY_PATH_NAME_INL_H_
#else
#define HWY_PATH_NAME_INL_H_
#endif
// contents to include once per target
#endif  // HWY_PATH_NAME_INL_H_
```

This toggles the include guard between defined and undefined, which is
sufficient to 'reset' the include guard when beginning a new 'compilation pass'
for the next target. This is accomplished by simply re-#including the user's
translation unit, which may in turn `#include` one or more `-inl.h` files. As an
exception, `hwy/ops/*-inl.h` do not require include guards because they are all
included from highway.h, which takes care of this in a single location. Note
that platforms such as RISC-V which currently only offer a single target do not
require multiple compilation, but the same mechanism is used without actually
re-#including. For both of those platforms, it is possible that additional
targets will later be added, which means this mechanism will then be required.

Instead of a -inl.h file, you can also use a normal .cc/.h component, where the
vector code is hidden inside the .cc file, and the header only declares a normal
non-template function whose implementation does `HWY_DYNAMIC_DISPATCH` into the
vector code. For an example of this, see
[vqsort.cc](../hwy/contrib/sort/vqsort.cc).

Considerations for choosing between these alternatives are similar to those for
regular headers. Inlining and thus `-inl.h` makes sense for short functions, or
when the function must support many input types and is defined as a template.
Conversely, non-inline `.cc` files make sense when the function is very long
(such that call overhead does not matter), and/or is only required for a small
set of input types. [Math functions](../hwy/contrib/math/math-inl.h)
can fall into either case, hence we provide both inline functions and `Call*`
wrappers.

## Use of macros

Highway ops are implemented for up to 12 lane types, which can make for
considerable repetition - even more so for RISC-V, which can have seven times as
many variants (one per LMUL in `[1/8, 8]`). The various backends
(implementations of one or more targets) differ in their strategies for handling
this, in increasing order of macro complexity:

*   `x86_*` and `wasm_*` simply write out all the overloads, which is
    straightforward but results in 4K-6K line files.

*   [arm_sve-inl.h](../hwy/ops/arm_sve-inl.h) defines 'type list'
    macros `HWY_SVE_FOREACH*` to define all overloads for most ops in a single
    line. Such an approach makes sense because SVE ops are quite orthogonal
    (i.e. generally defined for all types and consistent).

*   [arm_neon-inl.h](../hwy/ops/arm_neon-inl.h) also uses type list
    macros, but with a more general 'function builder' which helps to define
    custom function templates required for 'unusual' ops such as `ShiftLeft`.

*   [rvv-inl.h](../hwy/ops/rvv-inl.h) has the most complex system
    because it deals with both type lists and LMUL, plus support for widening or
    narrowing operations. The type lists thus have additional arguments, and
    there are also additional lists for LMUL which can be extended or truncated.

## Code reuse across targets

The set of Highway ops is carefully chosen such that most of them map to a
single platform-specific intrinsic. However, there are some important functions
such as `AESRound` which may require emulation, and are non-trivial enough that
we don't want to copy them into each target's implementation. Instead, we
implement such functions in
[generic_ops-inl.h](../hwy/ops/generic_ops-inl.h), which is included
into every backend. To allow some targets to override these functions, we use
the same per-target include guard mechanism, e.g. `HWY_NATIVE_AES`.

The functions there are typically templated on the vector and/or tag types. This
is necessary because the vector type depends on the target. Although `Vec128` is
available on most targets, `HWY_SCALAR`, `HWY_RVV` and `HWY_SVE*` lack this
type. To enable specialized overloads (e.g. only for signed integers), we use
the `HWY_IF` SFINAE helpers. Example: `template <class V, class D = DFromV<V>,
HWY_IF_SIGNED_D(D)>`. Note that there is a limited set of `HWY_IF` that work
directly with vectors, identified by their `_V` suffix. However, the functions
likely use a `D` type anyway, thus it is convenient to obtain one in the
template arguments and also use that for `HWY_IF_*_D`.

For x86, we also avoid some duplication by implementing only once the functions
which are shared between all targets. They reside in
[x86_128-inl.h](../hwy/ops/x86_128-inl.h) and are also templated on the
vector type.

## Adding a new op

Adding an op consists of three steps, listed below. As an example, consider
https://github.com/google/highway/commit/6c285d64ae50e0f48866072ed3a476fc12df5ab6.

1) Document the new op in `g3doc/quick_reference.md` with its function signature
and a description of what the op does.

2) Implement the op in each `ops/*-inl.h` header. There are two exceptions,
detailed in the previous section: first, `generic_ops-inl.h` is not changed in
the common case where the op has a unique definition for every target. Second,
if the op's definition would be duplicated in `x86_256-inl.h` and
`x86_512-inl.h`, it may be expressed as a template in `x86_128-inl.h` with a
`class V` template argument, e.g. `TableLookupBytesOr0`.

3) Pick the appropriate `hwy/tests/*_test.cc` and add a test. This is also a
three step process: first define a functor that implements the test logic (e.g.
`TestPlusMinus`), then a function (e.g. `TestAllPlusMinus`) that invokes this
functor for all lane types the op supports, and finally a line near the end of
the file that invokes the function for all targets:
`HWY_EXPORT_AND_TEST_P(HwyArithmeticTest, TestAllPlusMinus);`. Note the naming
convention that the function has the same name as the functor except for the
`TestAll` prefix.

## Reducing the number of overloads via templates

Most ops are supported for many types. Often it is possible to reuse the same
implementation. When this works for every possible type, we simply use a
template. C++ provides several mechanisms for constraining the types:

*   We can extend templates with SFINAE. Highway provides some internal-only
    `HWY_IF_*` macros for this, e.g. `template <typename T, HWY_IF_FLOAT(T)>
    bool IsFiniteT(T t) {`. Variants of these with `_D` and `_V` suffixes exist
    for when the argument is a tag or vector type. Although convenient and
    fairly readable, this style sometimes encounters limits in compiler support,
    especially with older MSVC.

*   When the implementation is lengthy and only a few types are supported, it
    can make sense to move the implementation into namespace detail and provide
    one non-template overload for each type; each calls the implementation.

*   When the implementation only depends on the size in bits of the lane type
    (instead of whether it is signed/float), we sometimes add overloads with an
    additional `SizeTag` argument to namespace detail, and call those from the
    user-visible template. This may avoid compiler limitations relating to the
    otherwise equivalent `HWY_IF_T_SIZE(T, 1)`.

## Deducing the Simd<> argument type

For functions that take a `d` argument such as `Load`, we usually deduce it as a
`class D` template argument rather than deducing the individual `T`, `N`,
`kPow2` arguments to `Simd`. To obtain `T` e.g. for the pointer argument to
`Load`, use `TFromD<D>`. Rather than `N`, e.g. for stack-allocated arrays on
targets where `!HWY_HAVE_SCALABLE`, use `MaxLanes(d)`, or where no `d` lvalue is
available, `HWY_MAX_LANES_D(D)`.

When there are constraints, such as "only enable when the `D` is exactly 128
bits", be careful not to use `Full128<T>` as the function argument type, because
this will not match `Simd<T, 8 / sizeof(T), 1>`, i.e. twice a half-vector.
Instead use `HWY_IF_V_SIZE_D(D, 16)`.

We could perhaps skip the `HWY_IF_V_SIZE_D` if fixed-size vector or mask
arguments are present, because they already have the same effect of overload
resolution. For example, when the arguments are `Vec256` the overload defined in
x86_256 will be selected. However, also verifying the `D` matches the other
arguments helps prevent erroneous or questionable code from compiling. For
example, passing a different `D` to `Store` than the one used to create the
vector argument might point to an error; both should match.

For functions that accept multiple vector types (these are mainly in x86_128,
and avoid duplicating those functions in x86_256 and x86_512), we use
`VFrom<D>`.

## Documentation of platform-specific intrinsics

When adding a new op, it is often necessary to consult the reference for each
platform's intrinsics.

For x86 targets `HWY_SSE2`, `HWY_SSSE3`, `HWY_SSE4`, `HWY_AVX2`, `HWY_AVX3`,
`HWY_AVX3_DL`, `HWY_AVX3_ZEN4`, `HWY_AVX3_SPR` Intel provides a
[searchable reference](https://www.intel.com/content/www/us/en/docs/intrinsics-guide).

For Arm targets `HWY_NEON`, `HWY_NEON_WITHOUT_AES`, `HWY_NEON_BF16`, `HWY_SVE`
(plus its specialization for 256-bit vectors `HWY_SVE_256`), `HWY_SVE2` (plus
its specialization for 128-bit vectors `HWY_SVE2_128`), Arm provides a
[searchable reference](https://developer.arm.com/architectures/instruction-sets/intrinsics).

For RISC-V target `HWY_RVV`, we refer to the assembly language
[specification](https://github.com/riscv/riscv-v-spec/blob/master/v-spec.adoc)
plus the separate
[intrinsics specification](https://github.com/riscv-non-isa/rvv-intrinsic-doc).

For WebAssembly target `HWY_WASM`, we recommend consulting the
[intrinsics header](https://github.com/llvm/llvm-project/blob/main/clang/lib/Headers/wasm_simd128.h).
There is also an unofficial
[searchable list of intrinsics](https://nemequ.github.io/waspr/intrinsics).

For POWER targets `HWY_PPC8`, `HWY_PPC9`, `HWY_PPC10`, there is
[documentation of intrinsics](https://files.openpower.foundation/s/9nRDmJgfjM8MpR7),
the [ISA](https://files.openpower.foundation/s/dAYSdGzTfW4j2r2), plus a
[searchable reference](https://www.ibm.com/docs/en/openxl-c-and-cpp-aix/17.1.1?).

For ZVector targets `HWY_Z14`, `HWY_Z15`, `HWY_Z16`, there is the
[ISA](https://www.ibm.com/support/pages/zarchitecture-principles-operation)
(requires IBMid login), plus a
[searchable reference](https://www.ibm.com/docs/en/zos/2.5.0?topic=topics-using-vector-programming-support).

For LoongArch, there is a
[list of intrinsics](https://jia.je/unofficial-loongarch-intrinsics-guide/lsx/integer_computation/)
and
[ISA reference](https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html)
and
[summary of highlights](http://0x80.pl/notesen/2025-01-21-loongarch64-highlights.html).

## Why scalar target

There can be various reasons to avoid using vector intrinsics:

*   The current CPU may not support any instruction sets generated by Highway
    (on x86, we only target S-SSE3 or newer because its predecessor SSE3 was
    introduced in 2004 and it seems unlikely that many users will want to
    support such old CPUs);
*   The compiler may crash or emit incorrect code for certain intrinsics or
    instruction sets;
*   We may want to estimate the speedup from the vector implementation compared
    to scalar code.

Highway provides either the `HWY_SCALAR` or the `HWY_EMU128` target for such
use-cases. Both implement ops using standard C++ instead of intrinsics. They
differ in the vector size: the former always uses single-lane vectors and thus
cannot implement ops such as `AESRound` or `TableLookupBytes`. The latter
guarantees 16-byte vectors are available like all other Highway targets, and
supports all ops. Both of these alternatives are slower than native vector code,
but they allow testing your code even when actual vectors are unavailable.

One of the above targets is used if the CPU does not support any actual SIMD
target. To avoid compiling any intrinsics, define `HWY_COMPILE_ONLY_EMU128`.

`HWY_SCALAR` is only enabled/used `#ifdef HWY_COMPILE_ONLY_SCALAR` (or `#if
HWY_BROKEN_EMU128`). Projects that intend to use it may require `#if HWY_TARGET
!= HWY_SCALAR` around the ops it does not support to prevent compile errors.