Soon we'll run 100B models on cheap hardware
Because ternary (1.58-bit) models use weights that are either -1, 0, or 1, massive hardware optimizations become possible. This means multiplication between layers is eliminated, replaced by just addition, subtraction, and memory lookups. Furthermore, the activation function can be pre-computed and stored in a 16-bit look-up table (LUT) that takes up just 64KB.
To make things simple, here is what the data journey looks like for a single neuron. Let's say the neuron receives an INT8 input with a value of 42, and its weight is -1. The multiplication step is now trivial because the operation can only produce three possible values: -42, 0, or 42. In our case, the result is -42. The circuit can be massively simplified.
Next comes the accumulation. If we assume the neuron has a bias of 12, we pass this through a simple addition circuit, which is vastly smaller and more power-efficient than a multiplication circuit. Our accumulated value becomes -30.
Finally, we apply the activation function. Instead of performing crazy expensive floating-point math, we just use -30 as an index address to check our 64KB LUT. If the precomputed value at that address is 0, we simply fetch it from the L1 cache, which is incredibly fast and cheap. Our final output is 0.
Because of these massive silicon shortcuts, I estimate that a dedicated ASIC with highly optimized on-board memory could run a 100B parameter model at around 35W.
Likewise a 7B model could run entirely on a dedicated ASIC's internal SRAM (L3 cache, possibly 3D stacked), and it could potentially run at under 1W.