teknav-bak/global/article-contents-en-2.js

export const ART_EN2_CONTENT = `
<div class="article-content">
  <p class="lead">The GPU's dominance in AI is real but narrower than its reputation suggests. For inference at scale — the workload that actually drives AI cloud bills — custom silicon from Google, Apple, Amazon, and a growing list of startups is consistently beating the H100 on cost-per-token and latency, and the architectural reasons why are worth understanding before your next infrastructure decision.</p>

  <figure style="margin: 0 0 28px 0; border-radius: 10px; overflow: hidden; border: 1px solid #E4DDD2;">
    <img src="/images/articles/custom-ai-chip.jpg" alt="Custom AI inference chips: TPU, NPU, and specialized silicon architecture" loading="lazy" style="width: 100%; height: auto; display: block;" />
    <figcaption style="font-size: 12px; color: #5F6B6D; padding: 10px 14px; background: #F4EFE6; line-height: 1.7;">Specialized AI accelerators — from Google's TPU v5e to Apple's Neural Engine — are purpose-built for the matrix operations that dominate transformer inference, offering fundamentally different memory and compute trade-offs than general-purpose GPU architectures.</figcaption>
  </figure>

  <h2>1. Why Inference Is a Different Problem Than Training</h2>
  <p>Training a neural network is a batch workload: you process large chunks of data in parallel, update weights via backpropagation, and repeat. Throughput is the dominant metric, and the NVIDIA A100/H100 excel here — 312 TFLOPS of FP16 on the H100 SXM5, with 80GB of HBM2e and 2TB/s of memory bandwidth. This combination is hard to beat for dense matrix multiplication at training scale.</p>
  <p>Inference is a fundamentally different workload. For autoregressive LLMs, each new token is generated sequentially — you cannot parallelize across the sequence dimension. The critical bottleneck shifts from compute (FLOPS) to <strong>memory bandwidth</strong>: loading the model weights from DRAM into compute units for every single token generation step. An H100 with 2TB/s memory bandwidth generates tokens at roughly 50–100 tokens/second for a 70B parameter model — not because the compute is exhausted, but because the model weights take time to shuttle from memory to compute.</p>
  <p>This is the <em>memory bandwidth bottleneck</em>, and it is the architectural gap that custom silicon exploits. When your forward pass is memory-bound rather than compute-bound, adding more FLOPS does nothing. What you need is either more memory bandwidth, or techniques to reduce the number of weight loads per token.</p>

  <h2>2. Google TPU v5: The Systolic Array Advantage</h2>
  <p>Google's <strong>TPU v5e</strong> and <strong>TPU v5p</strong> (announced late 2023) are the most mature custom AI chips in production at scale. The architecture centers on systolic arrays — a grid of multiply-accumulate units arranged so that data flows through the grid without being stored in registers between operations. This eliminates memory traffic that a general GPU architecture would incur for intermediate values.</p>
  <p>TPU v5e delivers 393 TFLOPS of bfloat16 performance per chip, with 1,600 GB/s of HBM memory bandwidth. More importantly, the chips are designed to scale: a TPU v5e pod of 256 chips achieves 100 petaFLOPS of aggregate compute with a 1.6 Tbps inter-chip interconnect fabric. Google reports that TPU v5e reaches 2x better cost-performance than TPU v4 for inference workloads on Gemma-class models.</p>
  <p>For latency-sensitive inference (batch size 1, interactive chatbots), the TPU's strength is its deterministic latency profile. Because the hardware is purpose-built without the general-purpose CUDA runtime overhead, tail latencies are more predictable than equivalent GPU deployments. Google's own Gemini API runs on TPU infrastructure — the choice is not theoretical.</p>

  <h2>3. AWS Trainium and Inferentia: Vertical Integration at Cloud Scale</h2>
  <p>Amazon's <strong>Trainium2</strong> (2024) and <strong>Inferentia2</strong> sit at different points in the training/inference spectrum. Inferentia2 is explicitly designed for low-latency inference and offers 190 TOPS of compute per chip with 384GB/s of memory bandwidth per chip — modest on raw numbers but dramatically cheaper per unit than Nvidia equivalents on AWS pricing. AWS reports Inferentia2 delivers up to 4x higher throughput and 10x lower latency versus comparable GPU instances for common model sizes.</p>
  <p>The practical implication: Llama 3 70B running on an <code>inf2.48xlarge</code> instance (12 Inferentia2 chips) costs roughly $12.98/hour on-demand. The equivalent H100 GPU instance (<code>p4de.24xlarge</code>) costs $37.23/hour. For sustained inference workloads where throughput matters more than time-to-first-token, the cost math favors Inferentia by a wide margin. The constraint is tooling: Inferentia requires the <em>AWS Neuron SDK</em>, which adds compilation steps and is still less mature than CUDA for exotic model architectures.</p>

  <blockquote style="border-left: 4px solid #D49A2A; margin: 28px 0; padding: 16px 20px; background: #D49A2A0A; border-radius: 0 8px 8px 0; font-size: 16px; color: #263238; font-style: normal;">"The H100 is the best general-purpose AI chip. But general-purpose means you're paying for capability you don't use. Inference at scale demands specialization." — Annapurna Labs engineering blog, 2024</blockquote>

  <h2>4. Apple Neural Engine: On-Device Inference Rewritten</h2>
  <p>Apple's <strong>Neural Engine</strong> — integrated into every Apple Silicon chip since the A11 Bionic (2017) and significantly upgraded in M4 (2024) — takes a different architectural approach driven by extreme power constraints. The M4's Neural Engine delivers 38 TOPS at under 15W thermal envelope. For context, an H100 delivers roughly 1979 TOPS but draws 700W.</p>
  <p>The raw TOPS comparison is misleading because Apple's use case is fundamentally different: running quantized, pruned models for on-device inference where the alternative is no AI at all (not a cloud GPU). The Neural Engine's significance is that it enables inference on 7–13 billion parameter models locally on a MacBook Pro — with Apple claiming the M4 Pro can run a 70B parameter model at approximately 30 tokens/second via <em>MLX framework</em> with 4-bit quantization.</p>
  <p>On-device inference is not just a hardware story. The unified memory architecture of Apple Silicon means the CPU, GPU, and Neural Engine share the same memory pool. A 13B model at 4-bit quantization occupies roughly 7GB — which fits in the 24GB or 36GB configurations. The memory bandwidth on M4 Max reaches 400GB/s, which is competitive with discrete GPU boards for inference-class workloads.</p>

  <h2>5. NPU vs GPU: The Right Question Is the Workload</h2>
  <p>Neural Processing Units (<strong>NPUs</strong>) are fixed-function accelerators optimized for the specific operation patterns of neural network inference: matrix-vector products, convolutions, activation functions. They do these operations efficiently and power-efficiently, but they cannot be reprogrammed to run Monte Carlo simulations or ray tracing workloads. GPUs are programmable — their CUDA cores can execute arbitrary parallel code, which is why they dominate research where model architectures change frequently.</p>
  <p>The practical decision tree: if you are running a stable, deployed model architecture in production and your primary metric is cost-per-inference-call, custom silicon (TPU, Inferentia, or an NPU-class chip) will almost always win on unit economics once the engineering investment in the toolchain is amortized. If you are doing active research, fine-tuning, or rapid prototyping where the model architecture changes weekly, GPU flexibility is worth the premium.</p>
  <p>Qualcomm's <strong>Snapdragon X Elite</strong> NPU (45 TOPS, 2024 laptops) and <strong>Snapdragon 8 Gen 3</strong> (on-device mobile, 45 TOPS) represent the edge inference category — designed to run Llama 3 8B or Phi-3 mini locally on Android phones and Windows laptops without any cloud dependency. This is the hardware that makes edge AI viable rather than aspirational.</p>

  <div class="insight-box" style="padding: 14px; border-radius: 10px; background: #0F6B7311; border: 1px solid #0F6B7340; margin-bottom: 20px;">
  <strong>Memory Bandwidth Math:</strong> To estimate whether your inference workload is memory-bound or compute-bound, calculate the arithmetic intensity: total floating-point operations divided by total bytes moved from memory. For a 70B parameter model generating one token at a time, arithmetic intensity is approximately 2 (ops/byte) — well below the "roofline" threshold where memory bandwidth becomes the limit. This is why increasing FLOPS via a larger GPU does not improve token generation speed for these workloads.
  </div>

  <h2>6. The Startups Targeting the Gap</h2>
  <p>A cohort of specialized inference chip startups have raised significant funding targeting the gap between GPU generality and the specific demands of transformer inference. <strong>Groq</strong>'s Language Processing Units (LPUs) use a deterministic execution model — no caches, no speculation, no dynamic scheduling — to achieve extremely low latency: the company reports 500+ tokens/second on Llama 3 70B, roughly 5–10x faster than comparable GPU deployments for interactive latency. The trade-off is that the LPU's fixed dataflow is difficult to adapt to novel architectures.</p>
  <p><strong>Cerebras</strong> takes the opposite approach to memory bandwidth: instead of high-bandwidth DRAM, the CS-3 system integrates 900,000 cores and 44GB of on-chip SRAM at terabytes-per-second of bandwidth. The chip is physically enormous — a full silicon wafer, not a die — and the elimination of off-chip memory transfers for model weights enables performance that GPU clusters cannot match for certain workloads.</p>

  <div class="footnote-box" style="padding: 14px; border-radius: 10px; background: #FAF7F0; border: 1px solid #E4DDD2; font-size: 12px; margin-bottom: 20px;">Sources: Google, "TPU v5e and TPU v5p Product Documentation" (Google Cloud, 2024). AWS, "Amazon EC2 Inf2 Instances" technical overview (Amazon Web Services, 2024). Apple, "M4 chip technical specifications" (Apple Newsroom, 2024). Horace He, "Making Deep Learning Go Brrrr From First Principles" (horace.io, 2022) — foundational analysis of memory bandwidth vs compute bottlenecks.</div>

  <h2>Conclusion: Specialize for the Workload You Actually Have</h2>
  <p>The GPU will remain the right choice for research, training, and heterogeneous AI infrastructure for the foreseeable future. But the inference workload that constitutes the majority of production AI cost is memory-bandwidth-bound, not compute-bound — and that distinction is precisely where purpose-built silicon wins. Before defaulting to another H100 cluster, benchmark your specific model on TPU v5e, Inferentia2, or a Groq endpoint: the economics of AI inference are changing faster than procurement cycles, and the teams that recognize this first will hold a real cost advantage.</p>
</div>
`;