kernels

ForgeKernel Kernels

GPU kernels generated by ForgeKernel on NVIDIA H100 and B200

English | 中文

This directory releases the operator (kernel) code produced by ForgeKernel in the paper ForgeKernel: Self-Evolving Kernels Catching and Surpassing Experts and Agents (under review at ICLR 2027), plus a kernel ForgeKernel generated after the paper was submitted, marked "after the paper" below.

ForgeKernel is a self-evolving kernel agent. It reads a reference kernel's architecture from hardware behavior alone (running, timing and profiling a sealed binary, never its source) and turns it into an architecture-first curriculum that a coding agent implements. Every accepted kernel can be sealed and promoted to be the reference of the next run, so optimization continues past the best known implementation. The paper's kernels are written in CuTe DSL (nvidia-cutlass-dsl); the post-paper kernel is CUDA C++. None of them was produced with access to the source code of the library or system it is compared against.

Layout

Kernels are grouped by operator type, then by operator and GPU:

kernels/
├── attention/          dense attention: forward, backward, paged decode/prefill
├── sparse_attention/   DeepSeek sparse attention (DSA) and the DeepSeek-V4.1 fused prefill
├── linear_attention/   Gated DeltaNet and Kimi Delta Attention
├── gemm/               dense, FP8 and FP8 × FP4 grouped GEMM
└── conv/               2-D and 3-D convolution

Each leaf directory holds one kernel (kernel.py, plus its CUDA sources for the post-paper kernel).

Kernels

Positive numbers mean the ForgeKernel kernel is faster. "Source" is where the number comes from: paper Figure 3 (against expert libraries), Figure 4 (against other kernel-generation systems), Table 1 (self-evolution by reference promotion), or after the paper.

Attention

Operator	GPU	Compared against	Speedup	Source	Directory
Attention forward, causal, GQA 16:1, gen 0	H100	FlashAttention-4	+0.3%	Figure 3	attention/attn_fwd_gqa16/h100/gen0
Attention forward, causal, GQA 16:1, gen 1	H100	ForgeKernel gen 0 (sealed)	+9.9% (+10% vs FlashAttention-4)	Table 1	attention/attn_fwd_gqa16/h100/gen1
Attention forward, causal, GQA 16:1, gen 2	H100	ForgeKernel gen 1 (sealed)	+4.2% (+15% vs FlashAttention-4)	Table 1	attention/attn_fwd_gqa16/h100/gen2
Attention forward, causal, GQA 16:1	H100	/goal agent kernel (sealed)	+5.0% (+29% vs FlashAttention-4)	Table 1	attention/attn_fwd_gqa16/h100/from_agent
Attention backward, causal, MHA	H100	FlashAttention-4	+5.1%	Figure 3	attention/attn_bwd_mha/h100
Attention backward, causal, MHA	B200	FlashAttention-4	+2.5%	Figure 3	attention/attn_bwd_mha/b200
Attention backward, causal, GQA 8:1	H100	FlashAttention-4	+2.7%	Figure 3	attention/attn_bwd_gqa8/h100
GQA paged decode	H100	K-Search	+24.5%	Figure 4	attention/gqa_paged_decode/h100
MLA paged decode	H100	K-Search	+170.6%	Figure 4	attention/mla_paged_decode/h100
MLA paged prefill	H100	K-Search	+146.1%	Figure 4	attention/mla_paged_prefill/h100

Sparse attention

Operator	GPU	Compared against	Speedup	Source	Directory
DSA sparse attention backward	H100	cuDNN-frontend	+1.4%	Figure 3	sparse_attention/dsa_sparse_bwd/h100
DSA score recompute	H100	cuDNN-frontend	+6.2%	Figure 3	sparse_attention/dsa_score_recompute/h100
DSA indexer backward	H100	cuDNN-frontend	+2.5%	Figure 3	sparse_attention/dsa_indexer_bwd/h100
DeepSeek-V4.1 fused prefill: Q RoPE + sparse MQA attention with sink + O RoPE + FP8 cast	B200	FlashMLA	1.104–1.105×	After the paper	sparse_attention/dsv41_fused_attn_rope_cast/b200

Linear attention

Operator	GPU	Compared against	Speedup	Source	Directory
Gated DeltaNet prefill forward, gen 0	H100	FlashInfer	−0.2%	Figure 3	linear_attention/gdn_prefill_fwd/h100/gen0
Gated DeltaNet prefill forward	H100	/goal agent kernel (sealed)	+9.5% (+100% vs FlashInfer)	Table 1	linear_attention/gdn_prefill_fwd/h100/from_agent
Gated DeltaNet prefill backward	H100	FlashQLA	+1.1%	Figure 3	linear_attention/gdn_prefill_bwd/h100
Gated DeltaNet prefill, varlen	B200	Kernel Design Agents (KDA-v0.5)	+5.8%	Figure 4	linear_attention/gdn_prefill_varlen/b200
Recurrent Kimi Delta Attention prefill	B200	CAKE	+0.4%	Figure 4	linear_attention/recurrent_kda_prefill/b200

GEMM

Operator	GPU	Compared against	Speedup	Source	Directory
Dense GEMM, 16-bit	H100	cuBLAS	+0.9%	Figure 3	gemm/gemm_fp16/h100
FP8 GEMM, 128-block scaling	H100	DeepGEMM	+0.8%	Figure 3	gemm/fp8_gemm/h100
FP8 × FP4 grouped GEMM (MoE)	B200	DeepGEMM	+0.6%	Figure 3	gemm/fp4_grouped_gemm/b200

Convolution

Operator	GPU	Compared against	Speedup	Source	Directory
Conv2d weight gradient	H100	cuDNN	+1.0%	Figure 3	conv/conv2d_wgrad/h100
Conv2d data gradient, stride 2	H100	cuDNN	+0.7%	Figure 3	conv/conv2d_dgrad/h100
Conv3d forward	H100	cuDNN	+7.6%	Figure 3	conv/conv3d_fwd/h100

Generations

A Table 1 kernel was produced with the best known kernel sealed as its reference; its speedup is over that reference, and the number in parentheses is cumulative over the expert library. Where an operator has several kernels on one GPU, they sit in subdirectories:

• gen0 is ForgeKernel's first kernel, produced against the expert library (Figure 3).
• gen1 and gen2 are successive promotions: each was produced with the previous generation sealed as its reference. Attention forward is the only chain the paper promoted twice in succession, so it is the only one with gen1 and gen2.
• from_agent comes from a single promotion whose reference was written by the /goal agent, not by ForgeKernel, so that reference is not part of this directory.

Environment

GPU	CUDA	PyTorch	nvidia-cutlass-dsl
H100 SXM (sm_90a)	12.8 or 13.1	2.6 or 2.10	4.4.2, 4.6.0.dev0 or 4.6.1
B200 (sm_100a)	13.0	2.13	4.6.0 or 4.6.2 (not used by the CUDA C++ kernel)

Times are device time (median over pipelined back-to-back launches); several H100 measurements used SM clocks locked at 1590 MHz.

License

Apache License 2.0; see LICENSE and NOTICE. The CuTe DSL kernels use NVIDIA CUTLASS CuTe DSL (nvidia-cutlass-dsl), which is not included and is distributed under its own license.