Triton Partners

TileLang for High-Performance GPU Kernels

This technical tutorial introduces TileLang, a Python-first DSL for writing high-performance GPU kernels that balances the high-level convenience of Triton with the low-level control of CUTLASS/CuTe. TileLang exposes tiles as first-class objects, requires explicit buffer placement (shared memory, registers), and relies on a layout-inference pass to derive thread mappings and memory layouts. The article walks through a GEMM example, an MLA (Multi‑Head Latent Attention) decode kernel (DeepSeek) where TileLang's layout inference enables a compact ~80-line implementation matching FlashMLA H100 fp16 performance, and a production RMSNorm+SiLU drop-in used at AtlasCloud that expands supported channel widths and improved latency. The post details primitives (T.alloc_shared, T.gemm, T.Pipelined, etc.), backend targets, and one-line optimization knobs like swizzling and warp policies.

Fixing Slow PyTorch Training on High-End GPUs

A Dev.to technical guide explains why PyTorch training can show low GPU utilization even on powerful cards (e.g., NVIDIA A100) and provides a step-by-step workflow to diagnose and fix performance issues. The author categorizes slow workloads into three regimes—compute-bound, memory-bandwidth-bound, and overhead-bound—and recommends profiling with torch.profiler to identify the regime. For memory-bound workloads the post advocates operator fusion (via torch.compile) or custom kernels (Triton, FlashAttention); for overhead-bound cases it recommends CUDA Graphs and static input buffers. The article also gives practical prevention tips: profile early, avoid dynamic shapes where possible, minimize .cpu()/.item() syncs, and monitor utilization (e.g., nvidia-smi).