Senior ML Engineer | Kimchi (Inference Stack Optimization)

<h4>Why Kimchi?</h4> <p>Great models are raw ingredients. Kimchi is what happens when you ferment them with good engineering. We started inside CAST AI helping companies run LLMs on their own Kubernetes clusters; today we run our own coding harness on any model you pick, and we run the models underneath it ourselves.<br><br><strong>Kimchi Inference</strong></p> <p>One API line and a customer&#39;s existing client just works: DeepSeek, GLM, Kimi, MiniMax, Nemotron served serverless from our own GPU fleet (OpenAI- and Anthropic-compatible, pay per token, spend visible per user, team, and key). When control matters, the same models run on the customer&#39;s own cluster instead. Behind that one line sits the machinery you&#39;ll own: SGLang and vLLM engines, B300 nodes, a shared cluster filesystem that holds every checkpoint exactly once, KV caches riding coding traffic with heavy prefix reuse, and the benchmark harness that gates every change we ship.<br><br></p> <p><em data-prosemirror-content-type=&quot;mark&quot; data-prosemirror-mark-name=&quot;em&quot;><span id=&quot;edb2bdea-40e7-48f0-9485-a0bd7169c7c7&quot; data-mark-type=&quot;annotation&quot; data-mark-annotation-type=&quot;inlineComment&quot; data-id=&quot;edb2bdea-40e7-48f0-9485-a0bd7169c7c7&quot; data-prosemirror-content-type=&quot;mark&quot; data-prosemirror-mark-name=&quot;annotation&quot;>Made by Cast AI — a Miami-headquartered, globally distributed team across 34 countries that hit unicorn status in January 2026 with a $50B+ backed investment pushing its valuation past $1 billion.</span></em></p> <h4>About the role</h4> <p>Throughput. Latency. Cold starts. Cost per token.</p> <p>Move those four numbers in the right direction and two things happen: customers get faster, cheaper inference, and our margins improve. That&#39;s the entire thesis of this role. Every loader you tune, every scheduler decision you land, every kernel you stare at shows up in a customer&#39;s p99 and on our P&amp;L. Your mandate is the inference stack&#39;s optimization, whatever it takes, and the technical direction of how we optimize it is yours to argue with data.</p> <p><strong>The problem:</strong> the checkpoints run 0.3-1.5 TB, and the &quot;right&quot; serving configuration depends on traffic shape, sequence-length distribution, batch dynamics, GPU SKU, memory bandwidth, and quantization tolerance, all variables that shift week to week. Most teams pick a configuration once, over-provision GPUs, and absorb the cost. We don&#39;t: we took Kimi-K3 (1.5 TB) from over an hour cold-to-serving to under twelve minutes with a shared filesystem, O_DIRECT weight loaders, and node-local JIT caches, and we&#39;re nowhere near done. Prefill/decode disaggregation and tiered KV caches are next.</p> <p><strong>Stack:</strong> Python; SGLang; vLLM; TensorRT-LLM; PyTorch; fastsafetensors; CUDA-adjacent tooling; Kubernetes; virtiofs shared filesystems; ClickHouse; OpenTelemetry; Prometheus; Grafana; GitLab CI.</p> <h4>Requirements</h4> <ul> <li>You&#39;ve built real ML systems for 5+ years, and your portfolio shows depth in inference or training infrastructure, not just model training notebooks.</li> <li>Your Python is production-grade: services, not scripts.</li> <li>You&#39;ve run vLLM, SGLang, or TensorRT-LLM for real, and you have a working mental model of why an engine performs the way it does on a given GPU.</li> <li>You&#39;ve measured quantization regressions, not just compression ratios.</li> <li>Multi-GPU and multi-node setups hold no fear: tensor and expert parallelism, collective communication, and their practical failure modes.</li> <li>You know where the bytes live (page caches, O_DIRECT, network filesystems under terabyte-scale reads) and how fast they move.</li> <li>You instrument before you optimize, and you can tell a real win from a benchmark artifact.</li> <li>A wide mandate excites you more than it unsettles you.</li> </ul> <h4>Nice to haves</h4> <ul> <li>You&#39;ve worked on LLM routing specifically: RouterBench-class evals, cost/quality/latency Pareto fronts.</li> <li>Calibration, abstention, or conformal methods in production decision systems.</li> <li>You&#39;ve run online A/B loops with drift monitoring on a model that touches real money.</li> </ul> <h4>Responsibilities</h4> <ul> <li>Push throughput. Continuous batching, chunked prefill, speculative decoding with draft models, CUDA-graph capture, and scheduler tuning across the engine<br>&nbsp;fleet. Find the ceiling on each GPU SKU, then raise it.</li> <li>Cut latency. Attack TTFT and inter-token latency separately; profile to find the actual bottleneck (compute, memory bandwidth, scheduling, storage, or<br>&nbsp;network) and fix it, not the one you assumed.</li> <li>Get more out of the KV cache. Hierarchical and paged attention, prefix caching on heavy-reuse coding traffic, eviction policies, quantized KV. A lot of the<br>&nbsp;unrealized throughput lives here, and you&#39;ll own it.</li> <li>Kill cold starts. Weight loaders (O_DIRECT, fastsafetensors-class), JIT and compile caches, shared-filesystem read paths. A 1.5 TB checkpoint has no business<br>&nbsp;taking an hour to reach serving.</li> <li>Quantize without regressing quality. INT8/INT4, FP8/FP4 across weights, activations, and KV, measured on real workloads, not just perplexity benchmarks.</li> <li>Scale across nodes. Distributed inference topologies, network-aware placement, prefill/decode disaggregation, checkpointing strategies that don&#39;t bottleneck<br>&nbsp;on storage or interconnect.</li> <li>Set the technical direction. Decide what we benchmark, what we adopt, and what we build ourselves. Bring the team along with strong writeups and reproducible<br>&nbsp;experiments.</li> </ul> <h4>What’s in it for you?</h4> <ul> <li>Competitive salary (depending on the level of experience).</li> <li>Enjoy a flexible, remote-first global environment.</li> <li>Collaborate with a global team of cloud experts and innovators, passionate a…