About the Role
TLDR: We are looking for an ML Infrastructure Engineer to build the systems behind our LLM post-training, RL, evaluation, inference, and agentic development workflows. You will work close to researchers, GPUs, training loops, data control systems, evals, inference stacks, and the infrastructure decisions that directly affect model learning and product quality.About usWhite Circle is an AI Safety company building the safety, reliability, and optimization layer for AI systems. At the core of our platform are policies – simple natural-language rules that define what an AI model should and shouldn’t do. We automatically test, enforce, and continuously improve these policies at scale.We’ve raised $11M from top funds, founders, and senior leaders at OpenAI, Anthropic, HuggingFace, Mistral, DeepMind, Datadog, Sentry, and othersWe process over 100M+ API calls every monthWe fine-tune and train our own LLMs so they run faster and cheaper than any open or proprietary modelWe’re a small, highly focused team. If you want to work deeply on hard problems, see your work ship to production quickly, and influence how AI safety is actually built – you’re the one we need.You will:Build robust, flexible, and scalable RL and post-training pipelines, including smoke tuning runs for quality testing and approach ablationsDesign data control systems that govern what the model sees, when it sees it, and how training data flows through rollouts, replay, filtering, evaluation, and policy updatesTune training and inference end-to-end for high throughput across the systems that matter: networking, memory, compute scheduling, data loading, storage, checkpointing, and I/OInvestigate how infrastructure choices affect learning dynamics, eval quality, model behavior, and training stability – staying close to the state of the art in LLMs, RL, and post-trainingBuild infrastructure for model iteration: experiment runs, artifacts, evals, dashboards, failure inspection, reproducibility, and cost visibilityWork on inference infrastructure where it affects post-training and evaluation loopsBuild and improve agentic development environments: coding-agent harnesses, browser/tool integrations, terminal/runtime sandboxes, repo-aware workflows, and multi-agent orchestrationWork closely with the team: plan future steps, discuss tradeoffs, share context early, and stay in touch while buildingYou’ll fit right in if you:Have designed, built, or maintained distributed RL/post-training systems at scale and are fluent in their moving parts: rollouts, replay buffers, reward signals, data filtering, policy updates, evaluation loops, and failure analysisAre familiar with deep learning frameworks such as PyTorch or JAXAre proficient in Python, including concurrency, asynchronous programming, multiprocessing, and performance optimizationCan debug distributed GPU workloads across CUDA runtime, container runtime, driver versions, NCCL or equivalent communication layers, networking, storage, scheduling, and checkpointingHave experience with profiling tools across the stack, for example py-spy, PyTorch profiler, Nsight, perf, tracing, metrics, logs, or custom instrumentationHave experience with inference stacks such as vLLM, SGLang, TensorRT-LLM, Dynamo, or custom serving infrastructureCan reason from system metrics back to model behavior: when latency, queueing, sampling, data order, rollout throughput, or infrastructure failures affect learningHave a strong ownership mindset: you can take an ambiguous infrastructure problem, make it concrete, ship a working system, and improve it from real feedback