Level: Staff (MTS 4) · Reports to: Chief AI Officer, later the Senior Engineering Manager
The role. Make our own platform dependable and scalable, working with the Principal Engineer. This is reliability engineering for Horizon AI’s software rather than for the GPU fleet, though the two meet when our agents are the ones being trusted with fleet actions.
You will
Build the caching layer that stops every agent tool call from hitting external systems directly
Implement event streaming so services stop depending on synchronous HTTP chains
Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation
Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on
Contribute to the orchestration migration and to infrastructure-as-code
Improve deployment practice: canary, traffic shifting, and real rollback
You have
Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency
Comfort in Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration
Experience with event streaming and with the failure modes each option brings - Observability depth: metrics, distributed tracing, and the judgment to alert on symptoms rather than causes
A record of shipping reliability improvements with measurable before and after
Nice to have: durable workflow orchestration, blue/green and canary deployment at scale, performance profiling of Python services
Skills Required
- Production experience making distributed systems reliable, including caching, queuing, circuit breaking, backpressure, timeouts, and idempotency
- Proficiency with Python
- Experience working in containerized environments
- Real-world experience with Kubernetes or equivalent orchestration
- Experience with event streaming and its associated failure modes
- Strong observability experience with metrics and distributed tracing
- Ability to distinguish symptoms from causes when creating alerts
- Demonstrated record of shipping reliability improvements with measurable before-and-after results
- Experience with durable workflow orchestration
- Experience with blue/green and canary deployment at scale
- Performance profiling experience with Python services
What We Do
STN, Inc. is a managed technology and infrastructure provider serving enterprise, regulated, and AI-driven organizations. It designs, operates, and supports secure, scalable systems, including managed IT, cloud and platform services, cybersecurity, data management, compliance engineering, enterprise hardware and software, and GPU One, its GPU-as-a-Service platform for AI training, tuning, inference, and other high-performance workloads. STN emphasizes reliability, audit readiness, and ongoing human support.
