Senior Software Engineer, Reliability

Posted 7 Hours Ago
Be an Early Applicant
Hiring Remotely in United States
Remote
141K-231K Annually
Senior level
Software
The Role
Own end-to-end reliability for Temporal Cloud by building an AI-powered Reliability Agent, improving incident response, defining SLOs and operational readiness practices, running gamedays and chaos testing, and leading load, performance, capacity, and observability initiatives. The role also includes post-incident reviews, reliability scorecards, cross-team collaboration, mentoring, and on-call participation.
Summary Generated by Built In
Role Summary

Join the Reliability team as a Senior Software Engineer, reporting to Tushar Roy, and help improve the reliability of operating Temporal Cloud end to end. You’ll build an AI-first reliability experience, including building a Reliability Agent from scratch that helps the SRE team triage common issues, surface useful context, and recommend next steps. You’ll also help make reliability knowledge and practices accessible across engineering so ownership is shared rather than concentrated with a small number of experts.

 
What You’ll Do
  • Own reliability outcomes for operating Temporal Cloud end to end, partnering across engineering, infrastructure, and product to drive measurable improvements.

  • Build and evolve the Reliability Agent, including workflows that gather context from telemetry, incidents, runbooks, deployments, and service history to support faster, more consistent triage.

  • Design safe, useful agent experiences for handling routine reliability questions and issues, with appropriate human oversight, escalation paths, auditability, and safeguards against incorrect actions.

  • Strengthen incident response for Temporal Cloud, from detection and triage through escalation, mitigation, and communication, and lead post-incident reviews that turn into durable corrective actions.

  • Define and evolve reliability practices, including service-level objectives, alerting thresholds, operational readiness criteria, and escalation paths.

  • Plan and run gamedays and safely expand chaos testing to validate failure response, evaluate agent-assisted operations, and reduce operational risk.

  • Build and maintain reliability scorecards that make service health, operational readiness, agent effectiveness, and improvement priorities visible.

  • Lead load, performance, and capacity testing, and improve observability standards across services.

 
What You’ll Bring
  • Strong computer science fundamentals, especially in distributed systems, concurrency, performance, and fault-tolerant design.

  • Experience designing, building, or operating complex systems under high load and partial failure.

  • Experience building AI-powered tools, agents, or production automation, with an understanding of tool use, evaluation, state, failure modes, and human-in-the-loop design.

  • A track record of driving reliability improvements across multiple services, systems, or engineering teams.

  • Hands-on experience with incident response and post-incident reviews, including leading incidents and following corrective actions through to completion.

  • Hands-on experience with one or more of gamedays, chaos testing, load testing, capacity planning, reliability scorecards, or observability programs.

  • Sound judgment in ambiguous situations, with the ability to balance reliability, agent autonomy, safety, customer impact, and delivery speed.

  • Clear communication skills and the ability to align stakeholders, share operational knowledge, mentor others, and participate in an on-call rotation. This is a remote United States role with no regular travel expected.

 
Nice to Haves
  • Experience operating multi-tenant systems and designing protections against noisy-neighbor behavior.

  • Experience building internal platforms or tooling that enables teams to meet reliability standards.

  • Familiarity with LLM observability, agent evaluation, retrieval-augmented generation, or workflow orchestration systems.

  • Open source contributions in infrastructure, distributed systems, AI, or reliability engineering.

Temporal Technologies is an Equal Opportunity Employer. Temporal Technologies does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status, or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need. We embrace and celebrate differences and diversity.

Temporal is committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, its services, programs, and activities. If you need to request a reasonable accommodation, please let your Recruiter know so we can assist.

Skills Required

  • Strong computer science fundamentals in distributed systems, concurrency, performance, and fault-tolerant design
  • Experience designing, building, or operating complex systems under high load and partial failure
  • Experience building AI-powered tools, agents, or production automation
  • Understanding of tool use, evaluation, state, failure modes, and human-in-the-loop design
  • Track record of driving reliability improvements across multiple services, systems, or engineering teams
  • Hands-on incident response and post-incident review experience, including leading incidents and completing corrective actions
  • Experience with gamedays, chaos testing, load testing, capacity planning, reliability scorecards, or observability programs
  • Sound judgment in ambiguous situations and ability to balance reliability, agent autonomy, safety, customer impact, and delivery speed
  • Clear communication, stakeholder alignment, operational knowledge sharing, mentoring, and participation in an on-call rotation
  • Experience operating multi-tenant systems and protecting against noisy-neighbor behavior
  • Experience building internal platforms or tooling that enables teams to meet reliability standards
  • Familiarity with LLM observability, agent evaluation, retrieval-augmented generation, or workflow orchestration systems
  • Open source contributions in infrastructure, distributed systems, AI, or reliability engineering

Temporal Technologies Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Temporal Technologies and has not been reviewed or approved by Temporal Technologies.

  • Healthcare Strength — Healthcare coverage is described as 100% employer-paid for medical, dental, and vision, with AD&D, short- and long-term disability, and life insurance included. Feedback suggests this breadth and cost coverage is a strong differentiator for a remote-first employer.
  • Leave & Time Off Breadth — Time off includes unlimited PTO alongside 12 standard holidays and 2 floating holidays. Feedback suggests this structure supports rest and recharge across teams.
  • Wellbeing & Lifestyle Benefits — Wellbeing and remote-work support include a home office stipend, internet reimbursement, WFH meals, Calm app access, a lifestyle spending account, and learning/professional membership budgets. Feedback suggests these perks enhance overall total rewards beyond base pay.

Temporal Technologies Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Bellevue, Washington
501 Employees
Year Founded: 2019

What We Do

Temporal develops and distributes the world's leading open source durable execution system. We make code fault tolerant, durable and simple. Innovative companies like Datadog, Glovo, Indeed, Netflix, Qualtrics, Remitly, Snap and Yum! Brands build their services and applications with Temporal to make them reliable to run, productive to enhance and easy to troubleshoot and repair. More than a decade in the making, Temporal is powered by veterans behind some of the industry's most loved systems technologies, programming frameworks and open source communities as well as investors like Amplify Partners, Sequoia Capital and Index Ventures.

Similar Jobs

Samsara Logo Samsara

Artificial Intelligence Engineer

Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Easy Apply
Remote or Hybrid
United States
4000 Employees
162K-290K Annually

MongoDB Logo MongoDB

Senior Software Engineer

Big Data • Cloud • Software • Database
Easy Apply
Remote or Hybrid
10 Locations
5550 Employees
126K-248K Annually

Freenome Logo Freenome

Senior Software Engineer

Healthtech • Biotech
Remote
USA
372 Employees
131K-190K Annually

NVIDIA Logo NVIDIA

Senior Software Engineer

Artificial Intelligence • Computer Vision • Hardware • Robotics • Metaverse
In-Office or Remote
2 Locations
21960 Employees
184K-288K Annually

Similar Companies Hiring

Ford Energy Thumbnail
Automotive • Software • Energy • Utilities • Manufacturing • Renewable Energy
US
Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account