Staff Engineer – Cloud Data Store (Visibility)
Temporal is an open-source programming model that can simplify code, make applications more reliable, and help developers focus on the important things like delivering features faster. We are on a mission to be the reliable foundation of every developer's toolbox, and are building the team that will make that happen.
Our values guide us — they are present in how we show up, make decisions, and work together to make an impact. We're curious, driven, collaborative, genuine, and humble.
Temporal is growing and we are looking for those who share our values, challenge "standard" thinking, and want to influence our future. If you have a passion for improving the developer experience, building world-class open-source software and communities, and want to be part of our amazing team, we'd love to hear from you!
⸻
Summary
Cloud Data Store (CDS) owns the storage, retrieval, and lifecycle of all workflow data at planet scale. We design the persistence APIs, build storage abstractions that run across cloud vendors, and deliver the observability that lets customers trust their state machines for years.
[To see more detail re: the Temporal CDS Eng team, click here [new window]
A core part of CDS is Temporal Visibility: the system that powers workflow listing, filtering, search, and observability across massive, long-lived state machines. Visibility has demanding characteristics — high write throughput, complex secondary indexing, low-latency queries, strict correctness guarantees, and the need to operate continuously while schemas and backends evolve.
As a Staff Engineer on CDS, you will take a leading role in a ground-up rewrite of the Visibility persistence layer, with the goal of making it dramatically more performant, scalable, and operable. This includes evaluating and driving deep integrations with modern analytical and search data stores such as ClickHouse and Elasticsearch, with a strong preference for self-hosted operation (while supporting managed offerings where appropriate).
You will design, build, and operate significant portions of our backend for highly scalable, multi-tenant services, and own large technical initiatives end-to-end — from initial design through live data migration, rollout, and long-term operational stewardship.
⸻
What You'll Do
Re-architect Temporal Visibility at scale
• Lead the design and implementation of a new persistence layer for Temporal Visibility, informed by its real-world access patterns (high-volume writes, time-based queries, filtering, sorting, and pagination across long-running workflows).
• Evaluate and select the most appropriate storage technologies (e.g., ClickHouse, Elasticsearch, or complementary systems), clearly articulating tradeoffs around indexing models, consistency, cost, latency, and operational complexity.
• Design schemas, APIs, and query models that make Visibility both powerful and intuitive for customers.
Deliver safe, large-scale data migrations
• Plan and execute online migrations of live Visibility data from existing persistence stores to new backends, at scale and without customer downtime.
• Design dual-write, backfill, validation, and cutover strategies that prioritize correctness, observability, and rollback safety.
• Build tooling and automation to validate data integrity and performance throughout the migration lifecycle.
Own performance, reliability, and operability
• Define and own SLOs for Visibility storage and query paths.
• Profile hot paths, design benchmarks, and lead systematic performance tuning efforts.
• Build operational playbooks, dashboards, and alerting that make the system understandable and debuggable for on-call engineers.
• Lead incident reviews and reliability improvements related to persistence and indexing systems.
Technical leadership and collaboration
• Break down large, ambiguous roadmap initiatives into concrete, executable phases.
• Author and steward design docs and RFCs through review with peers and stakeholders.
• Mentor and unblock other engineers working in the persistence and storage domain.
• Partner closely with Server, Cloud, and Developer Experience teams to land features end-to-end.
⸻
What You'll Bring
- 5 or more years of experience as an "Arranger" and/or "Builder/Enhancer" of highly scalable distributed systems. see HERE for more info re: "Arranger" and/or "Builder/Enhancer".
- Strong computer science fundamentals in distributed systems, including concurrency, consistency models, and failure modes
- Significant experience writing and operating concurrent production systems in Go, Java, or similar languages, at a high-end intermediate to expert level
- Experience writing concurrent code in production with languages like Go or Java or other applicable languages with skill level as "high end of Intermediate" and/or "Advanced" or "Expert" levels. see HERE for more info re: "high end of Intermediate" and/or "Advanced" or "Expert levels"
- Hands-on experience designing, operating, and tuning ClickHouse and/or Elasticsearch, ideally in self-hosted environments (managed services are a strong plus).
- Experience building and running services on AWS. Bonus: Azure and/or GCP experience.
- Demonstrated ability to lead large, multi-quarter technical initiatives, especially those involving core data infrastructure and live data migrations.
⸻
Nice to Have
• Prior contributions to Temporal, Cadence, or other workflow engines.
• Deep expertise in storage internals (e.g., columnar stores, LSM trees, inverted indexes, transactional logs).
• Experience operating multi-region services with ≥99.99% uptime.
• Strong background in operating and evolving Open Source systems.
• Experience building Kubernetes controllers and/or CRDs.
⸻
- The estimated pay range for this role is $180,000 - 260,000.
- Additionally, this role is eligible to participate in Temporal's equity plan.
- Unlimited PTO, 12 Holidays + 2 Floating Holidays
- 100% Premiums Coverage for Medical, Dental, and Vision
- AD&D, LT & ST Disability, and Life Insurance (Standard & Supplemental Available)
- Empower 401K Plan
- Additional Perks for Learning & Development, Lifestyle Spending, In-Home Office Setup, Professional Memberships, WFH Meals, Internet Stipend and more!
Paid Time Off (PTO) and Benefits outside the United States vary by country, and are issued in partnership with Remote.com. Additionally, Temporal offers perks to all international employees for learning & career development, a lifestyle spending account, in-home office setup (in addition to company-issued hardware), professional memberships, work-from-home meals, and access to the Calm app for mental wellness.
Travel
Temporal is a globally distributed, collaborative team that values opportunities for in-person connection. Occasional travel may be required for company events, team offsites, and other meaningful moments that bring us together.
- $3,600 / Year Work from Home Meals
- $1,800 / Year Professional Enrichment (Career Development & Professional Memberships)
- $1,200 / Year Lifestyle Spending Account
- $1,000 / Year In-Home Office Setup (In addition to Temporal issued equipment - laptop, monitor, keyboard, mouse, trackpad, and extension power cable at no cost to you)
- $74 / Month Reimbursement for Internet
- Calm App Subscription for Mental Health & Wellness
Top Skills
What We Do
Temporal develops and distributes the world's leading open source durable execution system. We make code fault tolerant, durable and simple.
Innovative companies like Datadog, Glovo, Indeed, Netflix, Qualtrics, Remitly, Snap and Yum! Brands build their services and applications with Temporal to make them reliable to run, productive to enhance and easy to troubleshoot and repair.
More than a decade in the making, Temporal is powered by veterans behind some of the industry's most loved systems technologies, programming frameworks and open source communities as well as investors like Amplify Partners, Sequoia Capital and Index Ventures.
