About Subzero Labs
Subzero Labs is building the next generation of decentralized infrastructure.
Site Reliability Engineer
We're building the infrastructure behind a next-generation decentralized programmable network with reliability, observability, and confidentiality baked in from the ground up. As a Site Reliability Engineer, you'll ensure the scalability, performance, and reliability of our large-scale blockchain applications and infrastructure.
Position Overview
Combining software engineering and systems administration expertise, you'll adopt a proactive, software-centric approach to tackle operational challenges. Your responsibilities include detecting issues, automating failure handling, devising disaster recovery plans, maintaining system uptime, and mitigating broken systems to prevent future disruptions. You'll leverage coding, automation, and engineering principles to build resilient, self-healing systems that scale to meet growing demands.
What You'll Do
Network Infrastructure Reliability: Design fault-tolerant systems to run validators, nodes, and indexers across cloud and bare-metal environments. Build self-healing mechanisms that recover automatically from faults, crashes, and partitions.
Infrastructure as Code: Define production systems using Terraform, Helm, Kubernetes, or Pulumi—supporting reproducible deployments, rapid scaling, and multi-region HA clusters.
TEE-Backed Secure Computation: Deploy and manage trusted execution environments (TEEs) such as Intel TDX, AMD SEV-SNP, or Azure Confidential VMs for secure blockchain operations.
Observability & Alerting: Build comprehensive Grafana dashboards and AlertManager alerts to monitor chain liveness, network performance, and quality metrics. Instrument services with tracing, metrics, and logs down to the hardware level.
Performance & Resource Tuning: Profile and tune workloads under sustained high throughput—optimizing CPU/memory/disk I/O pressure. Build tools to detect degraded validators or slow block propagation in real time.
Security Hardening & Key Management: Engineer hardened signing pipelines integrating TEEs, HSMs, or cloud-native KMS systems. Manage key lifecycle (rotation, expiration, revocation) with zero downtime while reducing attack surface area.
CI/CD & Safe Rollouts: Build GitHub Actions workflows testing and shipping changes across multiple environments. Own release engineering across devnet, testnet, and mainnet, ensuring protocol compatibility and seamless validator upgrades.
Incident Response & Chaos Engineering: Run fire drills, simulate node failures and partitions, and lead incident postmortems. Design for failure and validate assumptions under pressure.
Cross-Functional Communication: Work closely with engineers, product managers, node operators, and partners to support deployments, debug edge cases, and share best practices as a key interface between core protocol teams and the network operator ecosystem.
Requirements
10+ years in DevOps or SRE roles with focus on tooling, automation, and infrastructure
Proficiency in systems languages: Rust, Go, Python, Shell scripting
Experience writing and reviewing code, developing documentation and disaster recovery plans, debugging complex problems on live blockchain systems
Advanced knowledge of cloud infrastructure: networking, orchestration tools, containerization, compute, and storage systems
Proven ability to design, develop, and deploy systems enhancing throughput, latency, reliability, availability, and security
Clear communication skills: ability to explain technical concepts simply
Self-starter mindset: continuous learning and critical thinking under pressure
Preferred Qualifications
Background in distributed systems and consensus protocols
Experience with monitoring and observability platforms
Knowledge of security best practices for distributed systems
Top Skills
What We Do
Subzero Labs is the original contributor behind Rialo, the Real World Blockchain. Our team comprises experienced builders and researchers from leading organizations across blockchain infrastructure and hyper-growth products. Collectively, we have contributed to some of the most foundational projects in the ecosystem. Today, we are focused on designing a next-generation blockchain architecture that transcends incremental innovation.
