About Us
- Design and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments
- Core capability development:
- Fault Injection Engine: Pod-level / Node-level / AZ-level fault simulation, network latency / packet loss / partition, dependency timeout / error injection
- Production Safety Assurance: Blast radius control, one-click Kill Switch, automatic rollback, real-time impact monitoring
- Traffic Isolation: Experiment traffic tagging and isolation to ensure fault injection does not impact real users
- Fault Isolation: Precise impact scoping at service / cluster / AZ granularity
- Design experiment orchestration capabilities supporting complex fault scenario composition (e.g., simultaneous network latency + downstream timeout + cache invalidation)
- Deep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail
- Define safety standards and approval workflows for mainnet fault injection
- Design and drive routine chaos experiments:
- Daily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis
- Periodic validation experiments: Monthly/quarterly resilience verification of critical paths
- Large-scale drills: Cross-AZ / cross-region disaster recovery failover validation
- Establish a resilience scoring system to quantify system health based on experiment results
- Deliver improvement recommendations and drive business teams to remediate identified weaknesses
- Evaluate and select the technology foundation (Chaos Mesh / Litmus / custom components — hybrid strategy)
- Develop chaos engineering best practices and playbooks to enable SRE teams and application developers
- Mentor and grow the team (2–3 engineers) in chaos engineering capabilities
- Stay current with industry developments and introduce cutting-edge practices (e.g., AI-driven fault scenario discovery)
- 8+ years of backend / infrastructure engineering experience, with 3+ years dedicated to chaos engineering or stability engineering
- Hands-on experience with large-scale fault injection in production environments (not just test environments), with deep understanding of production safety constraints
- Expert-level proficiency in Kubernetes fault injection (Chaos Mesh / Litmus / custom solutions), familiar with CRD / Operator development
- Proficient in at least one backend language (Go preferred), with platform-level system architecture design capability
- Deep understanding of distributed system failure modes (network partitions, split-brain, cascading failures, data inconsistency, etc.)
- Familiarity with observability tech stack (Prometheus / Grafana / Thanos / OpenTelemetry)
- Excellent technical documentation and solution design skills
- Experience in financial / trading system stability (understanding of transaction consistency and fund safety constraints)
- Experience building SLO / Error Budget frameworks
- Experience building automated fault recovery (self-healing) systems
- Familiarity with AWS infrastructure (EC2 / EKS / Multi-AZ / Multi-Region)
- Knowledge of Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin
- Open-source community contributions (Chaos Mesh / Litmus or similar projects)
- Ability to balance "safety" and "validation depth" — not afraid of production injection, while maintaining strict risk control
- Strong cross-team collaboration and influence — chaos engineering requires buy-in from business teams; this role demands persuasion skills
- Self-driven, capable of independently planning and executing in ambiguous situations
Why Join Us
At Bybit, we are committed to fostering a supportive and enriching work environment.
Our benefits include:
- Study Growth Fund: We support your professional development and continuous learning.
- Internal Events: Participate in regular team-building activities, workshops, and events designed to promote collaboration and innovation.
- Global Collaboration: Be part of a diverse, international team, working alongside colleagues from around the world.
- Career Advancement: Access opportunities for growth and advancement within a rapidly expanding global company.
- Internal Mobility: Grow with us- Your long-term development is important to us. We offer internal job opportunities to help build your career path.
Skills Required
- 8+ years of backend or infrastructure engineering experience
- 3+ years of chaos engineering or stability engineering experience
- Hands-on experience with large-scale fault injection in production environments
- Expert proficiency in Kubernetes fault injection using Chaos Mesh, Litmus, or custom solutions
- Familiarity with CRD and Operator development
- Proficiency in at least one backend language, preferably Go
- Platform-level system architecture design capability
- Deep understanding of distributed system failure modes
- Familiarity with Prometheus, Grafana, Thanos, or OpenTelemetry
- Excellent technical documentation and solution design skills
- Experience with financial or trading system stability
- Experience building SLO and error budget frameworks
- Experience building automated fault recovery or self-healing systems
- Familiarity with AWS infrastructure, including EC2, EKS, Multi-AZ, and Multi-Region
- Knowledge of Netflix Chaos Engineering, AWS Fault Injection Simulator, or Gremlin
- Open-source contributions to Chaos Mesh, Litmus, or similar projects
What We Do
Since its inception in March 2018, Bybit has emerged as a leading cryptocurrency exchange, offering a comprehensive suite of tailored crypto services and product solutions meticulously crafted for retail and institutional traders alike. Trusted by millions worldwide, Bybit continues to push the boundaries of innovation, consistently refining and expanding its multi-spectral product offerings. At the heart of Bybit's mission lies the unwavering commitment to providing a secure, reliable, and transparent trading platform that empowers investors to seamlessly navigate the dynamic world of crypto. Whether you seek to explore Spot or Derivatives trading, harness the potential of Mining and Staking products, or leverage API support for algorithmic trading, Bybit empowers you to take control of your crypto journey. As pioneers in the ever-evolving Web3 landscape, Bybit meticulously curates and incubates innovative crypto projects, decentralized solutions, and data-driven research, fostering a vibrant ecosystem of crypto education and growth opportunities. Our core values of empathy, responsiveness, and continuous improvement permeate every aspect of our operations, shaping an inclusive environment where every individual feels valued and empowered. Bybit actively collaborates with regulatory bodies and institutions worldwide, playing a pivotal role in establishing robust safety and security frameworks, paving the way for a secure and dynamic future of crypto. Embrace the Digital Revolution with Bybit: Your Trusted Partner in the Crypto Frontier. #CryptoArk #Bybit
