Senior Software Engineer (python, infrastructure)

Sauce Labs is the leading provider of continuous test and error reporting solutions that give companies the confidence to develop, deliver and update high quality software at speed.  The Sauce Labs Continuous Testing Cloud identifies quality signals in development and production, accelerating the ability to release and update web and mobile applications that look, function and perform exactly as they should on every browser, operating system and device, every single time.  Sauce Labs is a privately held company funded by TPG and Riverwood Capital.

Sauce Labs seeks a Senior Software Engineer to join the Virtual Device Cloud team, which builds and operates the infrastructure that hosts remote iOS, macOS, Windows, and Linux devices at scale for automated app and browser testing.

This is not a narrow specialist role. We need someone who can move across the whole stack of a distributed system: understand how the pieces fit together, make the system observable enough that we know what's working and can diagnose what isn't, write and structure code with clear boundaries and the right things encapsulated, and be just as comfortable SSHing into a fleet machine to dig into a live problem as designing the system on a whiteboard.

You'll join the Virtual Device Cloud team, which has deep, concentrated strength in macOS, CoreSimulator and APFS internals, and work closely with the Product Manager and with the Test Orchestration and Cloud teams. Your job is to improve testing discipline and cross-platform observability, and to raise delivery cadence, correctness, and visibility across the fleet.

• Lead by example - set the team’s standard for testing, observability and delivery through code review and pairing, and bring others along.
• Own the device lifecycle - design and build the services this team owns across the whole device lifecycle, from the image a device boots to its teardown. That covers readiness, session lifecycle and the fleet definitions that plug simulators and VMs into our orchestration layer, whether the fleet runs as pods on Kubernetes with Agones or as VMs on bare-metal and cloud hosts.
• Automate build and release - bring image, browser and payload releases onto CI/CD with automated checks, so a release is a pipeline run rather than a manual procedure. Ship in Go or Python on standard, repeatable deployment paths.
• Make the fleet observable - build and use the observability tooling that answers "is it working as intended, and if not, why not" across the fleet and the image build pipeline, including guests and VMs that fail silently before their payload even starts.
• Fix lifecycle problems for good - root-cause and fix recurring lifecycle regressions with tests and automated release checks, so the same bug does not ship twice.
• Make devices boot and recycle faster - measure and reduce boot, ready and teardown time end to end, and put image and side-disk footprint on a budget.
• Work across teams - work with the Cloud and Test Orchestration teams to propose and land evidence-backed changes to systems you do not own. Keep the contracts between our layers explicit.

• Solid experience designing, building, and operating distributed systems in production.
• Hands-on experience with Kubernetes, ideally including fleet orchestration such as Agones or a comparable session-fleet model.
• Proven ability to ship production code in Go and/or Python through a standard CI/CD deployment path, including building and maintaining the pipelines themselves.
• Strong observability instincts and experience: metrics, logging, tracing, and building visibility into systems that don't have it yet.
• Hands-on DevOps and Linux/Unix systems experience: able to log into a machine (VM, container host, or bare metal) and work confidently - diagnosing, fixing, and understanding what's actually happening on the box.
• Delivers through the team: a track record of shipping via review and pairing on systems others operate, not alone.

• macOS literacy: simctl, launchd, XPC, and how Apple's virtualization/simulator stack fits together.
• Storage and copy-on-write design knowledge (e.g. APFS or similar filesystem internals).
• Working experience across Windows and Linux infrastructure.
• Familiarity with the Apple Virtualization Framework and related tooling (e.g. tart).
• Browser or mobile app test-automation familiarity (Appium, XCTest, WebDriver).
• Has owned a startup, boot or ready-time number for VMs, containers, CI runners or game servers and driven it down.
• Swift: some of our repositories use it, and reviewing them is part of the role.
• Experience building and maintaining OS or VM images with tools such as Packer, tart, Ansible, cloud-init or sysprep.

We are a hybrid workplace that recognizes the importance of flexibility while valuing in-person collaboration and relationship building. As a result, Saucers located near an office location must be able and willing to come into the office. Those hired remotely must be able and willing to travel to an office as required by the specific role. 

Please note our privacy terms when applying for a job at Sauce Labs.

Sauce Labs is proud to be an Equal Opportunity employer and values diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender identity/expression/status, sexual orientation, age, marital status, veteran status or disability status.

Security responsibilities at Sauce

At Sauce, we will commit to supporting the health and safety of employees and properties, partnering with internal stakeholders to learn and act on ever-evolving security protocols and procedures. You’ll be expected to fully comply with all policies and procedures related to security at the department and org wide level and exercise a ‘security first’ approach to how we design, build & run our products and services.