Principal Software Engineer - Kubernetes

Atlassian Cloud Storage Engineering (ACSE) is composed of software engineering teams tasked with developing and upkeeping the persistent data stores used by Atlassian's product and platform teams. The Managed Relational Data team has set ambitious goals, including operational efficiencies company-wide, improving reliability, cutting down the Total Cost of Ownership across Atlassian, and enhancing Trust (Security & Compliance).
In pursuit of these goals, the Managed Relational Data team is developing a self-hosted platform for use within Atlassian, aiming to achieve the ACSE vision to improve the clock speed of Atlassian by providing reliable, secure and cost effective storage solutions.
This position is for a Principal Engineer on the ACSE Homa team (Managed Relational Data team ~12 engineers) reporting to the team's Senior Engineering Manager. This role will require deep, expertise to build, operate and optimise high-quality relational data infrastructure in Kubernetes, outstanding collaboration skills to work effectively within a distributed team and engage with a broad range of internal customers, solid industry knowledge on building stateful application in Kubernetes and multicloud technologies, and great design and hands-on development skills to build automation and and operational tooling.
As a Principal Engineer, you will contribute to the architectural and technical direction of the Managed Relation Data team, help set the standard for engineering practices and provide mentoring to team members. You will also work with teams across Atlassian to provide guidance around data and storage solutions, identify cross-cutting areas where the platform can be enhanced and design platform capabilities.
Here, you'll collaborate with and provide guidance to experienced and inquisitive engineers to build the infrastructure that enables thousands of Atlassians to deploy and operate applications in the cloud.
Responsibilities & Activities:

• Design the implementation and operation of new and existing Managed Relational Data components. For example:
• Operating Kubernetes clusters at high load.
  
  • Deep understanding of managing high numbers of clusters for reliability, such as ensuring reliable version upgrades and effective cluster configuration management.
  • Building tooling and automation to facilitate the provisioning and operation of increasing numbers of clusters.
  • Advanced understanding of capacity management to ensure optimal performance and resource allocation within a system. This involves monitoring and analyzing the resources available within a cluster to effectively plan and manage workload distribution.
  • Advanced understanding of Kubernetes ecosystem, Kubernetes operator, and managing stateful application in Kubernetes.
  • Understanding of encryption at rest including KMS/data key management and BYOK, and other enterprise requirements.
• Engagement with product teams (JIRA/Confluence/Opsgenie team) to:
  
  • Support and guide them as they onboard the service to the self-hosted platform.
  • Adapt the platform to cater for their use cases - without compromising other customers.
  • Tuning and hardening the clusters based on a deep understanding of their data and usage patterns.
  • Contributing to the definition of appropriate SLAs that are suitable for customers and realistic for the Managed Relational Data team.
• Contributions towards technical leadership within the team
  
  • Determining and understanding priorities based on the broader view of Managed Relational Data within Atlassian.
  • Driving & documenting key technical decisions
  • Identifying opportunities & mitigating risks based on deep knowledge of the Managed Relational Data systems, as well as broad knowledge of adjacent systems (such as the consumers' systems) and underlying infra (such as Kubernetes).

Key Results Areas:

• Quality: The Managed Relational Data team will be key to both customer-facing functionality and internal business-critical workflows, so the platform's reliability and quality are essential metrics.
• Scale: The Managed Relational Data team must be able to scale out as customers' workloads increase and add clusters as the number of customers increases.
• Adoption: The Managed Relational Data team is responsible for building a Multi Cloud platform that is desirable to its consumers, engaging with customers to build trust in its product, and shipping in a sufficiently timely and incremental manner to enable dev teams to build on their components.

• Technical:
  
  • Deep experience with building and operating stateful application in Kubernetes with high load, including operating and tuning large clusters, capacity planning, implementing backup and recovery mechanisms, predicting and preventing cluster issues via monitoring, diagnosing and fixing unhealthy clusters, and implementing preventative solutions to avoid repeat failures.
  • Experience with a range of AWS services, their advantages and limitations, and understanding when to use specific services.
  • Experience building operationally mature systems with appropriate logging, monitoring, SLAs, alerting, and runbooks.
  • A high standard for quality software engineering (CI/CD, testing).
  • Experience progressively and safely rolling out changes to complex live systems.
  • Experience with Java/Kotlin.
  • Experience with Docker, Kubernetes.
  • Knowledge of Golang.
• Less technical:
  
  • Must be used to ownership of large deliverables and complex problems, and be able to do any of the following as appropriate to the situation:
    
    • clearly break up the work into smaller tasks to be carried out with the rest of the team
    • do a significant portion of the work themselves
    • enlist help/advice from other teams
    • demonstrate early that the task is not feasible / not the right focus.
  • Must be a top-notch team worker. Will work within the team's agreed processes (that they may help define); will be exemplary in communicating progress and risks, and helping to co-ordinate incremental deliverables within the team.
  • Experience working with remote teams.
  • Experience engaging with and building trust amongst internal customers. Excellent communication skills when engaging with customers and in written communication with the wider organisation.
  • Experience with incident management processes.
  • Experience participating in 24/7 on-call rosters (and willingness to do so on this team).
  • Non-hero attitude: Rather than celebrating a heroic effort to resolve an incident, prefer engaging in engineering practices that avoid the incidents in the first place.