Senior Staff Engineer - Observability Infrastructure

Salary Range: PLN 350,700 - 474,400 + Benefits + Equity

Subject to alignment to the responsibilities and duties of the role.

About Graphcore

At Graphcore, we’re building the future of AI compute.We’re a team of semiconductor, software and AI experts, with deep experience in creating the complete AI compute stack - from silicon and software to infrastructure at datacenter scale.As part of the SoftBank Group, backed by significant long-term investment, we are delivering key technology into the fast-growing SoftBank AI ecosystem.To meet the vast and exciting AI opportunity, Graphcore is expanding its teams around the world.We are bringing together the brightest minds to solve the toughest problems, in a place where everyone has the opportunity to make an impact on the company, our products and the future of artificial intelligence.

Job Summary

As a Senior Staff Engineer you will contribute to the architecture, implementation and deployment of highly scalable management and observability solutions for AI infrastructure built using our next-generation products. Covering monitoring, observability, control, and data center infrastructure management, you will work with software, cloud and customer-facing teams, to establish first-hand knowledge of these solutions, creating proof-of-concepts, reference designs and integrations with third-party tooling.

Your team will work with product, architecture and other delivery teams to ensure that functionally complete, simple to deploy, and easy to use solutions are deployed internally to support engineering efforts and supply reference designs to our customers.

Responsibilities and Duties

• Contribute to all phases of overall product development, from product definition, architecture, and design, through implementation, debugging, testing and early customer support.
• Contribute to the design and implementation of fault-remediation solutions at scale.
• Implement multi-component integrations based on Graphcore and third-party technology stacks, from data ingestion to decision making, ensuring seamless management, monitoring and UI.
• Create reference designs including documentation, configuration files, scripts, source code, etc.
• Deploy solution(s) internally for use by our engineering teams to aid debugging, performance analysis, benchmarking, test/QA, etc. at all scales.
• Maintain and improve deployed infrastructure long term to provide the best service for our customers.• Ensure solutions are properly tested by working with the development and QA teams to enhance unit testing and ensure a comprehensive test plan is in place.
• Mentor and guide junior engineers, fostering a culture of continuous learning and improvement.

Skills and Experience

• BSc or MSc degree in Computer Engineering, Computer Science, or related degree or equivalent experience. 
• Demonstrated success in architecting and implementing scalable, performant, reliable cluster management systems including telemetry collection & analysis engines. 
• Experience in managing big scale datacenters focusing on hardware observability solutions. 
• Experience in maintaining and scaling modern observability stacks using Prometheus, Grafana, OTEL, ClickHouse, Kafka, Superset or Elastic Stack understanding of secure telemetry practices and data exposure controls.
• Working knowledge of Datadog, Dynatrace or Splunk. 
• Experience in large-scale telemetry datasets, time series databases, down-sampling techniques, and creating actionable dashboards. 
• Experience automation technologies, such as Ansible or Terraform. 
• Experience in containerisation (Docker, Kubernetes) 
• Experience in managing or developing in Linux environments. 
• Strong and demonstrable skill in at least one of C/C++/Go and Python. 
• Excellent written and verbal communication skills.

Desirable:

• 10+ years of relevant post-degree experience.
• Knowledge of cloud-native development and deployment methodologies (SaaS/PaaS/IaaS).
• Knowledge of data center networking and monitoring best practices.
• Knowledge of monitoring, observability and management solutions in use by hyperscalers.
• Knowledge of declarative management systems.

Benefits

In addition to a competitive salary, annual leave policy, medical and dental health plans, a gym card and employee pension (matched up to 4%). We review our benefits on a yearly basis to ensure we offer a valuable and rewarding benefits programme to our employees.  We welcome people of different backgrounds and experiences; we’re committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments.

Applicants for this position must hold the right to work in the Poland. Unfortunately at this time, we are unable to provide visa sponsorship or support for visa applications.