Principal Engineer - Platform Management & Observability

About Graphcore

How often do you get the chance to build a technology that transforms the future of humanity? Graphcore products have set the standard in made-for-AI compute hardware and software, gaining global attention and industry acclaim. Now we are developing the next generation of artificial intelligence compute with systems that will allow AI researchers to develop more advanced models, help scientists unlock exciting new discoveries, and power companies around the world as they put AI at the heart of their business. Graphcore recently joined SoftBank Group – bringing large and ongoing investment from one of the world’s leading backers of innovative AI companies. 

Job Summary 

As the Principal Engineer you will provide technical leadership for the architecture, implementation and deployment of highly scalable management solutions for AI infrastructure built using our next-generation products. Covering monitoring, observability, control, and data centre infrastructure management, you will work closely with software, cloud and customer-facing teams, to establish first-hand knowledge of these solutions, creating proof-of-concepts, reference designs and integrations with third-party tooling. 

Your team will work closely with product, architecture and other delivery teams to ensure that functionally complete, simple to deploy, and easy to use solutions are deployed internally and supplied to our customers. 

Responsibilities and Duties

• Contribute to all phases of overall product development, from product definition, architecture, and design, through implementation, debugging, testing and early customer support. 
• Identify and evaluate new technologies and innovations to both anticipate future customer needs and develop a strategy for Graphcore data center management solutions. 
• Work with product management, other architects and our customer-facing teams to refine requirements for implementation. 
• Architect the solution, manage multi-component integrations based on Graphcore and third-party technology stacks, from data ingestion to decision making, ensuring seamless management, monitoring and UI. 
• Validate architectural decisions through proofs-of-concept. 
• Create reference designs including documentation, configuration files, scripts, source code, etc. 
• Deploy solution(s) internally for use by our engineering teams to aid debugging, performance analysis, benchmarking, test/QA, etc. at all scales. 
• Ensure solutions are properly tested by working with the development and QA teams to enhance unit testing and ensure a comprehensive test plan is in place. 
• Contribute to the design and implementation of fault-remediation solutions at scale. 
• Mentor and guide junior engineers, fostering a culture of continuous learning and improvement. 

Skills and Experience

• BSc or MSc degree in Computer Engineering, Computer Science, or related degree or equivalent experience. 
• Demonstrated success in architecting and implementing scalable, performant, reliable cluster management systems including telemetry collection & analysis engines. 
• Expertise of in-band and out-of-band management architectures and working knowledge of associated tools. 
• Knowledge of system management protocols such as Redfish and IPMI. 
• Awareness of how to securely monitor hardware and collect observability data without exposing sensitive system details. 
• You will have working knowledge of a variety of component technologies, such as Prometheus, Grafana, OpenTelemetry, Clickhouse, Kafka, Superset, in addition to common integrated stacks such as Elastic Stack, Better Stack, LGTM. 
• You will have working knowledge of at least one of Datadog, Dynatrace or Splunk. 
• Strong knowledge of the design and implementation of versioned, RESTful APIs. 
• Good understanding of automation technologies, such as Ansible. 
• Excellent written and verbal communication skills. 
• Strong and demonstrable skill in at least one of C/C++/Go and Python. 
• Experience programming and debugging skills for server platforms. 
• Experience with containerisation (e.g. Docker) 
• Experience managing or developing in Linux environments 
• Experience in the use of Jira and Confluence for project management. 

Desirable: 

• 14+ years of relevant post-degree experience. 
• Previous experience working with system software for accelerators such as GPUs, DPUs, or FPGAs. 
• Detailed knowledge and experience of working with Redfish APIs. 
• Familiarity with Open Compute (OCP). 
• Familiarity with DMTF standards and working groups. 
• Demonstrated ability to prototype ideas and objectively evaluate their value. 
• Knowledge of cloud-native development and deployment methodologies (SaaS/PaaS/IaaS). 
• Background with data-center design and technologies. 
• Knowledge of data center networking and monitoring best practices. 
• Contribution to open-source communities. 
• Working knowledge of commercial observability solutions like Datadog, Dynatrace and Splunk. 
• Knowledge of monitoring, observability and management solutions in use by hyperscalers and offered to their customers. 
• Knowledge of declarative management systems. 

Benefits

In addition to a competitive salary, Graphcore offers flexible working, a generous annual leave policy, private medical insurance and health cash plan, a dental plan, pension (matched up to 5%), life assurance and income protection. We have a generous parental leave policy and an employee assistance programme (which includes health, mental wellbeing, and bereavement support). We offer a range of healthy food and snacks at our central Bristol office and have our own barista bar! We welcome people of different backgrounds and experiences; we’re committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments.