The Staff Software Development Engineer - Enterprise AI Infrastructure is a senior technical role responsible for leading the design, development, and scaling of a centralized, highly governed enterprise AI platform. This position serves as a technical expert focused on AWS and Kubernetes-based (EKS) AI infrastructure, agentic development, and multi-agent orchestration operating within a regulated environment. The Staff Engineer partners closely with cross-functional stakeholders across Software Engineering, Data Science, Security, Regulatory Affairs, and Product to build a unified control plane that securely connects large language models with enterprise tools and company knowledge.
This role is expected to drive technical excellence in cloud infrastructure, container orchestration, AI governance, identity-scoped integrations, and agentic workflows while mentoring engineering teams and advancing the organization's enterprise AI strategy.
This role is based in Sunnyvale, California in our new headquarters. We will move in September so you may potentially visit our current location in Menlo Park, CA for interviews. We will also consider candidates in our Durham, NC office. We offer a flexible work arrangement, with the ability to work from GRAIL's office or from home. Our current flexible work arrangement policy requires that a minimum of 60%, or 24 hours, of your total work week be on-site. Your specific schedule, determined in collaboration with your manager, will align with team and business needs and could exceed the 60% requirement for the site. At our Sunnyvale and Durham campuses, Tuesdays and Thursdays are the key days where we encourage on-site presence to engage in events and on-site activities.
Responsibilities
- Lead the end-to-end design, development, deployment, and monitoring of a scalable, governed enterprise AI platform leveraging Amazon EKS and AWS native services (e.g., Bedrock, OpenSearch Serverless, KMS, VPC).
- Design and implement agentic AI workflows, specialized autonomous agents, and multi-agent systems using advanced LLM orchestration techniques and agent frameworks.
- Architect and manage secure integrations using the Model Context Protocol (MCP) to connect the AI platform with internal systems, vector databases, and third-party SaaS applications (e.g., Google Workspace, Slack).
- Build and enforce strict identity, authorization, and zero-trust token brokering flows leveraging Okta, Auth0, and custom JWT authorizers to ensure secure, least-privilege tool execution.
- Implement deterministic policy controls (e.g., Cedar policy engine) to enforce role-based access, approval gates, and human-in-the-loop checks at the API gateway level.
- Develop and maintain highly isolated, scalable containerized runtime environments (e.g., Kubernetes pods on Amazon EKS) for secure AI model execution, tool usage, and knowledge retrieval.
- Establish and maintain comprehensive audit trails and observability for all AI interactions, utilizing AWS CloudTrail and GenAI observability tools (e.g., OpenTelemetry) to track cost, latency, and tool calls.
- Collaborate with Product Management, Security, Regulatory, and business stakeholders to translate enterprise requirements into scalable, compliant AI infrastructure solutions.
- Troubleshoot and resolve complex technical issues involving cloud infrastructure, Kubernetes networking, network isolation (PrivateLink), and agentic workflows.
- Contribute to technology roadmaps, AI infrastructure strategy, and long-term platform evolution initiatives.
- Mentor engineers, software developers, and technical teams while promoting engineering excellence, infrastructure-as-code (IaC) best practices, and continuous improvement.
- Partner with Quality, Regulatory, Privacy, Security, and Compliance functions to ensure software and AI systems operate in accordance with applicable regulatory requirements and company policies.
Adaptability and Growth Expectation
- Taking on additional responsibilities.
- Participating in cross-functional projects and initiatives.
- Adapting to new technologies, AI methodologies, software frameworks, processes, or engineering practices.
- Supporting other departments or teams during periods of high demand.
- Contributing to special projects or temporary assignments as needed.
As our organization continues to evolve and grow, this role may require flexibility in responsibilities and duties. Employees should expect that their role may expand, shift, or be modified to meet changing business needs, strategic priorities, and organizational objectives.
This may include:
These job duties are a summary of the primary duties and responsibilities of the position and are not intended to be a comprehensive or all-inclusive listing of duties. Contents are subject to change at the Company's discretion.
Required Qualifications
- Bachelor's degree or equivalent in Computer Science, Software Engineering, Artificial Intelligence, Cloud Computing, or related field; Master's or PhD preferred.
- 8-12 years of relevant software development and cloud infrastructure experience with demonstrated technical leadership.
- Deep expertise in AWS cloud architecture and container orchestration, specifically with Amazon EKS, Kubernetes networking, network isolation (VPC, PrivateLink), IAM, KMS, and GenAI services (e.g., AWS Bedrock).
- Proven experience in agentic AI development, building autonomous agents, and orchestrating LLM tool-calling workflows using frameworks like LangChain, LangGraph, AutoGen, or Claude Agent SDK.
- Hands-on experience implementing the Model Context Protocol (MCP) or building robust, governed API/tool integrations for LLMs.
- Strong background in identity and access management (IAM), OAuth, JWT, and integrating with enterprise IdPs (Okta, Auth0) for scoped, token-based authorization.
- Advanced proficiency in programming languages such as Python, TypeScript, or Go, and infrastructure-as-code tools (Terraform, AWS CDK).
- Experience with vector databases, RAG (Retrieval-Augmented Generation) architectures, and row-level access controls (e.g., OpenSearch, FAISS, pgvector).
- Proficiency with CI/CD pipelines, MLOps practices, Kubernetes ecosystem tools (e.g., Helm), containerization, and modern observability stacks.
- Demonstrated level of knowledge regarding applicable regulatory standards commensurate with the position's complexity and scope, contributing to organizational regulatory compliance. Minimal applicable standards for this position include:
- Cybersecurity principles, tools, and control frameworks (e.g., ISO 27001, NIST, SOC 2, HIPAA)
- Operations within the regulated medical device environment (e.g., IVDD, IVDR, FDA 21 CFR 800 series, FDA 21 CFR Part 11)
- AI governance, software validation, data integrity, and risk management principles applicable to regulated environments
- Deep expertise in cloud infrastructure, containerized environments, agentic artificial intelligence, and secure distributed system design.
- Exceptional problem-solving and analytical skills with the ability to address ambiguous, high-impact technical challenges in AI orchestration and Kubernetes scaling.
- Strong leadership and influence skills, capable of driving alignment across engineering, security, regulatory, and business stakeholders.
- Excellent communication skills with the ability to explain complex LLM behaviors, infrastructure architectures, and security boundaries to technical and non-technical audiences.
- Proven mentoring and coaching capabilities that elevate cloud engineering and AI talent.
- Strong understanding of AI safety, prompt injection defenses, secure tool execution, and deterministic policy enforcement.
- Strategic thinking with the ability to balance long-term enterprise AI platform vision with near-term business delivery.
- High adaptability and intellectual curiosity regarding emerging agentic AI frameworks, MCP specifications, and cloud computing trends.
Physical Working Conditions
- Standard office or hybrid work environment depending on company policy.
- Frequent use of software development tools, AI/ML platforms, cloud infrastructure, data engineering tools, and collaboration systems.
- May require extended hours during major project deadlines, AI model deployments, production incidents, regulatory audits, or strategic initiatives.
- Operates with significant independence and responsibility and is expected to provide leadership on complex software, AI, technical, and organizational decisions.
This role may be eligible for other forms of compensation, including an annual bonus and/or incentives, subject to the terms of the applicable plans and Company discretion. This range reflects a good-faith estimate of the range that the Company reasonably expects to pay for the position upon hire; the actual compensation offered may vary depending on factors such as the candidate’s qualifications. Employees in this role are also eligible for GRAIL’s comprehensive and competitive benefits package, offered in accordance with our applicable plans and policies. This package currently includes flexible time-off or vacation; a 401(k) retirement plan with employer match; medical, dental, and vision coverage; and carefully selected mindfulness programs.
GRAIL is an equal employment opportunity employer, and we are committed to building a workplace where every individual can thrive, contribute, and grow. All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, sex, gender, gender identity, sexual orientation, age, disability, status as a protected veteran, , or any other class or characteristic protected by applicable federal, state, and local laws. Additionally, GRAIL will consider for employment qualified applicants with arrest and conviction records in a manner consistent with applicable law and provide reasonable accommodations to qualified individuals with disabilities. Please contact us at [email protected] if you require an accommodation to apply for an open position.
GRAIL maintains a drug-free workplace. We welcome job-seekers from all backgrounds to join us!
Skills Required
- Bachelor’s degree or equivalent in Computer Science, Software Engineering, Artificial Intelligence, Cloud Computing, or a related field
- 8–12 years of relevant software development and cloud infrastructure experience with demonstrated technical leadership
- Deep expertise in AWS cloud architecture and Amazon EKS, including Kubernetes networking, VPC, PrivateLink, IAM, KMS, and AWS Bedrock
- Proven experience developing agentic AI, autonomous agents, and LLM tool-calling workflows using frameworks such as LangChain, LangGraph, AutoGen, or Claude Agent SDK
- Hands-on experience implementing the Model Context Protocol or robust governed API and tool integrations for LLMs
- Strong IAM, OAuth, and JWT experience, including enterprise identity-provider integrations with Okta or Auth0
- Advanced proficiency in Python, TypeScript, or Go
- Experience with Terraform or AWS CDK for infrastructure as code
- Experience with vector databases, RAG architectures, and row-level access controls, including OpenSearch, FAISS, or pgvector
- Proficiency with CI/CD pipelines, MLOps, Kubernetes ecosystem tools such as Helm, containerization, and modern observability stacks
- Knowledge of cybersecurity principles and control frameworks such as ISO 27001, NIST, SOC 2, or HIPAA
- Knowledge of regulated medical device environments, including IVDD, IVDR, FDA 21 CFR 800 series, or FDA 21 CFR Part 11
- Knowledge of AI governance, software validation, data integrity, and risk management in regulated environments
- Deep expertise in cloud infrastructure, containerized environments, agentic AI, and secure distributed system design
- Exceptional problem-solving and analytical skills for complex AI orchestration and Kubernetes scaling challenges
- Strong leadership and influence skills across engineering, security, regulatory, and business stakeholders
- Excellent communication skills for explaining LLM behavior, infrastructure architectures, and security boundaries
- Proven mentoring and coaching capabilities
- Strong understanding of AI safety, prompt-injection defenses, secure tool execution, and deterministic policy enforcement
- Strategic thinking, adaptability, and intellectual curiosity regarding agentic AI frameworks, MCP, and cloud-computing trends
- Master’s degree or PhD
What the Team is Saying
.JPG)
.JPG)
GRAIL Compensation & Benefits Highlights
-
Healthcare Strength — Multiple medical plan options, comprehensive medical/dental/vision coverage, and strong mental health programs are described, with employer HSA contributions up to $3,000 for the HDHP. These elements indicate a well‑rounded health offering.
-
Leave & Time Off Breadth — Flexible time off is highlighted, with examples of unlimited/FTO policies working well in practice. Paid holidays and company winter breaks further support time away from work.
-
Equity Value & Accessibility — New‑hire equity grants that vest over four years and a discounted Employee Stock Purchase Plan are positioned as meaningful parts of total compensation. Ongoing eligibility for additional equity is also noted.
GRAIL Insights
What We Do
GRAIL is a healthcare company whose mission is to detect cancer early, when it can be cured. GRAIL is using the power of high-intensity sequencing, population-scale clinical studies, and state-of-the-art computer science and data science to enhance the scientific understanding of cancer biology, and to develop and commercialize pioneering products.
Why Work With Us
Everything we do is guided by our mission to detect cancer early, when it can be cured. It’s the reason we’re here, and it’s no small task. The right people make all the difference. That’s why we’re looking for those who strive to share their knowledge, contribute their skills, inspire each other and commit to something bigger than themselves.
Gallery
GRAIL Offices
Hybrid Workspace
Employees engage in a combination of remote and on-site work.
GRAIL has a variety of work types depending on the roles. Some roles are onsite like a lab role, some are fully remote like our Galleri Sales Consultant roles. Others are hybrid with 2-3 days onsite. Typically Tuesday and Thursday.
.png)
