Application Deadline:
Address:
1630 Chicago RoadJob Family Group:
TechnologyPrincipal Engineer, AI Platform & Fabrics
Description
BMO is building the platform capabilities that make enterprise AI safe, governed, and scalable. We are seeking experienced Principal/Senior engineers to build and operate the core infrastructure that governs how AI runs at BMO — the AI Gateway, Policy Engine, Identity Fabric, AI Registry, Guardrails Runtime, and AI Observability.
This is a build-and-run engineering role. You will be part of the team that own Enterprise AI Platform capabilities end to end: Building, configuring and operating them in production, including on-call. You will not build the AI models or applications themselves (those are domain-owned); you build the governed platform they run on and the runtime evidence that proves they run within policy, across AWS, Azure, and Microsoft AI surfaces, under OSFI and OCC expectations.
You are a hands-on engineer who has built shared platform services at scale, cares deeply about operability, latency, and correctness, and understands that in a regulated bank the infrastructure must produce its own evidence. You are energized by taking real engineering assets that includes an existing developer portal, an AI registry, a body of policy-as-code, and gateway integrations and hardening, scaling, enhancing and governing them into enterprise-grade platform capabilities. You raise the technical bar for those around you and mentor as you build.
What You'll Build & Operate
Depending on your specialization, you will own one or more of the following capability areas:
Enterprise Control Plane
Portal & Registry — a federated AI Registry (agents, models, tools, channels, evaluations across 16+ asset types) with self-service onboarding and lifecycle workflows; federation with external registries (Agent 365, AgentCore, MLflow).
Policy Engine — policy-as-code infrastructure (Cedar/OPA), a policy compilation pipeline, GitOps-based domain-scoped bundle distribution, risk-tiered approval workflows, and a policy simulation sandbox.
Observability & Audit — a multi-pipeline telemetry architecture (operational + security + compliance), OpenTelemetry GenAI conventions, cross-pipeline trace correlation, lineage-stamped traces, and a 7-year tamper-evident audit lake producing regulator-ready evidence.
Governance & Lifecycle — certification workflows, automated compliance scoring, decommission governance, and evidence generation for architecture and model-risk review.
Domain Orchestration
Gateway Runtime — domain-hub deployment across AWS and Azure; an inline enforcement engine performing request-time policy evaluation, routing, residency, budget/quota, and circuit breaking within strict tiered latency budgets (Fast <5ms / Standard <25ms / Full <50ms); cross-region failover.
Guardrails Runtime — a multi-stage safety pipeline (input moderation → prompt-injection defense → PII → output validation → hallucination detection → policy enforcement) with bilingual EN/FR parity and behavioral guardrails for agentic workloads (goal hijacking, intent drift, excessive agency).
Identity Fabric — workload identity for AI (SPIFFE/SPIRE), token-exchange bridging, per-domain trust boundaries, Entra Agent ID integration, on-behalf-of identity propagation, and cross-cloud token federation with zero-trust attestation.
What You'll Do
Own capabilities end to end: design, implement, test, ship, and operate production infrastructure.
Engineer for operability and defensibility from day one: instrumentation, SLOs, latency budgets, failure modes, and runtime evidence built in, not bolted on.
Build the APIs, SD'able interfaces, and integrations through which domains, DevOps pipelines, and enterprise systems consume platform capabilities.
Implement policy enforcement, guardrails, identity attestation, and audit as first-class engineering concerns: correct, performant, and provable.
Ensure every capability produces runtime evidence connecting AI activity to policy enforcement, identity, and lineage for model-risk and regulatory review (OSFI E-23, OCC).
Assess emerging AI infrastructure, foundation-model access patterns, and standards; make deliberate, cost-aware engineering choices.
Mentor and raise the bar: set engineering standards, review designs and code, and grow depth across the team.
Partner closely with AI Developer Experience, AI Security and AI SDLC, and the Senior AI Architect in your platform build and operations.
Education & Experience
Bachelor's degree in Computer Science, Software Engineering, or a related technical discipline (Master's preferred).
8+ years of software/platform engineering experience (Principal), or 5+ years (Senior), with substantial time building and operating shared platform services at enterprise scale.
Demonstrated experience operating production infrastructure with real SLOs and on-call ownership, ideally in a regulated industry (financial services strongly preferred).
Depth in one or more of: API gateways / traffic enforcement; policy-as-code and authorization; workload identity / zero-trust; observability and telemetry pipelines; audit/compliance data platforms.
Required Core Skills
Platform Engineering Depth
Strong distributed-systems and platform-engineering fundamentals: latency-sensitive request paths, resilience patterns (circuit breakers, failover), multi-tenancy, and high availability.
Strong programming skills (Python and/or Go preferred; TypeScript/Java an asset) for building performant services, APIs, and integrations.
Cloud-native architecture across AWS and Azure: containers/Kubernetes, service mesh, and Infrastructure as Code (CDK, Terraform, CloudFormation/ARM).
Robust CI/CD, GitOps, and DevSecOps practice; Git-based workflows (Bitbucket/GitHub), Jira, Confluence.
Capability-Specific Depth (one or more)
Policy/Authorization: Cedar, OPA/Rego, policy compilation and distribution, risk-tiered approval workflows.
Identity/Zero-Trust: SPIFFE/SPIRE, mTLS, token exchange, OAuth/OIDC, federated and cross-cloud identity, Entra ID/Agent ID.
Observability/Audit: OpenTelemetry (incl. GenAI conventions), distributed tracing, Dynatrace/Splunk or equivalents, tamper-evident/immutable audit stores, data lineage.
Gateway/Guardrails: API gateway internals, inline enforcement, LLM routing/abstraction, prompt-injection and PII defenses, hallucination detection, AI evaluation.
Registry/Portal: service catalogues, asset registries, lifecycle workflows, federation with external registries.
GenAI & Governance Context
Working knowledge of GenAI platform patterns: LLM/AI gateways, RAG and agentic patterns, foundation models, embeddings, and guardrails — sufficient to build the infrastructure they depend on.
Familiarity with AI/ML platforms (Bedrock, Azure OpenAI, SageMaker, MLflow) and orchestration frameworks (LangChain, LlamaIndex).
Grounding in Responsible AI, AI/data governance, privacy, cloud security, and IAM as applied to AI workloads.
Certifications (Preferred)
AWS Certified Solutions Architect (Associate/Professional) / ML – Specialty
Microsoft Certified: Azure Solutions Architect Expert / Azure AI Engineer Associate
Security/identity certifications (relevant to Identity Fabric roles)
Databricks / Google Cloud ML credentials; relevant GenAI/LLM credentials
Other Skills
Strong communication and collaboration across engineering, security, architecture, and domain teams.
A critical thinker with strong analytical and problem-solving skills.
Self-directed, comfortable with ambiguity and a fast-evolving mandate.
Able to deliver complex work under tight timelines; participates in on-call rotation for owned services.
Salary:
$112,200.00 - $209,000.00Pay Type:
SalariedThe above represents BMO Financial Group’s pay range and type.
Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group’s expected target for the first year in this position.
BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: https://jobs.bmo.com/global/en/Total-Rewards
About Us
At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.
As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset.
To find out more visit us at http://jobs.bmo.com/us/en
BMO is proud to be an equal employment opportunity employer. We evaluate applicants without regard to race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or any other legally protected characteristics. We also consider applicants with criminal histories, consistent with applicable federal, state and local law.
BMO is committed to working with and providing reasonable accommodations to individuals with disabilities. If you need a reasonable accommodation because of a disability for any part of the employment process, please send an e-mail to [email protected] and let us know the nature of your request and your contact information.
Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.
Skills Required
- Bachelor's degree in Computer Science, Software Engineering, or a related technical discipline
- 8+ years of software or platform engineering experience for Principal level, or 5+ years for Senior level
- Experience building and operating shared platform services at enterprise scale
- Experience operating production infrastructure with SLO and on-call ownership
- Depth in one or more of API gateways and traffic enforcement, policy-as-code and authorization, workload identity and zero trust, observability and telemetry pipelines, or audit and compliance data platforms
- Strong distributed systems and platform engineering fundamentals, including resilience, multi-tenancy, and high availability
- Strong programming skills in Python and/or Go
- Experience with AWS and Azure cloud-native architecture, containers or Kubernetes, service mesh, and Infrastructure as Code
- Experience with CI/CD, GitOps, DevSecOps, and Git-based workflows
- Working knowledge of GenAI platform patterns, including LLM gateways, RAG, agentic systems, foundation models, embeddings, and guardrails
- Familiarity with AI/ML platforms such as Bedrock, Azure OpenAI, SageMaker, or MLflow
- Familiarity with orchestration frameworks such as LangChain or LlamaIndex
- Knowledge of responsible AI, AI and data governance, privacy, cloud security, and IAM for AI workloads
- Master's degree in a technical discipline
- TypeScript or Java programming experience
- AWS Certified Solutions Architect or ML Specialty certification
- Microsoft Azure Solutions Architect Expert or Azure AI Engineer Associate certification
- Security or identity certification
- Databricks, Google Cloud ML, or relevant GenAI/LLM credentials
- Experience in a regulated industry, preferably financial services
BMO Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about BMO and has not been reviewed or approved by BMO.
-
Parental & Family Support — Paid parental leave up to 16 weeks at full pay for all new parents, plus up to $20,000 for adoption, surrogacy, and fertility, and 10 days of paid backup childcare indicate robust family support. These elements stand out within BMO’s U.S. package.
-
Retirement Support — A 401(k) design combining a core employer contribution with dollar-for-dollar matching up to a set portion of pay, plus immediate vesting on match and employee contributions, signals strong retirement funding. The core contribution’s three-year vesting is clearly defined.
-
Leave & Time Off Breadth — Vacation accrual scales with grade and service, alongside 9–10 paid holidays and additional paid time off buckets (bereavement, school activities, civic duties, blood donation, volunteering). This breadth offers multiple avenues for time away beyond standard vacation.
BMO Insights
What We Do
At BMO, banking is our personal commitment to helping people at every stage of their financial lives. The truth is, people’s needs change: so we change too. But we never change who we are. Which means we’ll never waiver from providing our customers the best possible banking experience in the industry. Our incredible team of over 46,000 people is just the tip of the iceberg. You should get to know us. We’re here to help.
