Wells Fargo is seeking a Principal Engineer
In this role, you will:
- Act as an advisor to leadership to develop or influence applications, network, information security, database, operating systems, or web technologies for highly complex business and technical needs across multiple groups
- Lead the strategy and resolution of highly complex and unique challenges requiring in-depth evaluation across multiple areas or the enterprise, delivering solutions that are long-term, large-scale and require vision, creativity, innovation, advanced analytical and inductive thinking
- Translate advanced technology experience, an in-depth knowledge of the organizations tactical and strategic business objectives, the enterprise technological environment, the organization structure, and strategic technological opportunities and requirements into technical engineering solutions
- Provide vision, direction and expertise to leadership on implementing innovative and significant business solutions
- Maintain knowledge of industry best practices and new technologies and recommends innovations that enhance operations or provide a competitive advantage to the organization
- Strategically engage with all levels of professionals and managers across the enterprise and serve as an expert advisor to leadership
- 7+ years of Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
- Design GPU cluster topologies (H100/H200, NVLink/NVSwitch), networking, and storage paths for high-throughput inferencing; document sizing and perf baselines.
- Implement Run: AI constructs (Collections/Departments/Projects/workloads) for MDEV/MDEP/UCEP/MRM; codify quota, priority, and fair-share policies.
- POC & benchmark disaggregated inferencing (prefill/decode) with vLLM/TensorRT-LLM; publish guidance for H100/H200 tuning (FP8/INT8/AWQ) and KV-transfer behavior over NVLink.
- Operationalize OpenShift AI parity for GPU scheduling, time slicing/MIG profiles, and preemption; validate upgrade paths and helm/kustomize packaging.
- Integrate Triton Inference Server for multi-model serving; standardize model repository structure, batching, dynamic shapes, and telemetry hooks. (Supported broadly by platform docs; add repo specifics when you share them.)
- Harden NGDC environments with AVI/GSLB patterns (Prod1/Prod2) and BCP; execute DR failover runbooks and steady-state capacity planning.
- Publish steady-state runbooks (deploy → certify → promote): DEV → UAT → MDEP-Beta → MDEP-GA / UCEP; define promotion criteria and risk exceptions.
- Own endpoint product ionization via Apigee (AI Gateway)-authN/Z, rate limiting, API SLAs, versioning/deprecation and SDK generation for internal consumers.
- Embed observability/evaluations with Overwatch + Arize: prompt/agent/tool tracing, SLO dashboards, alerting, and data-retention/export workflows.
- Automate CI/CD for infra and model artifacts: image scanning (JFrog remote repo), chart releases, canaries, and rollback plans across OCP/GKE.
- Tune CUDA kernels/graph execution paths; profile NCCL collectives; resolve performance bottlenecks (HBM bandwidth, kernel fusion, p2p comms). (NCCL inferred per assumption.)
- Qualify LLM/SLM runtimes (Gemma, Llama, GPT-OSS, etc.) with Run: AI scheduling; publish per-model recipes for throughput, latency, cost and stability.
- Define GPU estate hygiene: image provenance, secrets handling, namespace/network policy baselines, and change controls for upgrades (e.g., Run: AI v2.21+).
- Partner with product/TPM/PO to align backlog to platform milestones (OpenShift AI go-forward, SuperPOD activation waves, endpoint rollouts).
- Mentor engineers; lead deep-dive reviews and present in exec/tech forums (CIO/ARB/offsites) with architecture readouts, performance data, and risk mitigations.
- NVIDIA & CUDA: CUDA/cuDNN usage, NVLink/NVSwitch understanding, MIG setup, NCCL tuning, GPU profiling, H100/H200 optimization. Optimize kernels and collectives, choose MIG profiles, validate interconnect bandwidth and NUMA/PCIe topology for LLM/SLM workloads.
- LLM/SLM Runtimes: Work with vLLM, TensorRT-LLM, Triton; apply FP8/INT4 quantization; tune KV-cache strategies. Build POCs for disaggregated prefill/decode, standardize Triton repos, and optimize batching.
- Orchestration: Use Run: AI structures (Collections/Departments/Projects), manage OCP/GKE environments. Implement GPU allocation patterns, enforce quotas, preemption, fair-share scheduling.
- OpenShift AI: Configure RHOAI GPU scheduling and time slicing, use helm/kustomize, validate upgrades. Achieve platform parity, certify charts and policies, ensure admission controls function reliably.
- API & Gateway: Apply Apigee authN/Z, manage quotas, rate limits, OpenAPI specs, SDK generation, SLA operations. Productionize model endpoints, manage versioning and deprecation, enforce gateway-level SLAs
- Observability & Evaluation: Use Overwatch + Arize for tracing and evals, define SLOs, alerts, retention/export processes. Trace prompts/tools/agents, enforce data retention, publish standardized dashboards.
- CI/CD & Artifacts: Manage JFrog repos, image scanning, helm releases, canary/rollback workflows. Standardize artifact flow, automate safe promotions, ensure compliant releases.
- Performance Engineering: Model throughput/latency, optimize token/sec, batch shaping, cache policies. Produce per-model performance recipes, tune cost/performance tradeoffs for LLM/SLM.
- Controls & SDLC: Apply JAD lite practices, manage change controls, secrets hygiene, namespace/network policies. Maintain compliance across GPU estate, ensure full auditability and proper access boundaries.
- Communication: Create executive-friendly narratives, write architectures and runbooks, present in forums. Deliver content in offsites/CIO forums, publish clear decision memos.
9 Feb 2026
*Job posting may come down early due to volume of applicants.
We Value Equal Opportunity
Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.
Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit's risk appetite and all risk and compliance program requirements.
Candidates applying to job openings posted in Canada: Applications for employment are encouraged from all qualified candidates, including women, persons with disabilities, aboriginal peoples and visible minorities. Accommodation for applicants with disabilities is available upon request in connection with the recruitment process.
Applicants with Disabilities
To request a medical accommodation during the application or interview process, visit Disability Inclusion at Wells Fargo .
Drug and Alcohol Policy
Wells Fargo maintains a drug free workplace. Please see our Drug and Alcohol Policy to learn more.
Wells Fargo Recruitment and Hiring Requirements:
a. Third-Party recordings are prohibited unless authorized by Wells Fargo.
b. Wells Fargo requires you to directly represent your own experiences during the recruiting and hiring process.
Top Skills
What We Do
Wells Fargo & Company (NYSE: WFC) is a leading financial services company that has approximately $2.1 trillion in assets. We provide a diversified set of banking, investment and mortgage products and services, as well as consumer and commercial finance, through our four reportable operating segments: Consumer Banking and Lending, Commercial Banking, Corporate and Investment Banking, and Wealth & Investment Management. Wells Fargo ranked No. 33 on Fortune’s 2025 rankings of America’s largest corporations.
Our technology professionals drive innovation, information security, and big data analytics while maintaining a network that handles more than 12 billion customer interactions a year. Join us!
Are you looking for more? Find it here.
At Wells Fargo, we're more than a financial services leader – we’re a global trailblazer committed to driving innovation, empowering communities, and helping our customers succeed. We believe that a meaningful career is much more than just a job – it’s about finding all of the elements to help you thrive, in one place. Living the Well Life means you’re supported in life, not just work. It means having robust benefits, competitive compensation, and programs designed to help you find work-life balance and well-being. You’ll be rewarded for investing in your community, celebrated for being your authentic self, and empowered to grow. And we’re recognized for it – Wells Fargo once again ranked in the top three – making us the #1 financial services employer – on the 2025 LinkedIn Top Companies list of best workplaces “to grow your career” in the U.S.
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic. © 2026 Wells Fargo Bank, N.A. All rights reserved. Member FDIC.
Why Work With Us
We're known for our “Well Life” approach to supporting employees’ career aspirations, work-life balance, and mental and physical health. We ranked in the top 3 on the 2025 LinkedIn Top Companies list – and #1 among financial services companies – as the best workplace “to grow your career” in the U.S.
Gallery
Wells Fargo Offices
Hybrid Workspace
Employees engage in a combination of remote and on-site work.
