The Role
Lead the design and delivery of AI context engineering and evaluation harness workstreams. Develop RAG, memory, token-management, ontology, MCP integration, sandboxing, orchestration, validation, and evaluation standards. Support pilot teams, define integrations with GitHub, Notion, and Slack, build YAML-based knowledge models and repo-native harness prototypes, and translate pilot evidence into company-wide best practices and technical standards.
Summary Generated by Built In
The Context & Harness Committee is moving from strategy to execution. We are seeking three hands-on AI Tech Leads to define and validate our context engineering standards and evaluation harness before company-wide rollout.
Core responsibilities
- Lead the technical design and delivery of the assigned Context and Harness workstreams
- Review context formats, ontology models, harness proposals, and integration approaches.
- Support pilot teams across MFBS divisions with context conversion, fixture authoring, and evaluation.
- Define how GitHub, Notion, and Slack content maps into the agent context engine.
- Establish guidelines for federated context routing and token-budget allocation across sources.
- Contribute to an ontology proof of concept covering one domain and one workflow using YAML-based schemas.
- Validate integration with internal or external agent platforms, including access and smoke testing.
- Capture pilot evidence and contribute to the standard and company best-practices package.
Requirements
Required skills
Context engineering skills
- RAG and retrieval pipelines:Design advanced search and retrieval systems that supply models with accurate, relevant, and timely information.
- Context-window and token management:Compress conversation history, prioritize useful context, and filter noise to optimize model attention and cost.
- Memory architecture:Build short-term and persistent memory systems that allow agents to retain state across tasks and interactions.
- Instruction curation:Structure plain-English operating rules, Markdown files, domain guides, and other instructions for dynamic injection into agent tasks.
- Context formats and knowledge modeling:Design context layers and reusable knowledge assets using Markdown, YAML, schemas, and related formats.
Harness engineering skills
- Tool and MCP integration: Connect models to external APIs, execution environments, tools, and Model Context Protocol (MCP) servers.
- Sandboxing and permissions: Define safety boundaries, access controls, approval flows, and secure execution environments for agent actions.
- Validation and guardrails: Implement linters, automated tests, evaluations, and verification layers that detect hallucinations, policy violations, or broken code.
- Orchestration and control loops: Design multi-step workflows, retry logic, state transitions, and error-correction loops for long-horizon tasks.
- Evaluation harnesses: Build fixtures, benchmark tasks, quality gates, and CI-integrated evaluations using DeepEval, Promptfoo, or comparable frameworks
Supporting technical and leadership skills
- LLM and agent fundamentals: Strong understanding of agent workflows, RAG, evaluation methods, and quality gates for AI-generated output.
- Software engineering:Strong Git/GitHub and CI practices, experience with repo-native tooling, and the ability to prototype quickly.
- Technical leadership: Ability to make pragmatic architecture decisions, lead POCs, and turn evidence into reusable standards.
- Communication: Ability to write clear technical proposals and provide constructive cross-team review and feedback.
- Experience with ontology or knowledge modeling using YAML-based schemas.
- Familiarity with internal or external agent platforms.
- Experience with hybrid search, indexing pipelines, or context freshness mechanisms.
Success outcomes
- Context architecture, format, and evaluation standards are validated through representative pilots.
- Pilot teams can convert source knowledge and author evaluation fixtures using documented playbooks.
- A working repo-native harness POC demonstrates repeatable benchmark execution and quality gates.
- The committee has sufficient evidence to ratify the standard and decide whether to adopt or build the long-term harness.
Skills Required
- Advanced experience designing RAG and retrieval pipelines
- Experience with context-window and token management
- Experience building short-term and persistent memory architectures
- Experience structuring instructions and knowledge assets using Markdown, YAML, and schemas
- Experience integrating tools, external APIs, execution environments, and MCP servers
- Experience defining sandboxing, permissions, access controls, approval flows, and secure execution environments
- Experience implementing validation, automated tests, evaluations, guardrails, and verification layers
- Experience designing orchestration, retry logic, state transitions, and error-correction loops
- Experience building evaluation harnesses, fixtures, benchmark tasks, quality gates, and CI-integrated evaluations using DeepEval, Promptfoo, or comparable frameworks
- Strong understanding of LLMs, agent workflows, RAG, evaluation methods, and quality gates
- Strong Git, GitHub, CI, and repo-native tooling experience
- Ability to make architecture decisions, lead proofs of concept, and create reusable standards
- Ability to write technical proposals and provide cross-team review and feedback
- Experience with ontology or knowledge modeling using YAML-based schemas
- Familiarity with internal or external agent platforms
- Experience with hybrid search, indexing pipelines, or context freshness mechanisms
Money Forward, Inc. Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Money Forward, Inc. and has not been reviewed or approved by Money Forward, Inc..
-
Healthcare Strength — Health coverage is considered comprehensive, with company-paid checkups, gynecological exams, flu vaccinations, and 24/7 medical advice including telemedicine and house-call options.
-
Flexible Benefits — Work flexibility is highlighted through remote work for all employees, flextime or discretionary work systems, and shorter-hours options that support varied needs.
-
Equity Value & Accessibility — Equity participation is described as attractive, featuring an employee stock ownership plan with a company incentive equal to 15% of contributions and tax-advantaged savings options.
Money Forward, Inc. Insights
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Money Forward, Inc. is a Japanese SaaS financial technology company that develops personal finance management tools and cloud-based services for individuals, businesses, accounting professionals, and financial institutions. Its platform supports household finance, accounting, tax, expense management, payments, and other back-office operations. The company serves consumer and corporate markets through integrated digital products and aims to operate as a comprehensive technology-driven financial platform.
