The harness is the product. Individual
agents should be quick to build because the hard parts are solved once: model
access, tool and data connectors, memory, permissions, evaluation,
observability and human approval.
• Define the reference
architecture for agents: orchestration loop, tool calling, context management,
memory, sub-agents and human-in-the-loop checkpoints.
• Build the harness as a reusable
internal platform with SDKs, templates and documentation that other engineering
teams can adopt.
• Design the tool and connector
layer, including Model Context Protocol (MCP) servers for internal systems
(CLM, CRM, ticketing, document stores, SAP sandboxes).
• Own the evaluation framework:
test sets, automated graders, regression gates in CI, and online quality
monitoring.
• Set security and governance
controls: identity and permission propagation, sandboxed code execution,
prompt-injection defenses, data classification, audit logging.
• Make model choices and routing
decisions on quality, cost and latency; keep the harness model-agnostic.
• Lead delivery of the first two
or three internal applications end to end, from discovery with business owners
to production.
• Set engineering standards,
review designs and mentor engineers moving into agent development.
• Track cost, usage and value per
agent, and report outcomes to leadership.
Area
What
the architect must be able to do
Priority
Agent architecture
Design agent loops (plan, act, observe),
multi-agent and sub-agent patterns, workflows vs. autonomous agents, and know
when an agent is the wrong tool
Must have
LLM engineering
Prompt and system-prompt design,
structured outputs, tool/function calling, context-window management,
caching, streaming, model routing across providers
Must have
Tools and integration
Build tool schemas and MCP servers; wrap
REST, OData and SOAP APIs safely; handle auth (OAuth 2.0, service accounts)
and rate limits
Must have
Retrieval and data
RAG design: chunking, embeddings, hybrid
search, reranking, vector stores; document parsing (PDF, Word, scanned
contracts); permission-aware retrieval
Must have
Evaluation
Build eval sets from real tasks,
LLM-as-judge and rule-based graders, regression suites in CI, A/B and online
metrics
Must have
Safety and security
Prompt-injection and data-exfiltration
defenses, least-privilege tools, sandboxed code execution, approval gates for
side-effecting actions, PII handling
Must have
Observability
Tracing of every model and tool call,
cost and latency dashboards, replay of failed runs, feedback capture
Must have
Backend engineering
Production Python and/or TypeScript;
async and event-driven design; queues, workflow engines, idempotency and
retries
Must have
Cloud and platform
Containers, Kubernetes or serverless,
infrastructure as code, secrets management, CI/CD on AWS, Azure or GCP
Must have
Memory and state
Short- and long-term memory design,
durable state for long-running agents, checkpointing and resumption
Strong plus
Agent frameworks
Hands-on with at least one framework
(e.g. LangGraph, Claude Agent SDK, OpenAI Agents SDK, Semantic Kernel) and
the judgment to build thin abstractions instead of locking in
Strong plus
Application layer
Chat, copilot and background-agent UX;
human review queues; embedding agents in Slack, Teams, browser or existing
apps
Strong plus
Cost management
Token budgeting, model tiering, caching,
batch processing, unit economics per agent
Strong plus
Fine-tuning and model ops
When to fine-tune vs. prompt vs.
retrieve; open-weight model hosting
Nice to have
Experience requirements
- 10+ years in software engineering, with at least 4 years in an architect or principal engineer role owning platform-level designs.
- 2+ years building LLM-based systems that reached production use, including at least one agentic system with tool calling and real side effects (not only chat or RAG demos).
- Has built an internal platform or framework adopted by other teams, and can show how adoption was earned (SDKs, docs, templates, support).
- Has designed an evaluation approach that caught regressions before release, and can explain how its graders were validated.
- Has run production systems with on-call responsibility, incident reviews and cost accountability.
- Has worked in an enterprise environment with SSO, data classification, security reviews and audit requirements.
- Bachelor’s or master’s in computer science or equivalent practical experience.
Nice-to-have and domain experience
- Enterprise integration: SAP S/4HANA or SAP Ariba APIs, SAP Integration Suite, Jitterbit or another iPaaS.
- Contract lifecycle management, procurement or legal tech: clause extraction, obligation tracking, document comparison.
- Document AI: OCR, layout-aware parsing, table extraction from contracts and PDFs.
- Workflow engines (Temporal, Step Functions or similar) for long-running agents.
- Experience with regulated customers (financial services, life sciences) and their AI governance requirements, e.g. ISO 42001, SOC 2, GDPR.
- Open-source contributions or published writing on agent design or evaluation.
Leadership and working style
- Builder first. Writes production code, not only diagrams; prototypes quickly, then hardens what works.
- Evidence over hype. Decides with evals and measurements, and is comfortable saying a use case does not need an agent.
- Platform mindset. Designs for the tenth agent, not just the first; treats internal teams as customers.
- Clear communicator. Explains trade-offs to engineers, security, legal and executives in their own terms.
- Security-minded. Assumes agents will be attacked through their inputs and designs accordingly.
- Mentor. Raises the team’s skill in a field where most engineers are still learning.
Skills Required
- 10+ years of software engineering experience
- At least 4 years in an architect or principal engineer role owning platform-level designs
- At least 2 years building production LLM-based systems
- Production experience with an agentic system using tool calling and real side effects
- Experience building an internal platform or framework adopted by other teams
- Experience designing evaluation approaches that catch regressions before release
- Production operations experience including on-call responsibility, incident reviews, and cost accountability
- Enterprise experience with SSO, data classification, security reviews, and audit requirements
- Bachelor's or master's degree in computer science, or equivalent practical experience
- Production Python and/or TypeScript development
- Experience with agent architecture, LLM engineering, tools and integrations, retrieval, evaluation, safety, observability, backend engineering, and cloud platforms
- Experience designing memory and durable state for long-running agents
- Hands-on experience with an agent framework such as LangGraph, Claude Agent SDK, OpenAI Agents SDK, or Semantic Kernel
- Experience with chat, copilot, background-agent interfaces and human review workflows
- Experience with token budgeting, model tiering, caching, batch processing, and agent unit economics
- Experience deciding between fine-tuning, prompting, retrieval, and open-weight model hosting
- Enterprise integration experience with SAP S/4HANA, SAP Ariba, SAP Integration Suite, Jitterbit, or another iPaaS
- Contract lifecycle management, procurement, legal technology, or document AI experience
- Experience with OCR, layout-aware parsing, table extraction, or contract document processing
- Experience with Temporal, Step Functions, or similar workflow engines
- Experience with regulated customers and AI governance requirements such as ISO 42001, SOC 2, or GDPR
- Open-source contributions or published writing on agent design or evaluation
What We Do
Execo is a next-generation legal services provider that closes the gap between legal AI investment and real results by delivering AI-powered services for contracts, compliance, and legal operations, blending technology with human expertise.









