Be an Early Applicant
Atlassian provides tools to help every team unleash their full potential.
The Role
Design, build, deploy, and operate evaluation and optimization systems for Rovo AI agents. Develop offline benchmarks, online quality metrics, model-based judges, and production monitoring. Use evaluation results to improve prompts, retrieval, tool use, routing, quality, cost, and latency. Lead projects from problem definition through delivery, contribute to technical reviews, coach engineers, and communicate ML trade-offs to stakeholders.
Summary Generated by Built In
Engineering | Sydney, Australia | Remote, Remote | Full-Time
Atlassian is seeking a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio. You'll build the systems that measure how well AI agents perform and use them to make those agents measurably better for thousands of companies.
Your future team
Studio is where Atlassian customers build, run and manage AI agents under the Rovo brand. Rovo Agents is a model-agnostic cloud agent platform. Teams give an agent instructions, knowledge, tools and skills, then put it to work in Jira and Confluence, inside automation rules, or as an autonomous agent that works on its own.
Because customers build these agents themselves, quality can't be checked once before release. Every agent has its own instructions, data and tools, and any of them can be misconfigured or connected to a tool that fails. We ask three questions of every agent: did it understand and follow the task, did it use its tools correctly and report honestly what it did, and did the result actually help?
The team is software and machine learning engineers based in Sydney and Melbourne, with remote options. We work closely with product, data science and Atlassian's central AI teams. This is applied ML on real agentic systems in production.
What you'll do
As a Senior Machine Learning Engineer, you will design, build, ship and operate the evaluation and optimization systems for Rovo agents.
Your background
On the first day, we'll expect you to have:
It's great, but not required, if you have:
Join the Atlassian Talent Community
Don't see an exact role match? No problem! Join our Talent Community and stay up-to-date on company and careers updates relevant to your career.
Join now
Atlassian is seeking a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio. You'll build the systems that measure how well AI agents perform and use them to make those agents measurably better for thousands of companies.
Your future team
Studio is where Atlassian customers build, run and manage AI agents under the Rovo brand. Rovo Agents is a model-agnostic cloud agent platform. Teams give an agent instructions, knowledge, tools and skills, then put it to work in Jira and Confluence, inside automation rules, or as an autonomous agent that works on its own.
Because customers build these agents themselves, quality can't be checked once before release. Every agent has its own instructions, data and tools, and any of them can be misconfigured or connected to a tool that fails. We ask three questions of every agent: did it understand and follow the task, did it use its tools correctly and report honestly what it did, and did the result actually help?
The team is software and machine learning engineers based in Sydney and Melbourne, with remote options. We work closely with product, data science and Atlassian's central AI teams. This is applied ML on real agentic systems in production.
What you'll do
As a Senior Machine Learning Engineer, you will design, build, ship and operate the evaluation and optimization systems for Rovo agents.
- Build offline evaluation suites and benchmarks that reflect realistic agent tasks, using real usage patterns, synthetic scenarios and simulated tools. Run them as regression tests before changes ship.
- Build online quality metrics on production traffic. This covers sampling strategy, cheap behavioral signals such as tool errors, retry loops, failed handoffs and abandoned sessions, and model-based judging of selected agent trajectories.
- Calibrate judges against human ratings and independent outcome signals, and evolve evals so they don't saturate as agents improve.
- Use evaluation results and production signals to improve agent quality across prompts, context selection, tool use, retrieval and agent routing, including finding the right agent for a request.
- Improve quality while reducing cost and latency across production LLM systems.
- Own projects from problem definition through experimentation, deployment and ongoing measurement. Contribute to design and code reviews, coach engineers, and help software engineers on the team write good evals.
Your background
On the first day, we'll expect you to have:
- A bachelor's or master's degree, preferably in Computer Science or a related field, or equivalent practical experience.
- 4+ years of related industry experience in machine learning, including hands-on experience designing, evaluating and operating LLM-based or agentic systems.
- Strong coding skills in Python and/or Kotlin, with the ability to write performant, production-quality code using AI
- Experience designing experiments, building evaluation datasets and benchmarks, and analyzing model and system behavior to guide improvements. Familiarity with SQL, distributed data processing and cloud data environments.
- Experience building and operating ML systems in production, including the practices needed to improve quality, reliability and cost.
- Regular use of AI tools in your daily engineering workflow, with the judgment to validate their outputs and take ownership of the results.
- A strong ownership mindset and the ability to lead ambiguous technical work from problem definition through delivery.
- Clear communication, with the ability to explain ML trade-offs to engineers, product managers and leaders.
It's great, but not required, if you have:
- Experience developing agent evaluations, including trajectory analysis, human evaluation or LLM-as-a-judge.
- Experience building agents that use tools, or retrieval-augmented generation applications.
- Experience with retrieval, ranking or recommendation.
- Experience in enterprise SaaS or similarly large-scale product environments.
Join the Atlassian Talent Community
Don't see an exact role match? No problem! Join our Talent Community and stay up-to-date on company and careers updates relevant to your career.
Join now
Skills Required
- Bachelor’s or master’s degree in Computer Science or a related field, or equivalent practical experience
- 4+ years of related industry experience in machine learning
- Hands-on experience designing, evaluating, and operating LLM-based or agentic systems
- Strong coding skills in Python and/or Kotlin
- Experience writing performant, production-quality code using AI tools
- Experience designing experiments, building evaluation datasets and benchmarks, and analyzing model and system behavior
- Familiarity with SQL, distributed data processing, and cloud data environments
- Experience building and operating ML systems in production, including quality, reliability, and cost optimization
- Regular use of AI tools in daily engineering workflows, with judgment to validate outputs and own results
- Strong ownership mindset and ability to lead ambiguous technical work from problem definition through delivery
- Clear communication skills for explaining ML trade-offs to engineers, product managers, and leaders
- Experience developing agent evaluations, including trajectory analysis, human evaluation, or LLM-as-a-judge
- Experience building tool-using agents or retrieval-augmented generation applications
- Experience with retrieval, ranking, or recommendation
- Experience in enterprise SaaS or similarly large-scale product environments
What the Team is Saying
Tiffany
Engineering Senior Associate, United States
Gautam
Software Engineer, United States
Tygrr
Product Manager, United States
Ashleigh
Frontend Engineer, United States
Jensen
Senior Frontend Software Engineer, United States
Javier
Senior Solutions Engineer, United States
Atlassian Compensation & Benefits Highlights
-
Parental & Family Support — Parental leave spans 26 weeks for birthing parents and 20 weeks for non-birthing parents, alongside inclusive family-formation benefits such as fertility treatment, adoption, and surrogacy. Additional supports include menopause resources and caregiving-related programs.
-
Retirement Support — Retirement programs include a U.S. 401(k) company match with immediate vesting and multiple contribution options. These are complemented by other financial wellbeing tools like HSAs/FSAs.
-
Healthcare Strength — Medical coverage is paired with mental-health resources, disability and life insurance, and options such as HSA/FSA. Descriptions also point to solid plan choices with access to counseling, coaching, and 24/7 support.
Atlassian Insights
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Atlassian creates teamwork solutions for high-performing teams. Our portfolio of collaboration and work management software products includes Jira, Confluence, Trello, Loom and Rovo. More than 300,000 businesses worldwide rely on Atlassian’s technology, including 80 percent of Fortune 500 companies. Our solutions support various business teams and they help organizations plan, track, and deliver their biggest ideas together.
Why Work With Us
At Atlassian, we believe we can accomplish so much more together than apart — which is why everything from our tooling — to our distributed workforce — to how our teams are structured is rooted in collaboration. Come join us and help unleash the potential of every team.
Gallery
.png)
.png)
Atlassian Teams
Atlassian Offices
Remote Workspace
Employees work remotely.
Atlassians have flexibility in where they work to support their family, personal goals, and other priorities. Our approach to distributed work allows us to tap into talent beyond our office locations, and to reimagine how work gets done.
Typical time on-site:
None
San Francisco, CA
Australia
Japan
Canada
Netherlands
Amsterdam, NL
Austin, TX
Austin, TX
Bellevue, WA
India
Boston, Massachusetts
Boston, MA
Chicago, IL
Poland
Mountain View, CA
New York, NY
Australia
Learn more
