AI Agent Quality Engineer

Posted 9 Hours Ago
Be an Early Applicant
Taguig, Southern Manila District, National Capital Region, PHL
In-Office
Senior level
Cloud • Security • Software • Cybersecurity
The Role
Lead quality and red-team testing for AI agents by building automated Python evaluation harnesses, adversarial prompt-injection and jailbreak tests, guardrail catalogs, CI/CD pipelines, and fleet-wide drift monitoring. Define automated pass/fail thresholds, validate tool boundaries and data exposure, produce audit-ready evidence, and block releases when agents fail security, resilience, or compliance standards.
Summary Generated by Built In
Join the Future of Security at Netskope

Netskope (NASDAQ: NTSK) is a leader in modern security and networking for the cloud and AI era. We secure and accelerate cloud, data, and AI in real time, everywhere. Thousands of customers, including more than 30 of the Fortune 100, trust the Netskope One platform, its Zero Trust Engine, and the powerful NewEdge network to gain full visibility and control without performance trade-offs.

At Netskope, our technology is driven by our greatest strength: our people. We believe that belonging powers innovation, and success is both personal and organizational. We embrace differences in gender, ethnicity, beliefs, ability, and identity, creating an environment where every voice is heard and respected. We empower our employees to bring their authentic selves to work, grow their careers through continuous education and mentorship, and lead with transparency and curiosity. Join a team where you belong, where you are encouraged to be an entrepreneur, and where together, we continue to redefine the landscape of security.

Visit Careers at Netskope to learn more. Follow us on LinkedIn and Instagram.

As a Senior AI Quality & Red Team Engineer at Netskope, you will lead the charge in testing, stress-testing, and breaking our AI agents before they ever reach production. From automating multi-turn prompt injections to tracking fleet-wide drift in CI/CD, you will own the automated harness that ensures our AI systems are secure, resilient, and compliant. If you love the idea of being the person who proves an agent isn't ready yet, welcome home.
 
Skills and competencies:
  • Build and grow the automated evaluation suite every agent runs against before it's approved for production, designed to run unattended and scale across a growing agent fleet, not something that needs a person babysitting each run.
  • Design adversarial test scenarios — prompt injection attempts, sycophancy checks where an agent has to correctly push back on a false premise, multi-attempt attacks rather than single-shot ones — and automate them so they run on every relevant change, not just before a big release.
  • Own the "break it on purpose" pass for every new agent: attempt to extract data it shouldn't expose, get it to act outside its registered tool boundaries, or get it to treat a synthetic test probe as real. As the fleet grows, build this into a repeatable, scriptable process rather than a manual exercise redone from scratch each time.
  • Partner with the Data Steward on data sensitivity classification for the systems agents touch, so your test scenarios reflect what's actually at stake, not a generic checklist.
  • Decide, for each agent capability, what "pass" actually means, and build that judgment into automated thresholds wherever possible so evaluation keeps up as the number of agents climbs into the hundreds.
  • Maintain the guardrail and negative-test catalog (fail-closed vs. fail-graceful behavior) across the platform, and add new cases as new failure modes get discovered in the wild.
  • Produce clear, audit-ready evidence for every agent's evaluation results, generated automatically as part of the pipeline rather than assembled by hand for each review.
  • Track drift over time across the whole fleet, not agent by agent, so a slow-moving problem in one corner doesn't go unnoticed just because no one's looking at that specific agent that week.
Must-Have:
  • At least 4 years in software quality, security testing, or a related discipline, with 1–2 years specifically evaluating or red-teaming LLM-based systems — not just running unit tests against traditional code.
  • Strong Python skills, since the evaluation harness, adversarial test scripts, and automated pipelines will mostly be built in it. Comfortable writing production-quality code, not just glue scripts.
  • Working knowledge of REST APIs and webhook/event-driven patterns, enough to build test harnesses that call an agent's tools directly and validate its inputs and outputs, not just its final chat response.
  • Real experience building automated test infrastructure and integrating it into CI/CD, not just manually running test cases — someone who thinks in pipelines and repeatability by default.
  • Hands-on familiarity with at least one adversarial testing or LLM eval tool (DeepTeam, Garak, PyRIT, Promptfoo, or similar), and an understanding of how these map to standards like the OWASP LLM Top 10 or NIST's AI risk framework.
  • Practical understanding of prompt injection, jailbreaking, and sycophancy failure modes — able to design new test cases for these, not just run ones someone else wrote.
  • Comfort making a hard call: willing to block a release when an agent doesn't meet its bar, even under schedule pressure.
Strong Advantage:
  • Enough understanding of data sensitivity and compliance classification to design tests that reflect real risk, even though the Data Steward owns the classification system itself.
  • Experience in a regulated environment where evaluation results had to hold up to an external audit, not just an internal review.
  • Familiarity with multi-attempt or persistent-attack testing methodology, rather than only single-shot adversarial prompts.
  • Some exposure to how agents are actually built (prompting, tool schemas, orchestration) — not required, but it makes it much easier to design tests that target real failure modes instead of generic ones.

#LI-CV1

Netskope is committed to implementing equal employment opportunities for all employees and applicants for employment. Netskope does not discriminate in employment opportunities or practices based on religion, race, color, sex, marital or veteran statues, age, national origin, ancestry, physical or mental disability, medical condition, sexual orientation, gender identity/expression, genetic information, pregnancy (including childbirth, lactation and related medical conditions), or any other characteristic protected by the laws or regulations of any jurisdiction in which we operate.

Netskope respects your privacy and is committed to protecting the personal information you share with us, please refer to Netskope's Privacy Policy for more details.

The application window for this position is expected to close within 50 days. You may apply by filling out the below information, or visiting our Netskope Careers site.

Skills Required

  • At least 4 years of experience in software quality, security testing, or a related discipline.
  • 1–2 years specifically evaluating or red-teaming LLM-based systems.
  • Strong Python skills and ability to write production-quality code.
  • Working knowledge of REST APIs and webhook or event-driven patterns.
  • Experience building automated test infrastructure and integrating it into CI/CD.
  • Hands-on familiarity with an adversarial testing or LLM evaluation tool such as DeepTeam, Garak, PyRIT, or Promptfoo.
  • Understanding of OWASP LLM Top 10 or NIST AI risk frameworks.
  • Practical understanding of prompt injection, jailbreaking, and sycophancy failure modes.
  • Willingness to block releases when AI agents fail quality or security standards.
  • Understanding of data sensitivity and compliance classification.
  • Experience in a regulated environment with audit-ready evaluation results.
  • Familiarity with multi-attempt or persistent-attack testing methodology.
  • Exposure to AI agent construction, including prompting, tool schemas, or orchestration.

Netskope Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Netskope and has not been reviewed or approved by Netskope.

  • Fair & Transparent Compensation — Pay is considered competitive in U.S. technical roles, with indications that total compensation benchmarks strongly versus peers. Employees are generally pleased with total compensation that includes pay, stock, equity, and benefits.
  • Healthcare Strength — Health coverage offers multiple medical plan choices alongside dental, vision, disability, HSA/FSA, and EAP, and is often characterized as very good. Employer-verified plan options and regional choices reinforce the breadth and quality of coverage.
  • Leave & Time Off Breadth — Time off includes flexible/unlimited PTO, paid holidays, and quarterly wellness days that support rest and flexibility. Paid parental leave is available, commonly cited at 12 weeks for birth parents and 8 weeks for non‑birth parents.

Netskope Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Santa Clara, CA
1,479 Employees
Year Founded: 2012

What We Do

Netskope, the SASE leader, safely and quickly connects users directly to the internet, any application, and their infrastructure from any device, on or off the network. With CASB, SWG, and ZTNA built natively in a single platform, the Netskope Security Cloud provides the most granular context, via patented technology, to enable conditional access and user awareness while enforcing zero trust principles across data protection and threat prevention everywhere. Unlike others who force tradeoffs between security and networking, Netskope’s global security private cloud provides full compute capabilities at the edge. Netskope is fast everywhere, data-centric, and cloud-smart, all while enabling good digital citizenship and providing a lower total-cost-of-ownership.

Similar Jobs

Optum Logo Optum

Forward Deployed Engineer

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Makati City, Metro Manila, National Capital Region, PHL
160000 Employees

Capital One Logo Capital One

Sr. Risk Manager, Control Testing

Fintech • Machine Learning • Payments • Software • Financial Services
Remote or Hybrid
Metro Manila, PHL
55000 Employees

Pfizer Logo Pfizer

Execution Hub Sr. Associate

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Hybrid
Makati City, Metro Manila, National Capital Region, PHL
121990 Employees

Smartly Logo Smartly

Managed Service Support

AdTech • Artificial Intelligence • Digital Media • Marketing Tech • Social Media • Software • Generative AI
Easy Apply
Remote or Hybrid
Philippines
805 Employees

Similar Companies Hiring

Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account