Business Analyst II

Microsoft AI is a customer-first organization focused on building AI-powered products that help customers and businesses achieve more in an evolving technology landscape. As AI reshapes how people discover information, engage with services, and conduct business, we need product leaders who can navigate ambiguity, bring clarity to complex challenges, and drive meaningful customer and business impact.

As a member of the Microsoft AI Monetization Platform Trust & Safety team, you will play a critical role in ensuring that our products and platforms remain safe, trustworthy, and effective. You will work on some of the most important challenges facing modern digital ecosystems, including trust and safety, fraud and abuse prevention, risk management, operational excellence, quality systems, automation, and AI-driven decision making. In this role, you will partner closely with Engineering, Data Science, Research, Operations, Policy, Risk, and Business stakeholders to define and deliver scalable solutions that protect users, customers, advertisers, and the broader ecosystem while enabling sustainable business growth. You will help shape how Trust & Safety systems operate, how risk is measured and managed, and how emerging technologies such as AI and agentic systems can be leveraged to improve outcomes at scale.

• Has experience in Trust & Safety, Integrity, Fraud Prevention, Abuse Prevention, Risk Management, Policy, Security, Compliance, Operations, Analytics, or related domains.
• Demonstrates strong ownership and a bias for action, consistently driving measurable impact through data-driven decision making.
• Operates effectively across complex cross-functional environments, balancing customer, business, operational, policy, and technical considerations.
• Excels at translating ambiguous problems into scalable solutions and influencing outcomes across diverse stakeholder groups.
• Possesses strong analytical capabilities and uses data to identify opportunities, risks, operational improvements, and investment priorities.
• Is energized by fast-moving environments and enjoys solving complex ecosystem-wide challenges at scale.

You will work within the Microsoft AI (MAI) organization, powering AI-first experiences across products including Copilot, Bing, Edge, MSN, and future AI-driven services. MAI brings together Product, Engineering, Research, Design, and Go-To-Market teams to deliver innovative solutions at global scale. As part of a highly collaborative, cross-functional team, you will partner with engineers, designers, researchers, operators, analysts, marketers, and business leaders to build impactful customer experiences. We operate with a strong sense of ownership, customer obsession, and data-driven decision making in a rapidly evolving AI landscape.

As AI transforms the digital ecosystem, this role offers a unique opportunity to help define how trust, safety, quality, and operational excellence evolve in an increasingly agentic world.

Microsoft's mission is to empower every person and every organization on the planet to achieve more. Our employees embrace a growth mindset, innovate to empower others, collaborate to realize shared goals, and contribute to a culture grounded in respect, integrity, accountability, and inclusion. This role is ideal for a builder who thrives in ambiguity, enjoys rapidly turning ideas into prototypes and scalable solutions, and is passionate about leveraging data, automation, AI, and operational excellence to improve Trust & Safety outcomes at global scale.

• Partner with Engineering, Data Science, Operations, Policy, and Risk teams to define and scale quality measurement frameworks across Trust & Safety systems.
• Design and manage evaluation programs, benchmarking methodologies, calibration processes, and quality standards for human reviewers and AI-powered systems.
• Define labeling strategies, annotation guidelines, adjudication processes, and ground-truth creation methodologies to support high-quality model development and evaluation.
• Perform hands-on labelling, content review, evaluation, and adjudication activities as needed to establish ground truth, validate evaluation methodologies, calibrate quality standards, and improve model performance.
• Establish quality metrics, scorecards, benchmarks, and performance indicators to measure reviewer effectiveness, model quality, and business impact.
• Generate actionable insights from labeling, evaluation, quality, audit, and operational datasets to identify risks, gaps, quality issues, and improvement opportunities.
• Partner with Engineering and Data Science teams to improve model performance through evaluation design, error analysis, feedback loops, and continuous quality improvement programs.
• Identify emerging quality risks, annotation inconsistencies, policy ambiguities, and model failure patterns and recommend corrective actions.
• Design experimentation and validation frameworks to assess the effectiveness of model, policy, and operational improvements.
• Drive cross-functional initiatives from problem identification through execution, balancing quality, operational efficiency, customer impact, and business objectives.
• Communicate quality insights, recommendations, and evaluation outcomes to technical and non-technical stakeholders.
• Embody Microsoft’s Culture and Values.

Required Qualifications:

• Bachelor's degree in business, Public Policy, Communications, Operations, Economics, Psychology, Social Sciences, Data Science, Analytics, Computer Science, or a related field, or equivalent practical experience.
• 3+ years of experience in Product Management, Program Management, Trust & Safety, Quality Operations, Risk Management, Fraud Prevention, Content Moderation, Analytics, Marketplace Integrity, or related domains.
• Experience designing, managing, or executing labeling, evaluation, quality assurance, benchmarking, calibration, annotation, adjudication, or review programs.
• Strong analytical and problem-solving skills with experience leveraging data to identify quality issues, generate insights, and drive operational or product improvements.
• Experience working across cross-functional teams including Engineering, Data Science, Operations, Policy, and Business stakeholders.
• Demonstrated ability to translate ambiguous quality challenges into scalable evaluation frameworks, quality metrics, operational processes, and actionable recommendations.
• Familiarity with AI/ML-powered systems and the role of labeling, human review, evaluations, and feedback loops in improving model quality and safety outcomes.
• Experience evaluating products, models, processes, or workflows using data, experimentation, metrics, user feedback, and operational performance indicators.
• Strong communication, stakeholder management, and influencing skills.

Preferred Qualifications:

• Experience building, deploying, or scaling AI-powered workflows, intelligent assistants, automation platforms, agents, or workflow orchestration systems.
• Experience applying Generative AI, Large Language Models (LLMs), prompt engineering, retrieval-augmented generation (RAG), workflow automation, or agentic frameworks to real-world business problems.
• Working knowledge of Python, SQL, Power BI, Power Platform, Azure AI services, or similar analytics and automation technologies.
• Experience prototyping solutions, building proofs of concept, or rapidly validating ideas through experimentation and iterative development.
• Experience designing evaluation methodologies, feedback loops, telemetry frameworks, or quality measurement systems for AI-powered products and workflows.
• Experience partnering with Engineering and Data Science teams to improve AI systems through experimentation, user feedback, analytics, and continuous improvement.
• Experience working in Trust & Safety, Marketplace Integrity, Fraud Prevention, Risk Management, Compliance, Operations, or related domains.
• Demonstrated builder mindset with a track record of transforming manual workflows into scalable, automation-first solutions.
• Proven ability to thrive in fast-paced, ambiguous environments while driving measurable customer and business impact through innovation and continuous learning.
• Hands-on experience performing labelling/data annotation, prompt writing and evaluation, model assessment, calibration reviews, or adjudication activities to support model development and quality improvement.
• Experience creating high-quality ground-truth datasets, benchmark sets, and evaluation methodologies used for model training, testing, and validation.
• Familiarity with prompt design, evaluation rubrics, benchmarking methodologies, and quality measurement frameworks for AI-powered systems.

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.