Master Thesis: Observability and Analytics for AI-Assisted Development

Join our Team
About this opportunity
AI-assisted command-line development tools are changing how software is built. They can select different models, consume varying amounts of tokens and credits, compact conversation context, retry failed requests, and delegate work to sub-agents. However, developers often have limited visibility into what is happening behind the scenes-and how these behaviours affect cost, latency, reliability, and the overall user experience.
This thesis will explore how to build an analytics and observability framework for an AI-assisted command-line development tool. The framework will make the tool's internal behaviour measurable and understandable, helping developers see how tokens and credits are consumed and how different workflows influence performance and cost. The work will also investigate whether data-driven recommendations can improve the tool's efficiency without compromising the developer experience.
What you will do
The thesis will begin by identifying the most valuable signals and metrics for understanding AI-assisted development workflows. Our flag AI agent is Kiro-CLI. You will design and implement an instrumentation and analytics framework that captures relevant events while respecting privacy and usability requirements.
The framework may include measurements such as:
Model selection and model switching
Token and credit consumption
Context size and context-compaction events
Retries, failures, and recovery behaviour
Sub-agent creation, orchestration, and execution time
End-to-end and component-level latency
User interactions and perceived experience
You will then use the collected data to analyse common usage patterns, identify sources of unnecessary cost or delay, and develop actionable optimization recommendations. Potential recommendations could include selecting a more suitable model, improving context-management strategies, reducing redundant retries, or changing sub-agent orchestration policies.
Finally, you will validate selected recommendations through a controlled cohort or user study. The evaluation will investigate whether the recommendations reduce token or credit consumption, improve latency and reliability, and help developers better understand and control their AI usage.
The exact scope can be adapted based on your interests and the maturity of the tool. Possible areas of exploration include:
• Designing a useful observability model for agentic command-line tools.• Creating dashboards or in-tool feedback for token and credit consumption.• Comparing different strategies for context compaction.• Evaluating model-routing and model-selection policies.• Measuring the impact of retries and sub-agent orchestration on cost and latency.• Defining user-experience metrics for AI-assisted software development.
• Investigating privacy-preserving collection and aggregation of developer telemetry.
The skills you bring
Knowledge in Computer Science, Software Engineering, Data Science, or a related field
Experience with at least one programming language, preferably Python, JavaScript, TypeScript, or a similar language
Interest in observability, analytics, distributed systems, developer tools, or artificial intelligence.
Familiarity with data collection, visualization, experimentation, or statistical analysis is beneficial.
Ability to work independently, investigate open-ended problems, and communicate findings clearly.
Bonus skills
Experience with command-line tools or developer productivity platforms.
Experience with large language models, AI agents, or multi-agent systems.
Knowledge of telemetry, event-driven systems, metrics, tracing, or dashboard development.
Experience designing user studies or controlled experiments.
Why join Ericsson?At Ericsson, you'll have an outstanding opportunity. The chance to use your skills and imagination to push the boundaries of what's possible. To build solutions never seen before to some of the world's toughest problems. You'll be challenged, but you won't be alone. You'll be joining a team of diverse innovators, all driven to go beyond the status quo to craft what comes next.
What happens once you apply? Click Here to find all you need to know about what our typical hiring process looks like.Encouraging a diverse and inclusive organization is core to our values at Ericsson, that's why we champion it in everything we do. We truly believe that by collaborating with people with different experiences we drive innovation, which is essential for our future growth. We encourage people from all backgrounds to apply and realize their full potential as part of our Ericsson team. Ericsson is proud to be an Equal Opportunity Employer. learn more.
Primary country and city: Sweden (SE) || Lund
Req ID: 790878