↓ Skip to main content
  1. Posts/

Measuring Coding Tool Effectiveness

··520 words·3 mins
Author
Hairizuan Noorazman
Software engineering experiments, implementation notes, and lessons learned.

Most online content regarding AI coding tools focuses heavily on input and output token counts. While these metrics are useful for understanding the raw volume of data processed, they often fail to address the actual effectiveness of those tokens in solving real-world engineering problems. Measuring the true impact of these tools on development workflows remains a challenge because volume does not equate to value.

Understanding DORA Metrics
#

To measure engineering effectiveness, many organizations turn to DORA metrics (DevOps Research and Assessment). These are four key indicators that have become the industry standard for measuring software development and delivery performance:

  1. Deployment Frequency: How often your organization successfully releases to production.
  2. Lead Time for Changes: The time it takes for a commit to reach production.
  3. Change Failure Rate: The percentage of deployments causing a failure in production.
  4. Failed Service Recovery Time: How long it takes to restore service when a production failure occurs.

While DORA metrics provide a high-level view of team performance and stability, they don’t explicitly account for the cost or efficiency of the tools used to achieve those results.

Moving Beyond Raw Tokens: Attributed Usage
#

One potential approach to bridge this gap is to combine DORA metrics with a new measure: attributed token usage linked directly to issue trackers.

Currently, we know how many tokens we use globally, but we rarely know what they were used for. By attributing token consumption to specific Jira tasks, GitHub issues, or pull requests, we can begin to see the “cost of completion” for different types of work.

For example, we could track how many tokens (and their associated cost) are consumed to resolve a specific bug or implement a feature. If a particular bug costs $6-$8 in tokens to solve, that provides a tangible data point. This isn’t just about the financial cost; it’s about the “cognitive load” the AI is carrying to understand and solve that specific problem.

Cost as a Signal for Investigation
#

On a higher level, this attribution allows teams to identify if certain types of issues or specific parts of the codebase are incurring disproportionately high costs.

  • Complex Legacy Code: If a relatively simple bug in a legacy module requires $20 of tokens to “explain” the context to the AI, it’s a strong signal that the code is too complex and might be a candidate for refactoring.
  • Poorly Defined Requirements: A high token-to-resolution ratio on new features might indicate that the requirements are ambiguous, leading the AI (and the developer) to iterate through many failed attempts.
  • Tool Inefficiency: It helps us evaluate whether a specific AI tool or model is actually effective for our specific tech stack compared to its cost.

While this won’t be a perfectly accurate measure of complexity—AI costs fluctuate and models evolve—it serves as a valuable signal. If an issue’s token cost spikes, it’s a prompt for a human lead to investigate whether the tool is struggling with that specific context or if the problem itself is fundamentally flawed. By layering these cost insights over DORA’s velocity and stability metrics, we get a much clearer picture of our true engineering efficiency.

Related

Fitting a 12B Model into Two Small CPU Nodes: Gemma 4 on Graviton with vLLM

In the last post I argued that you should not split a model that fits on one machine: the 1B model I benchmarked was faster, steadier and cheaper in a single pod. The flip side of that argument is the case where splitting is not a choice. When the weights do not fit, pipeline parallelism is how you run the model at all.

Don't Split a Model That Fits: One Pod vs Two for Gemma on CPU

In the last post I split a 1B-parameter Gemma model across two CPU nodes on purpose, to learn how LeaderWorkerSet and vLLM’s multi-node mode behave. I said then that splitting a model that fits on one machine makes it slower, and I listed “compare the split against a single replica” among the things I had not done.

Serving Gemma Across Two Nodes with vLLM and LeaderWorkerSet, on CPU

I wanted to try two things at once: Kubernetes’ LeaderWorkerSet (LWS), the API for running a group of pods as one unit, and vLLM’s multi-node serving. The cheapest way to exercise both is not a GPU cluster. It is two small CPU nodes, one small model, and a deliberate decision to split that model across the nodes even though it fits on one.