Skills Without Evals Are Just Markdown and Hope

The analysis and experiments conducted on skills for AI agents like Claude reveal several critical insights regarding their effectiveness, cost, and limitations:

Key Insights

1. Encoded Preferences vs. Capability Uplift

   • Skills that teach an encoded preference (idiomatic ways of doing things) are more likely to have a longer shelf life than those providing new capabilities the model didn't previously know.
   • The evals showed that the skill was effective in guiding the model towards preferred idioms and patterns for Angular state management using @ngrx/signals.

2. Triggering Behavior

   • The description optimization loop revealed that no matter how the skill's description is rephrased, the agent still under-triggers on conversational queries.
   • This means that even if a skill provides valuable information, it may not be consulted frequently enough to justify its inclusion.

3. Cost Implications

   • Each invocation of a skill incurs additional time and token costs:
     • Time: 13.7 seconds increase per call
     • Tokens: 12,416 tokens increase per call
   • At the current pricing model ($3/M input, $15/M output), this translates to an extra cost

Read the full article at DEV Community

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.