73
Linux Kernel Developers Consider Adding AGENTS.md To Help Guide AI/LLM Agents (www.phoronix.com)
submitted 2 days ago by to c/technology@lemmy.world
33 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 22 hours ago (5 children)

To quote the research paper conclusion...

Conclusion We evaluate the impact of context files on coding agent performance for four common coding agents on SWE-BENCH and the novel CTXBENCH, built from recent GitHub issues and less popular repositories containing developer-written context files. We find that all context files consistently increase the cost and number of steps required to complete tasks. LLM-generated context files have a marginal negative effect on task success rates, while developer-written ones provide a marginal performance gain, neither statistically significant. Our trace analyses show that instructions in context files are generally followed and lead to more test- ing and broader exploration; however, they do not function as effective repository overviews. Over- all, our results suggest that context files don’t improve coding agent performance, and should only contain specific additional instructions beyond what is already available in the codebase. This high- lights a concrete gap between current agent-developer recommendations and observed outcomes, and motivates future work on principled ways to automatically generate concise, task-relevant guid- ance for coding agents.

That sounds like "Agents.MD doesn't work" to me.

Am I missing something?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 0 points 11 minutes ago

    your research paper:

    llm generated agent.md files are harmful. handwritten ones on average don't have an effect on ability to complete a task

    thread article:

    agents keep signing off on commits wrong. we added it to the agents.md and now they don't

    benchmarks notoriously don't measure the likelihood that the results would be merged by maintainers, just that they passed a test

    the old style "repo map" agents.md files that every tool used to create is no longer useful, but of course things that would otherwise be repeated prompts are still good. I cut the ones at work down from hundreds of lines each to tens. without those last few lines the amount of iteration I do - both with agent and when reviewing PRs - goes up significantly

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 20 hours ago (3 children)

    Agents.md doesn't exist to make sure generated code functions, it's there to make sure it's written well.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 17 hours ago (2 children)

    What is the difference between the models writing code "well" and their performance in this context? Are we referring to readability?

    Genuine question. If we use agents to read, edit, and review code, why do we care about readability? That's a human constraint. Unless attempting to do those three is not effective and thus requires human attention to correct issues which would justify readable code. If that's the case; why use the agent to edit the code in the first place?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 7 hours ago* (1 child)

    If that's the case; why use the agent to edit the code in the first place?

    Agents for generation, humans for review

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 hour ago*

    One, that sounds truly miserable.

    Two, there is a decent body of evidence to suggest that this method does not actually speed up development unless you go full lights out software factory which... Why would you want to do that?

    https://ide.mit.edu/insights/ai-productivity-and-roi/

    EDIT: As an aside this actually reminds me of the Xerox park study into efficiency gains for keyboard heavy workflows vice mouse heavy workflows. Keyboards were perceived as faster by subjects but when actually measured the mouse was faster.

  • permalink
  • fedilink
  • source
  • parent