dan comments on I built a programming language that LLMs can't use

[–] dan@upvote.au 53 points 6 days ago (6 children)

This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.

permalink

fedilink

source

hideshow 6 child comments

[–] OldMrFish@feddit.dk 31 points 6 days ago* (last edited 6 days ago) (1 child)

Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn't been trained on. That's basically an ad for Claude Code..

permalink

fedilink

source

parent

hideshow 1 child comment

[–] dan@upvote.au 7 points 6 days ago

Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.

permalink

fedilink

source

parent

[–] Mirodir@discuss.tchncs.de 22 points 6 days ago (3 children)

And that's without any of this being in its training set. I wonder if they realized how big of a self own that is.

permalink

fedilink

source

parent

hideshow 3 child comments

[–] aldhissla@beehaw.org 8 points 6 days ago

As well-intentioned as this might be, I doubt they understand how trained models work.

permalink

fedilink

source

parent

[+] atzanteol@sh.itjust.works -7 points 6 days ago (1 child)

How the hell is this a "self own"?

permalink

fedilink

source

parent

hideshow 1 child comment

[–] aldhissla@beehaw.org 7 points 6 days ago*

The premise was that this language is significantly different from other languages in terms of being understood and replicated by an LLM.

The result showed that an LLM does exactly as well with it as you would expect.

permalink

fedilink

source

parent