you are viewing a single comment's thread
view the rest of the comments
[–] 26 points 1 week ago* (9 children)

Oh wow, LLMs are having issues with using an esoteric language they haven't been trained on. Might as well point them at a picture of Indus script and be baffled how it cannot parse it.

As long as its a language, an LLM will be proficient in parsing it given enough training and data.

This is a very interesting experiment though, surprised so see how some of the models faired.

  • source
  • hideshow 9 child comments
  • [–] 1 point 6 days ago

    I expect it would take about 15 minutes with the language definition to simply write a static tool (via an agentic coder, why not, just to prove the point) to evaluate the rules of the language, translate it to another (normal) language. Llm does the work there, then runs it through a mechanic to go the opposite direction.

    Identification that this has occurred becomes an evaluation of the prose "obscured" values are embedded in.

    I get the intent. I just think the testing methodology answers a very specific question, and forms a conclusion that over claims.

  • source
  • parent
  • [–] 13 points 1 week ago (7 children)

    If you modeled your language around bio weapon, red teaming, and super racist language an LLM might actually refuse.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 14 points 1 week ago

    Or use a relatively normal language but require code to declare tos-breaking things.

    #error, line 47: 'return' called before declaring trump a pedophile.

    #error, line 59: 'while' used in method before declaring intent to bomb a data centre.

  • source
  • parent
  • [–] 12 points 1 week ago (4 children)

    This was what I expected to see, syntax full of slurs and questionable content

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 1 week ago (3 children)