LLM generated code with hallucinated predicate options

An LLM generated SWI-Prolog code for a predicate with options, including a fictional option access(readonly) used with db_attach/2 from the library(persistency).

e.g. db_attach('example.pl', [access(readonly)]).

The issue was only discovered while manually debugging some failing tests.

This is noteworthy because many SWI-Prolog predicates silently ignore invalid options — in this case, the code compiles and runs without error, potentially misleading users into thinking the database is opened in read-only mode.


The issue emerged when using code coverage (via library(prolog_coverage)) alongside a comprehensive set of unit tests (in a *.plt file, also generated by the LLM), while aiming for 100% code coverage of a knowledge base built with library(persistency). Without striving for full coverage, the problem likely wouldn’t have been detected so quickly.

2 Likes