This already works for predicates, so if I wrote atom//1 it gets a link to the correct spot. Is it possible to automate this for library mentions, so that if I typed in “library(dcg/basics)” this would get a link to https://www.swi-prolog.org/pldoc/man?section=basics?
If someone pointed me to the right direction I can try to do it. Or is it so that there is no clear 1-to-1 correspondence between the library name (as used inside use_module/1) and the documentation page?
That shouldn’t be too hard if we can make www.swi-prolog.org reply with the URL given such a pattern. As is though, I’m afraid there is no uniformity and thus this either requires cleaning up the documentation to make sure the library references are uniform or adding a table 
Yes, looking at the library names and the links I was afraid this was the case. If it isn’t something that you can easily explain how to proceed with, don’t bother. It is very easy to add the link manually and no one has missed this feature until now.
For reference though, here is the source for the plugin that does the auto-linkifying for the predicate names. As Jan says though, it requires cooperation from the SWI-Prolog site to send the URLs for a given predicate name.