Preserving CR with LF when converting text files to character codes

Nice to know
1

When using SWI-Prolog on Windows and reading text files for conversion into characters codes and needing to preserve the CR with LF some special attention is needed.

Normally a file would be opened as a text file and read but that results in the CR with LF being converted to just a LF.

To preserve the CR with LF use set_stream/2 with newline(posix).

Example code

to_codes(File_path) :-
    setup_call_cleanup(
        open(File_path,read,Stream),
        (
            set_stream(Stream, newline(posix)),
            read_stream_to_codes(Stream,Codes),
            format('Codes: ~w~n',[Codes])
        ),
        close(Stream)
    ).

Example file

image
which is

; test followed by cr with lf.

Example run

?- to_codes('C:/Users/Groot./test.abnf').
Codes: [59,32,116,101,115,116,13,10]
true.

A case in point where preserving CR with LF is needed is when working with ABNF files.

ABNF (Augmented BNF) is the BNF used for IETF (Internet Engineering Task Force) RFPs (Request For Proposal) and documented in RFP 5234.

There are several good but dated references regarding SWI-Prolog and ABNF (search)

2

type(binary) and encoding(octet) are, AFAIK, equivalent. ISO only defines type. SWI-Prolog also has encoding.

If you want to read a text file using some encoding and preserving CR/LF you can use the normal open and then use set_stream(In, newline(posix)) to disable the CR/LF --> LF translation.

1 Like
3

Thanks. :grinning: I will fix the first post.