Package fine_upload corrupts files

General
1

Hi

The package fine_upload allows files to be uploaded from a web page to a SWI web server:
https://www.swi-prolog.org/pack/list?p=fine_upload

The UI works fine, but under Windows at least the uploaded files are corrupted (bytes missing). This can be demonstrated with the demo that comes with the package.

The attached file:
Prolog-logo-horizontal-120

is 28,029 bytes but the file created by the upload is only 27,893 bytes

Directory of c:\Users\Mike Elston\temp\fine_upload-0.1.5\uploads

15/03/2022 12:05 pm .
15/03/2022 12:05 pm …
15/03/2022 12:05 pm 27,893 3f866b80-fd6c-4a56-928d-5779e4e0470c.png
1 File(s) 27,893 bytes

Here is the console output:

Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.8)
<snip>
% [Thread httpd@8080_3] Params so far: [qquuid='3f866b80-fd6c-4a56-928d-5779e4e0470c',qqfilename='owl.png',qqtotalfilesize='28029',qqfile=file(_5936,_5938)]
% [Thread httpd@8080_3] IsMulti = false, Offset = 0
File 'owl.png' has been uploaded to 'uploads/3f866b80-fd6c-4a56-928d-5779e4e0470c.png' (27,893 bytes from ip(127,0,0,1))
% [Thread httpd@8080_3] Saved 'owl.png' in 'uploads/3f866b80-fd6c-4a56-928d-5779e4e0470c.png'
2

Tested the demo on Linux. That works fine. I guess we are either dealing with text encoding somewhere or something is going wrong while reassembling the parts due to some bug in the file handling on Windows. The critical predicate is save_file/4 in fine_uploader.pl. It has a debug topic. That may need some extension and possibly comparing to how it works on a non-Windows platform.

comparing the uploaded file with the original may also provide an indication on whether it is an encoding or offset problem.

3

Hi Jan

I created a test file with this:

open(‘uploads/egg.png’,write,S,[type(binary)]),forall(between(1,255,B),put_byte(S,B)),close(S).

The uploaded file is missing 0D

01 02 03 04 05 06 07 08 09 0a 0b 0c 0e 0f 10 11

4

Are you claiming simply writing a binary file is failing? Or did you create this file and uploaded it after which it was broken? A bit too lazy to start Windows. Under Wine, after running the above and examining the file using od -t x1, the file is correct.

5

Hi Jan

I created the file using the little Prolog program then uploaded it with fine_uploader. The uploaded file was one byte short, missing the 0D byte.

Cheers
Mike

6

od -t x1

For those like me who use Windows more than Unix/Linux.

See: od man page
od - dump files in octal and other formats

7

This looks like newline conversion happening (badly) where it shouldn’t. The Windows-style newline sequence is "\r\n" (0d 0a), whereas on Unix-likes it’s just "\n" (0a), so converting a text file from Windows to Unix style will remove 0d bytes. (Really, it should only replace the specific sequence 0d 0a with 0a, and not all 0d bytes - but at least this way the error is easier to spot I guess.)

I don’t know the code for SWI’s HTTP libraries very well, but it seems that http_chunked internally always opens its stream in text mode, which I assume will enable newline conversion on Windows:

2 Likes
8

Should be fixed with commit 494ea185c7090f6cf9e063113288be107b8bde81 for the http package. The problem was that switching back to octet mode in the multipart plugin didn’t clear the text property of the stream.

This first version of this patch also also activates setting the encoding we use for reading the file content to what the media type tells us. I have reverted that, so uploaded files should always be a binary copy. In theory we could try to recode received text files to the hosting system encoding. This will break the way fine_upload sends the file (in parts) and I doubt we can do so reliably anyway.

These are only the flags that it copies from the parent stream. This is surely unrelated as no chunked encoding is involved here (at least not for this test).

1 Like
9

Oops, I linked the wrong file - I meant multipart.c of course, which has a similar definition:

And now I see what you mean - the flags are just used as a bit mask, they are not set unconditionally on the new stream.

1 Like
10

Hi Jan

Thanks for the rapid fix. I went to pick up a daily Windows build to get the fix but they are a little out of date:

image.png

Cheers
Mike

11

The nightly build sometimes hangs, most likely due to some bug in Wine that causes the file system access to report errors due to overloading of the wineserver :frowning: Killed the hanging processes and restarted by hand. That seems to have done the job. I’m afraid this will happen again though. Not sure how to fix this …