xpra icon
Bug tracker and wiki

Opened 2 years ago

Closed 15 months ago

Last modified 4 weeks ago

#1026 closed task (fixed)

file transfer improvements

Reported by: Antoine Martin Owned by: alas
Priority: major Milestone: 1.0
Component: core Version: trunk
Keywords: Cc:

Description

Split from #494, remaining tasks:

  • each file needs to be chunked so we can interleave regular xpra traffic - high priority
  • ability to cancel transfers in progress
  • client option of showing a dialog to accept files before starting the transfer (and choose download location?)

Attachments (3)

file-transfer-refactoring.diff (29.6 KB) - added by Antoine Martin 18 months ago.
refactoring needed to add chunking code only in one place
chunked-file-transfers.patch (16.2 KB) - added by Antoine Martin 18 months ago.
mostly complete patch, just needs cleaning up and cancelling the timers
file-alert-dialog.png (13.0 KB) - added by Antoine Martin 15 months ago.
new alert dialog

Download all attachments as: .zip

Change History (15)

comment:1 Changed 2 years ago by J. Max Mena

Edit: comment about a file too big error moved to ticket:494#comment:6

We should probably also handle the file-too-big issue with an alert box?

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:2 Changed 2 years ago by Antoine Martin

Status: newassigned

Also from ticket:494#comment:8 : GtkWarning: Could not find the icon 'gtk-file'. The 'hicolor' theme was not found either, perhaps you need to install it
Looks like we may need to tweak the win32 packaging to include or point to the (more complete?) theme.

comment:3 Changed 22 months ago by Antoine Martin

#1124 is a nicer, more generic solution to this problem.

comment:4 Changed 21 months ago by Antoine Martin

Milestone: 0.170.18

Changed 18 months ago by Antoine Martin

refactoring needed to add chunking code only in one place

Changed 18 months ago by Antoine Martin

mostly complete patch, just needs cleaning up and cancelling the timers

comment:5 Changed 18 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

Done: preparatory refactoring in r12810 + r12812, actual chunking code in r12813.

The files are now sent in chunks. We wait for the file chunk ack packet before sending the next chunk, which helps to ensure that we don't use up all the bandwidth, though it does reduce the throughput (only 4MB/s on a powerfull system loopback).

New tunables which should be self-explanatory:

XPRA_FILE_CHUNKS_SIZE=65536
XPRA_MAX_CONCURRENT_FILES=10

Here's what the transfer looks like with -d file for an ~80MB file upload to the server:

  • client side:
    show_file_upload(<gtk.ImageMenuItem object at 0x7f95f25496e0 (GtkImageMenuItem at 0x56554cae59c0)>,) can open=False
    load_contents: filename=/home/antoine/Downloads/NVIDIA-Linux-x86_64-361.45.11.run, 86205078 bytes, entity=1464443710:327571, response=-5
    send_file('/home/antoine/Downloads/NVIDIA-Linux-x86_64-361.45.11.run', '', <type 'str'>, '86205078 bytes', False, False, {})
    sha1 digest(/home/antoine/Downloads/NVIDIA-Linux-x86_64-361.45.11.run)=28d1586dd300c04f2e1ce66604a939bccfa51fa6
    ack-file-chunk: ['9f4c40d1ebf74aa69f02e8ebd866786c', True, '', 0]
    ack-file-chunk: ['9f4c40d1ebf74aa69f02e8ebd866786c', True, '', 1]
    ...
    ack-file-chunk: ['9f4c40d1ebf74aa69f02e8ebd866786c', True, '', 1316]
    1316 chunks of 65536 bytes sent in 17338ms (4MB/s)
    _check_chunk_sending(9f4c40d1ebf74aa69f02e8ebd866786c, 1316) chunk_state found: False
    
  • server side:
    receiving file: ['bigfile', '', False, False, 86205078, '0 bytes', {'file-chunk-id': '9f4c40d1ebf74aa69f02e8ebd866786c', 'sha1': '28d1586dd300c04f2e1ce66604a939bccfa51fa6'}]
    using filename 'bigfile'
    _process_send_file_chunk('9f4c40d1ebf74aa69f02e8ebd866786c', 1, '65536 bytes', True)
    _process_send_file_chunk('9f4c40d1ebf74aa69f02e8ebd866786c', 2, '65536 bytes', True)
    ...
    sha1 digest matches: 28d1586dd300c04f2e1ce66604a939bccfa51fa6
    86205078 bytes received in 1316 chunks, took 16769ms
    downloaded 86205078 bytes to temporary file:
     '/home/guest/Downloads/bigfile'
    

@afarr: ready for testing, you should now be able to send or print big files without causing much of a slow down for anything else. Logging was also improved, small bugs fixed along the way. (I may backport some)

Last edited 18 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 18 months ago by J. Max Mena

Owner: changed from alas to Antoine Martin

Did some testing of trunk r12814 file uploads (Fedora 23 both ends):

  • All good with sending files - doesn't seem to hang anything anywhere.

Of note:

The select file dialogue will hang for a solid second or two if you try to upload a large file

comment:7 Changed 18 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

r12815 closes the dialog more quickly and loads the file in the background as much as possible.

comment:8 Changed 17 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:9 Changed 15 months ago by alas

Owner: changed from alas to Antoine Martin

Tested with a 1.0 r13101 win32 client against a 1.0 r13902 fedora 23 server.

Dialog closes very quickly, whether I'm uploading a 4 kb file or a 68 MB file, or even if I'm uploading a 218 MB file... though there wasn't anything to let me know that the 218 MB file was too ambitious, until I noticed the client side warning:

2016-09-28 17:31:12,180 Warning: cannot upload the file 'sound-test-weekend-log.txt'
2016-09-28 17:31:12,180  this file is too large: 218MB
2016-09-28 17:31:12,181  the local file size limit is 100MB

The session wasn't in any way impacted while uploading a 68 MB file though... so, other than considering a popup to warn a user of overzealous ambitions, I think this is ready to close.

Changed 15 months ago by Antoine Martin

Attachment: file-alert-dialog.png added

new alert dialog

comment:10 Changed 15 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

Good point. Alert dialog added in r13903, looks like this:
new alert dialog

We also check the file size sooner and avoid trying to load really big files into memory before showing the error.

comment:11 Changed 15 months ago by J. Max Mena

Resolution: fixed
Status: newclosed

r13937 Trunk Windows 8.1 client against an r13937 Trunk Fedora 23 machine:

  • Error noted in the popup (edited) when attempting to upload a 1.7GB ISO.

Not much else left here, closing.

Last edited 14 months ago by J. Max Mena (previous) (diff)

comment:12 Changed 4 weeks ago by Antoine Martin

Last edited 4 weeks ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.