xpra icon
Bug tracker and wiki

Opened 3 weeks ago

Last modified 2 weeks ago

#2055 assigned enhancement

compression dictionary

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: major Milestone: 3.1
Component: network Version: 2.4.x
Keywords: Cc:

Description (last modified by Antoine Martin)

Some packet types will be sent thousands of times, it is quite likely that lz4 (or zstandard) would be able to perform a lot better if we trained it first. We could bencode / rencode a bunch of common strings and train it with that.
We could send the dictionary to the other end as part of the handshake.

Another good training data set is pixel data: we use lz4 for very small areas (ie: cursors in terminals) and this doesn't compress well at the moment. Maybe this could be helped with training. (ie: repeated 0xffffff00 for white pixels)

Change History (1)

comment:1 Changed 2 weeks ago by Antoine Martin

Description: modified (diff)
Status: newassigned
Summary: lz4 dictionarycompression dictionary

There are python bindings for zstandard, including dictionary access.
See pypi: zstandard: Note: When using dictionary data and compress() is called multiple times, the ZstdCompressionParameters? derived from an integer compression level and the first compressed data’s size will be reused for all subsequent operations. This may not be desirable if source data size varies significantly.

So maybe use two different contexts? One for packet metadata and one for pixel data?

Note: See TracTickets for help on using tickets.