xpra icon
Bug tracker and wiki

Opened 7 years ago

Closed 7 years ago

#375 closed defect (fixed)

PIL Error with jpeg encoding: "Suspension not allowed here"

Reported by: Antoine Martin Owned by: alas
Priority: blocker Milestone: 0.10
Component: server Version: trunk
Keywords: Cc:

Description

as originally reported in #249,
we can hit this PIL/Pillow bug

What I don't understand is why we didn't hit it before..
It may be because we now "optimize" the output in some cases (when the speed is low enough and close to the min-speed) or because we pass the image info (metadata) to Image.save when encoding.


Please clarify:

  • which server os/version this was tested on, and which version of PIL or Pillow it uses
  • does setting min-speed > 50 fix things
  • does setting speed > 50 fix things
  • does this patch fix things:
    --- src/xpra/server/window_source.py	(revision 3787)
    +++ src/xpra/server/window_source.py	(working copy)
    @@ -957,9 +957,8 @@
                 if options:
                     q = options.get("quality", q)
                 q = int(min(99, max(1, q)))
    -            kwargs = im.info
    -            kwargs["quality"] = q
    -            kwargs["optimize"] = optimize
    +            kwargs = {"quality" : q,
    +                      "optimize": optimize}
                 im.save(buf, "JPEG", **kwargs)
                 client_options["quality"] = q
             else:
    
  • if not, does this one:
    --- src/xpra/server/window_source.py	(revision 3787)
    +++ src/xpra/server/window_source.py	(working copy)
    @@ -959,7 +959,6 @@
                 q = int(min(99, max(1, q)))
                 kwargs = im.info
                 kwargs["quality"] = q
    -            kwargs["optimize"] = optimize
                 im.save(buf, "JPEG", **kwargs)
                 client_options["quality"] = q
             else:
    

Attachments (1)

osxXpraTests_7-11 (8.9 KB) - added by alas 7 years ago.
list of test results with various included encodings, and error outputs

Download all attachments as: .zip

Change History (12)

comment:1 Changed 7 years ago by alas

Status: newassigned

That error came when I used the quit command from the osx tray icon (which quits the client but which I saw no indication that the server had acknowledged as a client severed connection) and then tried to restart the client and connect back to the still running server session using a different encoder option on the client side.

I was using the osx beta client you built yesterday, on an osx 10.6.8 macbook.

I'll try the rest of your suggested possibilities tomorrow, but I wanted to be sure that you realized the wackiness of what I had to do to get that particular error. In particular, I wanted to be sure that the still not-finished tray/dock icon's behavior isn't just in need of that detail being kept in mind while the work is done vs. it being something that needs to be tracked down. I will also try to use the tray/dock icon to disconnect then reconnect to the same session but maintaining the same encoding, to see if that might be what was causing the issue.

Changed 7 years ago by alas

Attachment: osxXpraTests_7-11 added

list of test results with various included encodings, and error outputs

comment:2 Changed 7 years ago by alas

I couldn't find any PIL or Pillow on my mac. It is os x 10.6.8 running python 2.6.1 ... and the internets are full of horror stories of trying to install PIL- which is all I know on the subject.

I experimented with the min-speed and speed options with the JPEG encoding, because that seemed to be where I was seeing most of the PIL and Suspension not allowed here messages were appearing (one every .023 seconds or so). I'll attach the results and some server and client logs. Will test the patches tomorrow (I hope).

comment:3 Changed 7 years ago by Antoine Martin

Status: assignednew

Wait, I am confused now.

The error in this ticket "Suspension not allowed here" is a server side issue, but you are talking about your osx client.

This issue should manifest itself *server-side* only, and with any client.

What I am interested in is in which (server-side) patche(s) fix the issue.

comment:4 Changed 7 years ago by alas

[Jim Beam@jimbeam PIL]$ uname -a
Linux jimbeam.spikes.eng 3.9.6-200.fc18.x86_64 # 1 SMP Thu Jun 13 18:56:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
PIL =
Installed Packages
Name        : python-imaging
Arch        : x86_64
Version     : 1.1.7
Release     : 7.fc18

I'll try the patches as soon as I get a chance.

Last edited 7 years ago by Antoine Martin (previous) (diff)

comment:5 Changed 7 years ago by Antoine Martin

Priority: majorblocker

Raising the priority as we cannot make the new release with this bug still in it.

Last edited 7 years ago by Antoine Martin (previous) (diff)

comment:6 Changed 7 years ago by Antoine Martin

afarr / smo: This is a release blocker, and has been for 4 weeks!

comment:7 Changed 7 years ago by Smo

Can't seem to reproduce this on
osx 10.7.5 with r4149

forced jpeg encoding with --encoding=jpeg and tried various
--speed and --min-speed settings

comment:8 Changed 7 years ago by Antoine Martin

As per comment:3, this is a server side issue I believe which only affects JPEG (and maybe PNG) encoding (not decoding AFAIK), and only at low speed when we "optimize".
We do care about those because we use them, even when x264 or vpx are set as primary encodings.
Unfortunately, it sounds like this may affect only some versions of PIL/Pillow - and maybe only some picture sizes (bigger more likely), so more testing is needed on as many versions as possible. I have tested Pillow 2.0 (Fedora 19) today and I can encode 1080p without problems.

Best way to make it fire for testing is to just force it:

--- src/xpra/server/window_source.py	(revision 4149)
+++ src/xpra/server/window_source.py	(working copy)
@@ -1008,7 +1008,7 @@
             raise e
         buf = StringIOClass()
         client_options = {}
-        optimize = options.get("optimize")
+        optimize = True
         if optimize is None and self.batch_config.delay>2*self.batch_config.START_DELAY:
             ces = self.get_current_speed()
             mes = self.get_min_speed()
Last edited 7 years ago by Antoine Martin (previous) (diff)

comment:9 Changed 7 years ago by Smo

Started xpra with

xpra attach tcp:1.2.3.4:1300 --opengl=off --encoding=jpeg --no-speaker
on osx 10.7.5

As you mentioned I made a build with the patch in comment:8

I cannot reproduce on fedora 19 which uses pillow 2.0.0

However under fedora 18 which uses python-imaging 1.1.7 I do get this error on the server still.

2013-08-12 11:59:31,974 error processing damage data: encoder error -2 when writing image file
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/xpra/server/source.py", line 1280, in data_to_packet
    encode_and_queue()
  File "/usr/lib/python2.7/site-packages/xpra/server/window_source.py", line 689, in make_data_packet_cb
    packet = self.make_data_packet(*data)
  File "/usr/lib/python2.7/site-packages/xpra/server/window_source.py", line 869, in make_data_packet
    data, client_options, outw, outh, outstride, bpp = encoder(coding, image, options)
  File "/usr/lib/python2.7/site-packages/xpra/server/window_source.py", line 1025, in PIL_encode
    im.save(buf, "JPEG", **kwargs)
  File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 1439, in save
    save_handler(self, fp, filename)
  File "/usr/lib64/python2.7/site-packages/PIL/JpegImagePlugin.py", line 471, in _save
    ImageFile._save(im, fp, [("jpeg", (0,0)+im.size, 0, rawmode)])
  File "/usr/lib64/python2.7/site-packages/PIL/ImageFile.py", line 491, in _save
    raise IOError("encoder error %d when writing image file" % s)
IOError: encoder error -2 when writing image file
Suspension not allowed here
2013-08-12 11:59:31,982 error processing damage data: encoder error -2 when writing image file
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/xpra/server/source.py", line 1280, in data_to_packet
    encode_and_queue()
  File "/usr/lib/python2.7/site-packages/xpra/server/window_source.py", line 689, in make_data_packet_cb
    packet = self.make_data_packet(*data)
  File "/usr/lib/python2.7/site-packages/xpra/server/window_source.py", line 869, in make_data_packet
    data, client_options, outw, outh, outstride, bpp = encoder(coding, image, options)
  File "/usr/lib/python2.7/site-packages/xpra/server/window_source.py", line 1025, in PIL_encode
    im.save(buf, "JPEG", **kwargs)
  File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 1439, in save
    save_handler(self, fp, filename)
  File "/usr/lib64/python2.7/site-packages/PIL/JpegImagePlugin.py", line 471, in _save
    ImageFile._save(im, fp, [("jpeg", (0,0)+im.size, 0, rawmode)])
  File "/usr/lib64/python2.7/site-packages/PIL/ImageFile.py", line 491, in _save
    raise IOError("encoder error %d when writing image file" % s)
IOError: encoder error -2 when writing image file

comment:10 Changed 7 years ago by Smo

Just to be sure I tried with with a linux and windows client not that I expected a different error but just as a test.

Same behavior

comment:11 Changed 7 years ago by Antoine Martin

Resolution: fixed
Status: newclosed

Right, that's what I thought - the bug is only in PIL, it is already fixed in Pillow.
So:

  • r4165 gives us the ability to turn the "optimize" support on or off for testing more easily (using the XPRA_PIL_OPTIMIZE env var), or as a workaround should it be needed. Note: this does not force-enable it, it only allows it to be used.
  • r4166 adds a patch we include during the build for outdated distros. From what I see, this will be needed for:
    • all Debian releases up to now, including sid
    • all Ubuntu releases prior to 'raring'
    • Fedora 18 and earlier
    • all CentOS releases up to now, including 6.4

OSX and MS Windows are not affected because we build stuff ourselves from scratch and the instructions have been updated to use Pillow a while back.

I am closing this ticket because I have verified that the packages do include the patch.

Note: See TracTickets for help on using tickets.