Xpra: Ticket #1187: h264 breaks Windows shadow servers when display is 4k

When the server has a display that is a 4k display (and assuming larger, but we don't really have a larger display...I think...), and a user attempts to connect with h264 enabled, the server crashes after a second or so when it attempts to paint with h264 with the following server side error(included normal server prints as well):

C:\Program Files (x86)\Xpra>Xpra_cmd.exe shadow :0 --bind-tcp=0.0.0.0:2200 --mdns=no
2016-04-22 10:43:36,642 created named pipe: \\.\pipe\Xpra\Main
2016-04-22 10:43:37,259 GStreamer version 1.6 for Python 3.4
2016-04-22 10:43:37,259 xpra shadow version 0.17.0-r12401
2016-04-22 10:43:37,260  running with pid 5256 on Microsoft Windows 7
2016-04-22 10:43:37,260  on display :0 of size 3840x3060
2016-04-22 10:43:37,262 xpra is ready.
2016-04-22 10:44:09,671 New tcp connection received from 10.0.32.209:54692
2016-04-22 10:44:09,789 Handshake complete; enabling connection
2016-04-22 10:44:09,812 Python/Gtk2 Linux Fedora 23 Twenty Three client version 0.18.0-r11814
2016-04-22 10:44:09,813  connected from 'verschlimmbessern-spikes-eng' as 'max' - 'J Max Mena'
2016-04-22 10:44:09,816  using h264 as primary encoding also available:
2016-04-22 10:44:09,818   vp8, png, png/P, png/L, webp, rgb24, jpeg, rgb32
2016-04-22 10:44:09,819  client root window size is 3840x2160
2016-04-22 10:44:09,822 shadow server: setting default keymap translation
2016-04-22 10:44:09,835 Attached to tcp:10.0.70.59:2200 (press Control-C to detach)
2016-04-22 10:44:10,427 X264: frame MB size (240x192) > level limit (36864)
2016-04-22 10:44:10,569 stopping speaker because of error: push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn>
2016-04-22 10:44:10,571 sound source stopping
2016-04-22 10:44:10,777 sound source using audio codec opus
x264 [error]: malloc of size 150332192 failed
2016-04-22 10:44:10,930 x264 encoding error: frame_size is invalid!
2016-04-22 10:44:10,930 video_encode: ouch, h264 compression failed

After this, an "Xpra has stopped working" error pops up, and the client immediately gets spinners, and times out.


Connecting without h264 enabled, such as xpra attach tcp:ip --encodings=webp,jpeg,png,rgb, the user can continue. It appears that h264 is the culprit.



Sat, 23 Apr 2016 03:15:51 GMT - Antoine Martin: status changed

Very interesting, could help in figuring out other crashes. The compression error should be problematic for sure, but not be fatal, on any platform.


Sat, 23 Apr 2016 04:06:16 GMT - Antoine Martin: owner, status changed

That said, the error x264 [error]: malloc of size 150332192 failed is ominous. If the system can't allocate ~143MB of RAM, things are likely to fail pretty quickly. How much RAM does this system have? How much was xpra using when you ran it? I guess we could add a big warning during startup if we find that there isn't enough RAM.


Tue, 26 Apr 2016 19:41:38 GMT - J. Max Mena: owner changed

How much RAM does this system have?


This system has 16GB of dedicated RAM.


How much was xpra using when you ran it?


Between 120MB and 170MB when the connection could be established (by disabling h264). With h264 enabled, the system RAM usage jumps by about 600-700MB before Xpra crashes.


Tue, 26 Apr 2016 19:42:50 GMT - J. Max Mena:

To me, it looks like it's trying to allocate a huge amount of memory, then crashes when it's unable to.


Sat, 30 Apr 2016 06:15:15 GMT - Antoine Martin: status changed

~143MB is not huge, even 700MB for a video service isn't very big: your video card alone probably has 3 times that amount.

So I am tempted to close this as wontfix, but it would be better to fail more gracefully at least.


Tue, 12 Jul 2016 16:52:22 GMT - Antoine Martin: milestone changed

Milestone renamed


Thu, 04 Aug 2016 04:57:34 GMT - Antoine Martin: owner, status changed

Testing with a screen slightly smaller than 4k, I see memory usage under 400MB with h264 and no crashes. Please post the "-d all" crash output.


Thu, 04 Aug 2016 18:29:51 GMT - J. Max Mena: attachment set


Thu, 04 Aug 2016 18:31:40 GMT - J. Max Mena:

r13173 Shadow Servers are failing to accept connections (attached log, will file new ticket) - as such this ticket is on hold until that issue is resolved.


Fri, 05 Aug 2016 01:32:35 GMT - Antoine Martin:

r13197 fixed that connection bug.


Mon, 08 Aug 2016 16:40:06 GMT - J. Max Mena: status changed; resolution set

Upped Windows server to r13235:

<Unicode console stderr>.write: UnicodeDecodeError('utf8', "2016-08-08 09:33:30,743 client 1: invalid img data length: expected 15356 but got 33177600 (<type 'str'>: \xab\x85g\xff\xbe\x98z\xff\xb0\x8bo\xff\xb0\x8eq\xff\xae\x8ar\xff\xb1\x8fw\xff\xb3\x93|\xff\xbc\x9d\x86\xff\xb8\x99\x84\xff\xb4\x97\x82\xff\xb0\x92\x7f\xff\xad\x8f|\xff\xb0\x93~\xff\xb7\x9a\x85\xff\xbb\x9e\x89\xff\xbb\x9e\x89\xff\xba\x9a\x83\xff\xb5\x95~\xff\xb5\x95~\xff\xb9\x99\x82\xff\xb9\x99\x82\xff\xb4\x94}\xff\xb3\x91y\xff\xb6\x94|\xff\xb8\x96~\xff\xb8\x96~\xff\xb5\x93{\xff\xb1\x8fw\xff\xb3\x90v\xff\xb6\x93y\xff\xb8\x95{\xff\xb6\x95{\xff\xb3\x95z\xff\xb3\x96{\xff\xb5\x97|\xff\xb3\x95z\xff\xb2\x94y\xff\xb2\x94y\xff\xb8\x97}\xff\xbb\x9a\x80\xff\xb3\x92x\xff\xb3\x92x\xff\xb6\x94w\xff\xb4\x92u\xff\xb1\x8fr\xff\xb0\x8eq\xff\xb3\x91t\xff\xb8\x96y\xff\xbc\x95y\xff\xb9\x92v\xff\xb2\x8ds\xff\xb1\x8cr\xff\xb5\x91y\xff\xba\x98\x80\xff\xbd\x9b\x84\xff\xb9\x99\x82\xff\xba\x9a\x87\xff\xbc\x9e\x8b\xff\xc0\xa1\x92\xff\xc0\xa3\x94\xff\xbf\xa4\x96\xff\xc1\xa6\x98\xff\xc2\xa8\x9a\xff\xc4\xab\x9b\xff)\n", 106, 107, 'invalid start byte')                                                                                                   Traceback (most recent call last):                                                                                        File "logging\__init__.pyc", line 884, in emit                                                                        UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position 106: invalid start byte                             Logged from file log.pyc, line 107                                                                                      2016-08-08 09:33:30,750 client 1: draw error                                                                            2016-08-08 09:33:30,752 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/ui_client_base.py", line 2971, in _do_draw                                                                                                             2016-08-08 09:33:30,753 client 1:     window.draw_region(x, y, width, height, coding, data, rowstride, packet_sequence, options, [record_decode_time])                                                                                          2016-08-08 09:33:30,755 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/client_window_base.py", line 553, in draw_region                                                                                                       2016-08-08 09:33:30,756 client 1:     backing.draw_region(x, y, width, height, coding, img_data, rowstride, options, callbacks)                                                                                                                 2016-08-08 09:33:30,757 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 490, in draw_region                                                                                                      2016-08-08 09:33:30,759 client 1:     self.paint_rgb(rgb_format, img_data, x, y, width, height, rowstride, options, callbacks)                                                                                                                  2016-08-08 09:33:30,759 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 267, in paint_rgb                                                                                                        2016-08-08 09:33:30,760 client 1:     rgb_data = self.process_delta(raw_data, width, height, rowstride, options)        2016-08-08 09:33:30,760 Warning: client decoding error: expected 15356 bytes for 3839x1 with rowstride=15356 but received 33177600 (15046933 compressed)                                                                                        2016-08-08 09:33:30,762 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 172, in process_delta                                                                                                    2016-08-08 09:33:30,762 client 1:     (rowstride * height, width, height, rowstride, len(img_data), len(raw_data)))     2016-08-08 09:33:30,763 client 1: error processing draw packet                                                          2016-08-08 09:33:30,763 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/ui_client_base.py", line 2911, in _draw_thread_loop                                                                                                    2016-08-08 09:33:30,763 client 1:     self._do_draw(packet)                                                             2016-08-08 09:33:30,763 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/ui_client_base.py", line 2971, in _do_draw                                                                                                             2016-08-08 09:33:30,763 client 1:     window.draw_region(x, y, width, height, coding, data, rowstride, packet_sequence, options, [record_decode_time])                                                                                          2016-08-08 09:33:30,763 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/client_window_base.py", line 553, in draw_region                                                                                                       2016-08-08 09:33:30,763 client 1:     backing.draw_region(x, y, width, height, coding, img_data, rowstride, options, callbacks)                                                                                                                 2016-08-08 09:33:30,763 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 490, in draw_region                                                                                                      2016-08-08 09:33:30,763 client 1:     self.paint_rgb(rgb_format, img_data, x, y, width, height, rowstride, options, callbacks)                                                                                                                  2016-08-08 09:33:30,763 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 267, in paint_rgb                                                                                                        2016-08-08 09:33:30,763 client 1:     rgb_data = self.process_delta(raw_data, width, height, rowstride, options)        2016-08-08 09:33:30,763 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 172, in process_delta                                                                                                    2016-08-08 09:33:30,763 client 1:     (rowstride * height, width, height, rowstride, len(img_data), len(raw_data)))     <Unicode console stderr>.write: UnicodeDecodeError('utf8', "2016-08-08 09:33:59,332 client 1: invalid img data length: expected 15356 but got 33177600 (<type 'str'>: \xab\x85g\xff\xbe\x98z\xff\xb0\x8bo\xff\xb0\x8eq\xff\xae\x8ar\xff\xb1\x8fw\xff\xb3\x93|\xff\xbc\x9d\x86\xff\xb8\x99\x84\xff\xb4\x97\x82\xff\xb0\x92\x7f\xff\xad\x8f|\xff\xb0\x93~\xff\xb7\x9a\x85\xff\xbb\x9e\x89\xff\xbb\x9e\x89\xff\xba\x9a\x83\xff\xb5\x95~\xff\xb5\x95~\xff\xb9\x99\x82\xff\xb9\x99\x82\xff\xb4\x94}\xff\xb3\x91y\xff\xb6\x94|\xff\xb8\x96~\xff\xb8\x96~\xff\xb5\x93{\xff\xb1\x8fw\xff\xb3\x90v\xff\xb6\x93y\xff\xb8\x95{\xff\xb6\x95{\xff\xb3\x95z\xff\xb3\x96{\xff\xb5\x97|\xff\xb3\x95z\xff\xb2\x94y\xff\xb2\x94y\xff\xb8\x97}\xff\xbb\x9a\x80\xff\xb3\x92x\xff\xb3\x92x\xff\xb6\x94w\xff\xb4\x92u\xff\xb1\x8fr\xff\xb0\x8eq\xff\xb3\x91t\xff\xb8\x96y\xff\xbc\x95y\xff\xb9\x92v\xff\xb2\x8ds\xff\xb1\x8cr\xff\xb5\x91y\xff\xba\x98\x80\xff\xbd\x9b\x84\xff\xb9\x99\x82\xff\xba\x9a\x87\xff\xbc\x9e\x8b\xff\xc0\xa1\x92\xff\xc0\xa3\x94\xff\xbf\xa4\x96\xff\xc1\xa6\x98\xff\xc2\xa8\x9a\xff\xc4\xab\x9b\xff)\n", 106, 107, 'invalid start byte')                                                                                                   Traceback (most recent call last):                                                                                        File "logging\__init__.pyc", line 884, in emit                                                                        UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position 106: invalid start byte                             Logged from file log.pyc, line 107                                                                                      2016-08-08 09:33:59,338 client 1: draw error                                                                            2016-08-08 09:33:59,339 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/ui_client_base.py", line 2971, in _do_draw                                                                                                             2016-08-08 09:33:59,339 client 1:     window.draw_region(x, y, width, height, coding, data, rowstride, packet_sequence, options, [record_decode_time])                                                                                          2016-08-08 09:33:59,341 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/client_window_base.py", line 553, in draw_region                                                                                                       2016-08-08 09:33:59,342 client 1:     backing.draw_region(x, y, width, height, coding, img_data, rowstride, options, callbacks)                                                                                                                 2016-08-08 09:33:59,344 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 490, in draw_region                                                                                                      2016-08-08 09:33:59,344 client 1:     self.paint_rgb(rgb_format, img_data, x, y, width, height, rowstride, options, callbacks)                                                                                                                  2016-08-08 09:33:59,345 Warning: client decoding error: expected 15356 bytes for 3839x1 with rowstride=15356 but received 33177600 (15041319 compressed)                                                                                        2016-08-08 09:33:59,345 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 267, in paint_rgb                                                                                                        2016-08-08 09:33:59,346 client 1:     rgb_data = self.process_delta(raw_data, width, height, rowstride, options)        2016-08-08 09:33:59,346 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 172, in process_delta                                                                                                    2016-08-08 09:33:59,348 client 1:     (rowstride * height, width, height, rowstride, len(img_data), len(raw_data)))     2016-08-08 09:33:59,348 client 1: error processing draw packet                                                          2016-08-08 09:33:59,349 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/ui_client_base.py", line 2911, in _draw_thread_loop                                                                                                    2016-08-08 09:33:59,351 client 1:     self._do_draw(packet)                                                             2016-08-08 09:33:59,351 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/ui_client_base.py", line 2971, in _do_draw                                                                                                             2016-08-08 09:33:59,351 client 1:     window.draw_region(x, y, width, height, coding, data, rowstride, packet_sequence, options, [record_decode_time])                                                                                          2016-08-08 09:33:59,351 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/client_window_base.py", line 553, in draw_region                                                                                                       2016-08-08 09:33:59,352 client 1:     backing.draw_region(x, y, width, height, coding, img_data, rowstride, options, callbacks)                                                                                                                 2016-08-08 09:33:59,354 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 490, in draw_region                                                                                                      2016-08-08 09:33:59,355 client 1:     self.paint_rgb(rgb_format, img_data, x, y, width, height, rowstride, options, callbacks)                                                                                                                  2016-08-08 09:33:59,355 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 267, in paint_rgb                                                                                                        2016-08-08 09:33:59,355 client 1:     rgb_data = self.process_delta(raw_data, width, height, rowstride, options)        2016-08-08 09:33:59,357 client 1:   File "/usr/lib64/python2.7/site-packages/xpra/client/window_backing_base.py", line 172, in process_delta                                                                                                    2016-08-08 09:33:59,358 client 1:     (rowstride * height, width, height, rowstride, len(img_data), len(raw_data)))

However, no crash. As such, I'm closing this as fixed, as it handled the errors gracefully and did not crash or kick my client.


Sat, 23 Jan 2021 05:17:21 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1187