xpra icon
Bug tracker and wiki

Opened 8 months ago

Closed 7 months ago

Last modified 7 months ago

#2579 closed defect (fixed)

Xpra Windows client crashes when x264 is used

Reported by: Konstantin Pelepelin Owned by: Konstantin Pelepelin
Priority: blocker Milestone: 4.0
Component: client Version: 3.0.x
Keywords: Cc:

Description (last modified by Antoine Martin)

Xpra Windows client crashes when showing intellisense in WebStorm.

A crash happens after several seconds of typing the code.

Full client log:

2020-02-06 19:25:25,681 Xpra GTK3 client version 3.0.5-r24942 64-bit
2020-02-06 19:25:25,683  running on Microsoft Windows 10
2020-02-06 19:25:25,727 Warning: failed to import opencv:
2020-02-06 19:25:25,728  No module named 'cv2'
2020-02-06 19:25:25,728  webcam forwarding is disabled
2020-02-06 19:25:26,187 GStreamer version 1.16.2 for Python 3.8.0 64-bit
2020-02-06 19:25:26,425 OpenGL_accelerate module loaded
2020-02-06 19:25:26,451 Using accelerated ArrayDatatype
2020-02-06 19:25:28,340 OpenGL enabled with GeForce GTX 660/PCIe/SSE2
2020-02-06 19:25:29,479 Connected (version 2.0, client OpenSSH_7.6p1)
2020-02-06 19:25:29,520 loaded RSA private key from 'C:\Users\konstantin.pelepelin/.ssh\id_rsa'
2020-02-06 19:25:29,524 Authentication (publickey) successful!
2020-02-06 19:25:29,879  keyboard settings: layout=us
2020-02-06 19:25:29,881  desktop size is 3840x1080 with 1 screen:
2020-02-06 19:25:29,881   Default (1016x285 mm - DPI: 96x96)
2020-02-06 19:25:29,882     BenQ BenQ G2420HD 1920x1080 (531x299 mm - DPI: 91x91) workarea: 1920x1050
2020-02-06 19:25:29,882     BenQ BenQ G2420HDB 1920x1080 at 1920x0 (477x268 mm - DPI: 102x102) workarea: 1920x1080
Traceback (most recent call last):
  File "E:\Xpra\tags\v3.0.x\src/xpra/client/mixins/tray.py", line 49, in setup_xpra_tray
  File "E:\Xpra\tags\v3.0.x\src/xpra/client/mixins/tray.py", line 111, in create_xpra_tray
  File "E:\Xpra\tags\v3.0.x\src/xpra/client/mixins/tray.py", line 138, in get_tray_title
TypeError: sequence item 0: expected str instance, tuple found
2020-02-06 19:25:30,986 enabled remote logging
2020-02-06 19:25:30,988 Xpra GTK3 X11 server version 3.0.5-r24939 64-bit
2020-02-06 19:25:30,988  running on Linux Ubuntu 18.04 bionic
2020-02-06 19:25:30,993 Attached to ssh://user@localhost:2222/100
2020-02-06 19:25:30,994  (press Control-C to detach)

2020-02-06 19:25:32,222 sound output using 'opus' audio codec
Exception code=0xc0000005 flags=0x0 at 0x0000000059005690. Access violation - attempting to read data at address 0x000000002ED65080
2020-02-06 19:26:05,946 Warning: long timer waiting time,
2020-02-06 19:26:05,948  UI thread polling waited 5.7 seconds longer than intended (6.2 vs 0.5)
2020-02-06 19:26:06,066 sound output stopping

Appropriate server log

2020-02-06 19:25:30,958 New unix-domain connection received
2020-02-06 19:25:30,958  on '/run/user/1000/xpra/ubuntu-100'
2020-02-06 19:25:30,964 Handshake complete; enabling connection
2020-02-06 19:25:30,970  automatic picture encoding enabled, also available:
2020-02-06 19:25:30,970   h264, vp9, vp8, png, png/P, png/L, webp, rgb24, rgb32, jpeg
2020-02-06 19:25:30,971 Python/GTK3 Microsoft Windows 10 aero client version 3.0.5-r24942 64-bit
2020-02-06 19:25:30,971  connected from 'spb-ws-s0003079' as 'user' - 'konstantin.pelepelin'
2020-02-06 19:25:30,974 setting key repeat rate from client: 500ms delay / 33ms interval
2020-02-06 19:25:30,980 keyboard mapping already configured (skipped)
2020-02-06 19:25:30,980  client root window size is 3840x1080 with 1 display:
2020-02-06 19:25:30,980   Default (1016x285 mm - DPI: 96x96)
2020-02-06 19:25:30,980     BenQ BenQ G2420HD 1920x1080 (531x299 mm - DPI: 91x91) workarea: 1920x1050
2020-02-06 19:25:30,981     BenQ BenQ G2420HDB 1920x1080 at 1920x0 (477x268 mm - DPI: 102x102) workarea: 1920x1080
2020-02-06 19:25:30,982 best resolution matching 3840x1080 is unchanged: 3840x1080
2020-02-06 19:25:31,038 Error parsing xdg menu data:
2020-02-06 19:25:31,038  ParsingError in file '/etc/xdg/menus/kde-debian-menu.menu', File not found
2020-02-06 19:25:31,038  this is either a bug in python-xdg,
2020-02-06 19:25:31,038  or an invalid system menu configuration
2020-02-06 19:25:31,079 client   9 @05.305 Xpra GTK3 X11 server version 3.0.5-r24939 64-bit
2020-02-06 19:25:31,080 client   9 @05.305  running on Linux Ubuntu 18.04 bionic
2020-02-06 19:25:31,162 client   9 @05.310 Attached to ssh://user@localhost:2222/100
2020-02-06 19:25:31,162 client   9 @05.311  (press Control-C to detach)
2020-02-06 19:25:31,606 New unix-domain connection received
2020-02-06 19:25:31,607  on '/run/user/1000/xpra/ubuntu-100'
2020-02-06 19:25:32,500 Warning: not adding duplicate printer 'OneNote'
2020-02-06 19:25:32,502 Warning: not adding duplicate printer 'Send To OneNote 2013'
2020-02-06 19:25:32,582 the remote printer 'Microsoft Print to PDF' has been configured
2020-02-06 19:25:32,596 Warning: failed to identify the pulseaudio default device to use
2020-02-06 19:25:32,596  No module named 'distutils.spawn'
2020-02-06 19:25:32,621 the remote printer 'SNIP' has been configured
2020-02-06 19:25:32,643 the remote printer 'SNIP' has been configured
2020-02-06 19:25:34,065 sound source using 'opus' audio codec
2020-02-06 19:26:06,039 client   9 @40.263 Warning: long timer waiting time,
2020-02-06 19:26:06,041 client   9 @40.264  UI thread polling waited 5.7 seconds longer than intended (6.2 vs 0.5)
2020-02-06 19:26:06,175 sound source stopping
2020-02-06 19:26:06,190 removed remote printer 'SNIP'
2020-02-06 19:26:06,209 removed remote printer 'SNIP'
2020-02-06 19:26:06,227 removed remote printer 'Microsoft Print to PDF'
2020-02-06 19:26:06,229 xpra client 9 disconnected.

Attachments (3)

report1.zip (11.9 KB) - added by Konstantin Pelepelin 8 months ago.
from bug report tool
xpra-save-x264-86.h264 (148.9 KB) - added by Konstantin Pelepelin 7 months ago.
xpra-02-25-18-06-cut.log (53.0 KB) - added by Konstantin Pelepelin 7 months ago.

Download all attachments as: .zip

Change History (28)

Changed 8 months ago by Konstantin Pelepelin

Attachment: report1.zip added

from bug report tool

comment:1 Changed 8 months ago by Konstantin Pelepelin

Windows client is installed from zip. Was started with xpra_cmd attach ssh:user@ubuntu:100

comment:2 Changed 8 months ago by Konstantin Pelepelin

Apparently does not happen when a local linux client is started on the same server instance.

comment:3 Changed 8 months ago by Antoine Martin

Description: modified (diff)
Milestone: 4.0
Owner: changed from Antoine Martin to Konstantin Pelepelin
Priority: majorcritical

Xpra Windows client crashes when showing intellisense in WebStorm?.

How do I reproduce? What do I need to install and how?

Last edited 8 months ago by Antoine Martin (previous) (diff)

comment:4 Changed 8 months ago by Konstantin Pelepelin

The application is WebStorm 2019.3.2 Build #WS-193.6015.40 running on a bundled Java. It should be https://download.jetbrains.com/webstorm/WebStorm-2019.3.2-jbr8.tar.gz, but my installation was auto-upgraded from previous version.

I'll check if I can reproduce it on a fresh installation and empty project.

I use it for TypeScript development.

Last edited 8 months ago by Antoine Martin (previous) (diff)

comment:5 Changed 8 months ago by Konstantin Pelepelin

Well, it seems it's not simple, as usual.
When I opened a new empty node.js project in the same WebStorm installation and started typing simple program from the top of my mind

const readdir = require('fs').readdir;
const promisify = require('util').promisify;
const readdirP = promisify(readdir);
readdirP('.').then(entries => entries.map(entry => console.log(entry)));

Moving cursor back and forward, using intellisense auto-completion, it crashed only once for this fragment, and while I typed the same code in another new file, it didn't crashed another time. But then when I opened my big work typescript project, it crashed immediately.

Unfortunately, I don't have crash debugging experience, so I don't know where to start.

Last edited 8 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 8 months ago by Konstantin Pelepelin

Probably I should have noted that linux box is Ubuntu in VirtualBox on Windows host. Also, vcxsrv X forwarding works pretty stable in this setup, I've seen probably only one crash in a work week.

Last edited 8 months ago by Antoine Martin (previous) (diff)

comment:7 Changed 8 months ago by Antoine Martin

Konstantin.Pelepelin

I'm unable to reproduce the crash. I installed WebStorm on Ubuntu Bionic, connected from my mswindows VM, typed in the code you posted in comment:5 and giving it time to show up the tooltip window - which I assume is the cause of the problem.
Can you try with opengl disabled on the client?

Maybe also run the client with -d all and post the last few lines that show up before it crashes.

comment:8 Changed 8 months ago by Konstantin Pelepelin

It seems with --opengl=no on the client I don't get a crash.
Will try with debug log later.

comment:9 Changed 7 months ago by Antoine Martin

Relevant samples from the log samples I received OOB using Xpra GTK3 client version 3.0.6-r25205 64-bit:

  • UI thread is locked:
    2020-02-11 17:42:55,637 wait(0.5000) actually waited 5.8922
    2020-02-11 17:42:55,638 Warning: long timer waiting time,
    2020-02-11 17:42:55,638  UI thread polling waited 5.4 seconds longer than intended (5.9 vs 0.5)
    

In some cases, there are h264 frames being decoded around that time:

gravity_adjust(21, 77, {b'frame': 0, b'pts': 0, b'type': b'IDR', b'csc': b'YUV444P', b'window-size': (1920, 1027), 'encoding': 'h264'}) window_size=[1920, 1027], size=(1932, 1045), gravity=NorthWest

For some reason, those are always on the first frame. (frame=0)
Can you try running the client with --video-decoders=none? Does the crash go away then?

comment:10 Changed 7 months ago by Konstantin Pelepelin

I'll try. Previously I've tried forcing RGB in menu but got crash anyway and there were still h264 mentions in the log.

More about the symptoms above:

  • Crashes happen when a new context window is open or when a tab is switched, but it's Swing Java UI toolkit AFAIK, so I don't know if it creates new native window or just repaints a region.
  • Yes, crashes look like UI is hanging for several seconds before xpra is killed, it corresponds to the exception happening several seconds before the end of log.

comment:11 Changed 7 months ago by Konstantin Pelepelin

Preliminary, it does not crash with --video-decoders=none for couple hours, will continue on Monday.

comment:12 Changed 7 months ago by Antoine Martin

I'll try. Previously I've tried forcing RGB in menu but got crash anyway and there were still h264 mentions in the log.

The menu sets the preference, but the server will still use a video codec when it detects video.

$ grep "violation" xpra*.log 
xpra1.log:Exception code=0xc0000005 flags=0x0 at 0x0000000059005690. Access violation - attempting to read data at address 0x000000003009F080
xpra2.log:Exception code=0xc0000005 flags=0x0 at 0x000000005900561F. Access violation - attempting to read data at address 0x000000003DA51000
xpra3.log:Exception code=0xc0000005 flags=0x0 at 0x0000000059005690. Access violation - attempting to read data at address 0x0000000033579080
xpra4.log:Exception code=0xc0000005 flags=0x0 at 0x0000000059005690. Access violation - attempting to read data at address 0x0000000000BA4080

In every log, always surrounded by h264 decoding...
So my guess is that this is a bug in ffmpeg's h264 decoder and running the client with --video-decoders=vpx will workaround this problem.

If that's the case, please run your server with:

XPRA_SAVE_TO_FILE=save xpra start ..

The x264 encoder will then save all the streams it generates to file. The last one should be the problematic one, we can then reproduce the problem reliably.

comment:13 Changed 7 months ago by Konstantin Pelepelin

With this Xpra_cmd.exe attach ssh:user@ubuntu:100 -d all --microphone=off --speaker=off --video-decoders=vpx I worked for several hours without a crash.

2020-02-17 23:17:41,191 process_draw:  362584    bytes for window 499, sequence    48822, 1835x337  at   50,648  using    vp9 encoding with options={b'frame': 1, b'csc': b'YUV444P', b'window-size': (1920, 1027)}

Will try to reproduce a crash with full dump on a non-sensitive project.

the server will still use a video codec when it detects video

Not related to this issue, but I wonder if there is a proper heuristic for using a lossy or video codec on an IDE window which is low color, low update rate, adding that in my case network is virtual and seems not saturated. Also unrelated but I see some artifacts which probably caused by video encoder skipping unchanged background pixels but client repainting them with black.

comment:14 Changed 7 months ago by Antoine Martin

... I worked for several hours without a crash

OK, so all builds starting with r25279 will no longer include the h264 decoder on ms windows.
We can re-enable this once the bug is fixed.

Not related to this issue, but I wonder if there is a proper heuristic for using a lossy or video codec on an IDE window which is low color

xpra should be using text mode for your IDE windows. If you can give us the xprop output for the IDE windows, we can add the definition here: browser/xpra/trunk/src/content-type/50_class.conf.
Until then, you should be able to get crisp text by raising the min-quality.

low update rate

Many text editors are known to do crazy repaints that trigger, ie: emacs, see #2545
This can be seen using:

set XPRA_OPENGL_PAINT_BOX
Xpra_cmd attach ..

See #760 for details.

comment:15 Changed 7 months ago by Antoine Martin

If the bug is not found, 3.0.7 will also have h264 decoding disabled: r25310.

In most cases, vp8 and vp9 are good enough.
The main downside is for those using nvenc server-side.

Changed 7 months ago by Konstantin Pelepelin

Attachment: xpra-save-x264-86.h264 added

Changed 7 months ago by Konstantin Pelepelin

Attachment: xpra-02-25-18-06-cut.log added

comment:16 Changed 7 months ago by Konstantin Pelepelin

Server log snip (no debugging).

2020-02-25 18:06:22,677 saving h264 stream to /tmp/1/xpra-save-x264-85.h264ESC[0m
2020-02-25 18:06:35,616 saving h264 stream to /tmp/1/xpra-save-x264-86.h264ESC[0m
2020-02-25 18:06:49,522 xpra client 1 disconnected.ESC[0m

xpra-save-x264-86.h264 last stream; file mod time is

2020-02-25 18:06:38.142317119 +0300 xpra-save-x264-86.h264

xpra-02-25-18-06-cut.log client debug log snip

Exception code=0xc0000005 flags=0x0 at 0x0000000059005690. Access violation - attempting to read data at address 0x0000000000000000
Last edited 7 months ago by Konstantin Pelepelin (previous) (diff)

comment:17 Changed 7 months ago by Konstantin Pelepelin

I've noticed this time normal frames log

2020-02-25 18:06:39,271 decompress_image() read back yuv plane 0: 2027520 bytes
2020-02-25 18:06:39,271 decompress_image() read back yuv plane 1: 2027520 bytes
2020-02-25 18:06:39,271 decompress_image() read back yuv plane 2: 2027520 bytes

And last frame logs

2020-02-25 18:06:39,275 decompress_image() read back yuv plane 2: 0 bytes

comment:18 Changed 7 months ago by Antoine Martin

Owner: changed from Konstantin Pelepelin to Antoine Martin
Priority: criticalblocker
Status: newassigned

decompress_image() read back yuv plane 2: 0 bytes

You're absolutely right, this looks like a race condition.

comment:19 Changed 7 months ago by Antoine Martin

Owner: changed from Antoine Martin to Konstantin Pelepelin
Status: assignednew

Thank you very much for the logs and sample.

I believe that the key difference is here:

EndPaint
AVImageWrapper.free()
AVImageWrapper.xpra_free_frame() av_frame=AVFrameWrapper(0x1d4eb7c0)
avcodec_decode_video2 returned 0
AVFrameWrapper(0x1d4eb7c0).xpra_free()
decompress_image() read back yuv plane 0: 2027520 bytes
AVFrameWrapper(0x1d4eb7c0).free() context=0x738ac00, frame=0x1d4eb7c0
decompress_image() read back yuv plane 1: 2027520 bytes
AVImageWrapper.free()
decompress_image() read back yuv plane 2: 0 bytes
AVImageWrapper.xpra_free_frame() av_frame=None
AVFrameWrapper(0x1d4eb7c0).set_context(0x738ac00, 0x1d4eb7c0)
wglMakeCurrent(0x19011450, 0x30000)

We're freeing the av_frame whilst we're reading back the YUV data of a new image.

We're calling av_frame_unref too early.
The idea here is that because we want to use zero-copy pixel upload and threading (decode thread vs paint thread), the image wrapper keeps track of the frame it's using so we can be sure that we're finished with it.

I think that r25326 fixes this and r25327 re-enables h264 decoding in the win32 client.
Does the latest beta: https://xpra.org/beta/windows work without crashing now?

comment:20 Changed 7 months ago by Konstantin Pelepelin

h264 is not listed in supported picture encodings (in session info window) in Xpra-Python3-x86_64_Setup_4.0-r25327.exe

"c:\Program Files\Xpra\Xpra_cmd.exe" attach ssh:user@ubuntu:100 --microphone=off --speaker=off --opengl=yes -d all

2020-02-26 15:36:13,250 init_video_decoders_options()
2020-02-26 15:36:13,250  will try video decoders: avcodec2, vpx
2020-02-26 15:36:13,250 dec_avcodec2:
2020-02-26 15:36:13,250  codec_import_check('dec_avcodec2', 'avcodec2 decoder', 'xpra.codecs.dec_avcodec2', 'xpra.codecs.dec_avcodec2.decoder', ('Decoder',))
2020-02-26 15:36:13,250  xpra.codecs.dec_avcodec2 found, will check for ('Decoder',) in xpra.codecs.dec_avcodec2.decoder
2020-02-26 15:36:13,286 Error importing avcodec2 decoder (dec_avcodec2)
2020-02-26 15:36:13,286  DLL load failed while importing decoder: The specified module could not be found.
2020-02-26 15:36:13,286 
Traceback (most recent call last):
  File "E:\Xpra\trunk\src/xpra/codecs/loader.py", line 52, in codec_import_check
ImportError: DLL load failed while importing decoder: The specified module could not be found.
2020-02-26 15:36:13,286 init_video_decoder_option(dec_avcodec2)
2020-02-26 15:36:13,286  module=None
2020-02-26 15:36:13,286  video decoder dec_avcodec2 could not be loaded:
2020-02-26 15:36:13,286  DLL load failed while importing decoder: The specified module could not be found.

Last edited 7 months ago by Konstantin Pelepelin (previous) (diff)

comment:21 Changed 7 months ago by Antoine Martin

h264 is not listed in supported picture encodings (in session info window) in Xpra-Python3-x86_64_Setup_4.0-r25327.exe

Oops, sorry.
That's #2588 which is now fixed. Updated builds are available.

comment:22 Changed 7 months ago by Konstantin Pelepelin

Resolution: fixed
Status: newclosed

I worked in Xpra-Python3-x86_64_4.0-r25345 for several hours with Xpra_cmd.exe attach ssh:user@ubuntu:100 -d all --microphone=off --speaker=off --encodings=h264 --opengl=yes

I think the issue is fixed.

comment:23 Changed 7 months ago by Konstantin Pelepelin

Summary: Xpra Windows client crashes when showing intellisense in WebStormXpra Windows client crashes when x264 is used

comment:24 Changed 7 months ago by Antoine Martin

Great!
Thank you so much for reporting this bug and taking the time to test the fix.

The backport to the v3.0.x branch is in r25433, this will be included in the 3.0.7 release.

comment:25 Changed 7 months ago by Antoine Martin

Description: modified (diff)
Note: See TracTickets for help on using tickets.