xpra icon
Bug tracker and wiki

This bug tracker and wiki are being discontinued
please use https://github.com/Xpra-org/xpra instead.


Opened 8 years ago

Closed 8 years ago

Last modified 16 months ago

#535 closed defect (fixed)

server-side memory leak

Reported by: onlyjob Owned by: onlyjob
Priority: major Milestone: 0.12
Component: server Version: 0.11.x
Keywords: Cc:

Description

0.11.4 leaks memory on server side. After using attached session for several days I've noticed that xpra process substantially grew in memory (RES > 2.5 GiB; VIRT > 3 GiB). I'm not sure how to troubleshoot this issue but I'm fairly certain that it is not related to audio which I had disabled. For now I have to run xpra upgrade periodically to release RAM.

Attachments (6)

xpra.conf​ (3.4 KB) - added by jamie 8 years ago.
jamie-etc-xpra-xpra.conf
:201.log​ (4.1 KB) - added by jamie 8 years ago.
jamie-server-log
xpra-info.txt​ (67.6 KB) - added by jamie 8 years ago.
Full output of xpra info
no-video-subregion.patch​ (454 bytes) - added by Antoine Martin 8 years ago.
disable subregion code
:201.2.log​ (39.4 KB) - added by jamie 8 years ago.
Server log with XPRA_DETECT_LEAKS=1
debux-xshmimage-leak.patch​ (1.3 KB) - added by Antoine Martin 8 years ago.
patch I've used to verify that we are not leaking xshm image wrappers

Download all attachments as: .zip

Change History (36)

comment:1 Changed 8 years ago by Antoine Martin

Milestone: 0.13 → 0.12
Owner: changed from Antoine Martin to onlyjob

Could be related to #457.

Does it go away if you remove the webp codec from the system? (see #491)
(remove xpra/codecs/webm)

It may also be libav related...

Last edited 8 years ago by Antoine Martin (previous) (diff)

comment:2 in reply to:  1 Changed 8 years ago by onlyjob

Thank you for quick reply.

Replying to totaam:

Could be related to #457.

Does it go away if you remove the webp codec from the system? (see #491)
(remove xpra/codecs/webm)


I'll try without webp but it will take days to see the difference. Besides I have impression that those bugs are about leaks on the client side while mine is on server side.

I may also be libav related...


Everything is possible. :)
I'm still running libav-9.11 on client and server. It is the same libav that I was using for a while but I started to notice memory leak only recently...

comment:3 Changed 8 years ago by Antoine Martin

I have impression that those bugs are about leaks on the client side while mine is on server side

True, but I suspect that the webp library may well be leaking when encoding as well as decoding.

I'm still running libav-9.11 on client and server

The reason why I suspect libav by default is twofold:

  • it has leaked before, and there are a number of memory related patches to make it build against libav instead of ffmpeg
  • the vast majority of leaks happen in areas of the code that handle large amounts of data: sound and video.

So, it would be interesting to see if the leak still occurs when using an encoding we know for sure does not leak: rgb or png have never shown signs of leaking - yet. (and to be certain video does not get used in any way, the best way is to not install the codec)

comment:4 Changed 8 years ago by onlyjob

For what's it worth this leak is with RGB encoding (set as default in config)...

comment:5 Changed 8 years ago by Antoine Martin

It's still worth deleting all the other codecs to ensure that only RGB can be used. More and more the code is using the chosen encoding only as a hint and will switch to other encodings as appropriate (though in theory, RGB is probably the only one that *should* be honoured at all times...).

Can you post the output of: xpra info | grep memory then things get bad?
Does minimizing the windows help? Re-connecting the client?

comment:6 Changed 8 years ago by jamie

I seem to be experiencing the same issue. I can easily reproduce (the memory increases with every click of the web browser that I access via xrpa).

I'm running the client and server on the same machine (so I can run my web browser as an isolated user on my machine).

Question: how do I disable the other encodings? I don't seem to have a codecs/webm folder (maybe remove codecs/vpx?). I tried moving the entire codecs folder to codecs.off but it didn't seem to have any effect.

I've attached my /etc/xpra/xrpa.conf file and the server log file.

Here's my memory after starting xpra:

0 jamie@animal:cdtemp.5reEHd$ ps -e -o rss,cmd | egrep "[X]pra" | grep iceweasel
76784 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:cdtemp.5reEHd$

After launching iceweasel (Firefox)

jamie@animal:cdtemp.5reEHd$ ps -e -o rss,cmd | egrep "[X]pra" | grep iceweasel
110616 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:cdtemp.5reEHd$

Here's my command for starting xpra (run as iceweasel user):

/usr/bin/python /usr/bin/xpra --mmap-group --socket-dir=/home/iceweasel/xpra-socket --no-mdns start :201

Here's the attach command (run as jamie user):

/usr/bin/python /usr/bin/xpra --socket-dir=/home/iceweasel/xpra-socket --mmap-group --tray-icon=/usr/share/icons/hicolor/scalable/apps/iceweasel.svg --encoding=rgb24 attach :201

Here's the output of the info command:

0 jamie@animal:bin$ sudo -u iceweasel /usr/bin/python /usr/bin/xpra --mmap-group --socket-dir=/home/iceweasel/xpra-socket --no-mdns info :201 | grep memory
memory.children.idrss=0
memory.children.inblock=0
memory.children.isrss=0
memory.children.ixrss=0
memory.children.majflt=0
memory.children.maxrss=30048
memory.children.minflt=19940
memory.children.msgrcv=0
memory.children.msgsnd=0
memory.children.nivcsw=54
memory.children.nsignals=0
memory.children.nswap=0
memory.children.nvcsw=261
memory.children.oublock=32
memory.children.stime=0
memory.children.utime=0
memory.server.idrss=0
memory.server.inblock=0
memory.server.isrss=0
memory.server.ixrss=0
memory.server.majflt=4
memory.server.maxrss=64392
memory.server.minflt=27163
memory.server.msgrcv=0
memory.server.msgsnd=0
memory.server.nivcsw=234
memory.server.nsignals=0
memory.server.nswap=0
memory.server.nvcsw=3989
memory.server.oublock=12024
memory.server.stime=0
memory.server.utime=0
0 jamie@animal:bin$
Last edited 8 years ago by jamie (previous) (diff)

Changed 8 years ago by jamie

Attachment: xpra.conf​ added

jamie-etc-xpra-xpra.conf

Changed 8 years ago by jamie

Attachment: :201.log​ added

jamie-server-log

comment:7 Changed 8 years ago by jamie

Also, minimizing the window causes the memory usage of the server to increase and maximizing the window causes the memory to increase.

Detaching has no impact on memory usage.

Attaching causes the memory usage to increase.

Last edited 8 years ago by jamie (previous) (diff)

comment:8 Changed 8 years ago by Antoine Martin

Owner: changed from onlyjob to jamie

@jamie: what is your distro? What version are you using? Do you get the same results with 0.12.x? Please include the full xpra info as an attachment, not just the memory bits.

how do I disable the other encodings?


You can disable most of them at build time.
At runtime, it is more difficult.

I don't seem to have a codecs/webm folder (maybe remove codecs/vpx?).


Strange, did you build without webp (aka webm) support?

I tried moving the entire codecs folder to codecs.off but it didn't seem to have any effect.


It will have an effect and will certainly break everything. If it didn't, maybe you moved the wrong one.

minimizing the window causes the memory usage of the server to increase and maximizing the window causes the memory to increase


Interesting, maximizing is understandable. But minimizing less so!

Attaching causes the memory usage to increase


This one is expected as we allocate resources for each client.

Last edited 8 years ago by Antoine Martin (previous) (diff)

Changed 8 years ago by jamie

Attachment: xpra-info.txt​ added

Full output of xpra info

comment:9 Changed 8 years ago by jamie

Thanks for the response. I just added the full output of xpra info.

I'm running Debian Jessie and xpra 0.11.4+dfsg-1.

I will try to compile latest xpra and see if the problem goes away.

comment:10 Changed 8 years ago by jamie

Alas, no luck.

I tried installing using the Debian packaging with a 0.12.2 tar ball and slight modifications to the Debian-shipped patches:

diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/disable_html5.patch debian/patches/disable_html5.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/disable_html5.patch  2014-02-03 11:23:26.000000000 -0500
+++ debian/patches/disable_html5.patch  2014-03-30 22:12:09.411752548 -0400
@@ -7,7 +7,7 @@
  gtk3_ENABLED = client_ENABLED
  qt4_ENABLED = client_ENABLED
  opengl_ENABLED = client_ENABLED
--html5_ENABLED = True
+-html5_ENABLED = not WIN32 and not OSX
 +html5_ENABLED = False
  
  bencode_ENABLED = True
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/enable-x264-opencl.patch debian/patches/enable-x264-opencl.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/enable-x264-opencl.patch 2014-02-05 09:54:38.000000000 -0500
+++ debian/patches/enable-x264-opencl.patch 2014-03-30 22:12:09.411752548 -0400
@@ -6,18 +6,16 @@
 
 --- a/xpra/codecs/enc_x264/encoder.pyx
 +++ b/xpra/codecs/enc_x264/encoder.pyx
-@@ -69,8 +69,11 @@
-         int i_bframe_pyramid    #Keep some B-frames as references: 0=off, 1=strict hierarchical, 2=normal
+@@ -117,6 +117,9 @@
          int b_open_gop
          int b_bluray_compat
  
 +        #not all builds support this:
 +        int b_opencl            #use OpenCL when available
 +
+         rc  rc                  #rate control
+ 
      ctypedef struct x264_t:
-         pass
-     ctypedef struct x264_nal_t:
-         int i_ref_idc
 @@ -250,8 +253,9 @@
          #we don't want IDR frames either:
          param.i_keyint_min = 999999
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav10.patch debian/patches/libav10.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav10.patch  2014-03-03 07:33:40.000000000 -0500
+++ debian/patches/libav10.patch  2014-03-30 22:15:27.323962377 -0400
@@ -1,137 +1,24 @@
-Last-Update: 2014-03-03
+Description: libav10 compatibility
+Author: Antoine Martin <antoine@devloop.org.uk>
 Forwarded: not-needed
-Author: Dmitry Smirnov <onlyjob@member.fsf.org>
 Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=739459
 Bug-Xpra: https://www.xpra.org/trac/ticket/524
-Description: fix FTBFS with libav10
 
---- a/xpra/codecs/dec_avcodec/decoder.pyx
-+++ b/xpra/codecs/dec_avcodec/decoder.pyx
-@@ -86,12 +86,12 @@
-         int thread_type
-         int flags
-         int flags2
- 
--    AVPixelFormat PIX_FMT_NONE
--    AVCodecID CODEC_ID_H264
--    AVCodecID CODEC_ID_VP8
--    #AVCodecID CODEC_ID_VP9
-+    AVPixelFormat AV_PIX_FMT_NONE
-+    AVCodecID AV_CODEC_ID_H264
-+    AVCodecID AV_CODEC_ID_VP8
-+    #AVCodecID AV_CODEC_ID_VP9
- 
-     #init and free:
-     void avcodec_register_all()
-     AVCodec *avcodec_find_decoder(AVCodecID id)
-@@ -164,13 +164,13 @@
-     global CODECS
-     if CODECS is None:
-         avcodec_register_all()
-         CODECS = []
--        if avcodec_find_decoder(CODEC_ID_H264)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_H264)!=NULL:
-             CODECS.append("h264")
--        if avcodec_find_decoder(CODEC_ID_VP8)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_VP8)!=NULL:
-             CODECS.append("vp8")
--        #if avcodec_find_decoder(CODEC_ID_VP9)!=NULL:
-+        #if avcodec_find_decoder(AV_CODEC_ID_VP9)!=NULL:
-         #    CODECS.append("vp9")
-     return CODECS
- 
- 
-@@ -366,24 +366,24 @@
-                 break
-         if self.colorspace==NULL:
-             error("invalid pixel format: %s", colorspace)
-             return  False
--        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, PIX_FMT_NONE)
--        if self.pix_fmt==PIX_FMT_NONE:
-+        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, AV_PIX_FMT_NONE)
-+        if self.pix_fmt==AV_PIX_FMT_NONE:
-             error("invalid pixel format: %s", colorspace)
-             return  False
-         self.actual_pix_fmt = self.pix_fmt
- 
-         avcodec_register_all()
- 
-         if self.encoding=="h264":
--            self.codec = avcodec_find_decoder(CODEC_ID_H264)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_H264)
-             if self.codec==NULL:
-                 error("codec H264 not found!")
-                 return  False
-         else:
-             assert self.encoding=="vp8"
--            self.codec = avcodec_find_decoder(CODEC_ID_VP8)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_VP8)
-             if self.codec==NULL:
-                 error("codec VP8 not found!")
-                 return  False
- 
---- a/xpra/codecs/dec_avcodec2/decoder.pyx
-+++ b/xpra/codecs/dec_avcodec2/decoder.pyx
-@@ -83,12 +83,12 @@
-         int flags
-         int flags2
-         int refcounted_frames
- 
--    AVPixelFormat PIX_FMT_NONE
--    AVCodecID CODEC_ID_H264
--    AVCodecID CODEC_ID_VP8
--    #AVCodecID CODEC_ID_VP9
-+    AVPixelFormat AV_PIX_FMT_NONE
-+    AVCodecID AV_CODEC_ID_H264
-+    AVCodecID AV_CODEC_ID_VP8
-+    #AVCodecID AV_CODEC_ID_VP9
- 
-     #init and free:
-     void avcodec_register_all()
-     AVCodec *avcodec_find_decoder(AVCodecID id)
-@@ -155,13 +155,13 @@
-     global CODECS
-     if CODECS is None:
-         avcodec_register_all()
-         CODECS = []
--        if avcodec_find_decoder(CODEC_ID_H264)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_H264)!=NULL:
-             CODECS.append("h264")
--        if avcodec_find_decoder(CODEC_ID_VP8)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_VP8)!=NULL:
-             CODECS.append("vp8")
--        #if avcodec_find_decoder(CODEC_ID_VP9)!=NULL:
-+        #if avcodec_find_decoder(AV_CODEC_ID_VP9)!=NULL:
-         #    CODECS.append("vp9")
-     return CODECS
- 
- 
-@@ -267,24 +267,24 @@
-                 break
-         if self.colorspace==NULL:
-             error("invalid pixel format: %s", colorspace)
-             return  False
--        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, PIX_FMT_NONE)
--        if self.pix_fmt==PIX_FMT_NONE:
-+        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, AV_PIX_FMT_NONE)
-+        if self.pix_fmt==AV_PIX_FMT_NONE:
-             error("invalid pixel format: %s", colorspace)
-             return  False
-         self.actual_pix_fmt = self.pix_fmt
- 
-         avcodec_register_all()
- 
-         if self.encoding=="h264":
--            self.codec = avcodec_find_decoder(CODEC_ID_H264)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_H264)
-             if self.codec==NULL:
-                 error("codec H264 not found!")
-                 return  False
-         else:
-             assert self.encoding=="vp8"
--            self.codec = avcodec_find_decoder(CODEC_ID_VP8)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_VP8)
-             if self.codec==NULL:
-                 error("codec VP8 not found!")
-                 return  False
+Index: xpra/codecs/dec_avcodec/decoder.pyx
+===================================================================
+--- a/xpra/codecs/dec_avcodec/decoder.pyx  (revision 5906)
++++ b/xpra/codecs/dec_avcodec/decoder.pyx  (working copy)
+@@ -109,7 +109,13 @@
+     int avcodec_default_get_buffer(AVCodecContext *s, AVFrame *pic)
+     void avcodec_default_release_buffer(AVCodecContext *s, AVFrame *pic)
+ 
++cdef extern from "libavcodec/old_codec_ids.h":
++    AVPixelFormat PIX_FMT_NONE
++    AVCodecID CODEC_ID_H264
++    AVCodecID CODEC_ID_VP8
++    #AVCodecID CODEC_ID_VP9
+ 
++
+ def get_version():
+     return (LIBAVCODEC_VERSION_MAJOR, LIBAVCODEC_VERSION_MINOR, LIBAVCODEC_VERSION_MICRO)
  
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav-no0RGB.patch debian/patches/libav-no0RGB.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav-no0RGB.patch 2014-02-02 22:17:00.000000000 -0500
+++ debian/patches/libav-no0RGB.patch 2014-03-30 22:12:09.411752548 -0400
@@ -36,12 +36,13 @@
  AV_PIX_FMT_GBRP
 --- a/xpra/codecs/dec_avcodec/decoder.pyx
 +++ b/xpra/codecs/dec_avcodec/decoder.pyx
-@@ -140,8 +140,6 @@
-             "YUV422P"   : "AV_PIX_FMT_YUV422P",
-             "YUV444P"   : "AV_PIX_FMT_YUV444P",
-             "RGB"       : "AV_PIX_FMT_RGB24",
--            "XRGB"      : "AV_PIX_FMT_0RGB",
--            "BGRX"      : "AV_PIX_FMT_BGR0",
-             "ARGB"      : "AV_PIX_FMT_ARGB",
-             "BGRA"      : "AV_PIX_FMT_BGRA",
-             "GBRP"      : "AV_PIX_FMT_GBRP",
+@@ -143,8 +143,6 @@
+         "YUV422P"   : "AV_PIX_FMT_YUV422P",
+         "YUV444P"   : "AV_PIX_FMT_YUV444P",
+         "RGB"       : "AV_PIX_FMT_RGB24",
+-        "XRGB"      : "AV_PIX_FMT_0RGB",
+-        "BGRX"      : "AV_PIX_FMT_BGR0",
+         "ARGB"      : "AV_PIX_FMT_ARGB",
+         "BGRA"      : "AV_PIX_FMT_BGRA",
+         "GBRP"      : "AV_PIX_FMT_GBRP",
+
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/series debian/patches/series
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/series 2014-03-03 16:49:16.000000000 -0500
+++ debian/patches/series 2014-03-30 22:12:09.411752548 -0400
@@ -19,4 +19,4 @@
 libav10.patch
 
 ## Conditional patch, must be the last one.
-old-libav.patch
+#old-libav.patch

However, same result.

Next, I did an svn checkout of the 0.12.x branch with no debian patches. I first got a compile error:

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python2.7 -c xpra/codecs/dec_avcodec/decoder.c -o build/temp.linux-x86_64-2.7/xpra/codecs/dec_avcodec/decoder.o -Wall -Werror=implicit-function-declaration -fPIC
xpra/codecs/dec_avcodec/decoder.c: In function ‘initdecoder’:
xpra/codecs/dec_avcodec/decoder.c:10104:45: error: ‘AV_PIX_FMT_0RGB’ undeclared (first use in this function)
   __pyx_t_1 = __Pyx_PyInt_From_unsigned_int(AV_PIX_FMT_0RGB); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 17; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
                                             ^
xpra/codecs/dec_avcodec/decoder.c:10104:45: note: each undeclared identifier is reported only once for each function it appears in
xpra/codecs/dec_avcodec/decoder.c:10116:45: error: ‘AV_PIX_FMT_BGR0’ undeclared (first use in this function)
   __pyx_t_1 = __Pyx_PyInt_From_unsigned_int(AV_PIX_FMT_BGR0); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 18; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
                                             ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
dh_auto_build: python setup.py build --force returned exit code 1
make: *** [build] Error 1
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1364:
dpkg-buildpackage -rfakeroot -D -us -uc -b failed
29 jamie@animal:src$

So, I just applied this patch:

--- a/xpra/codecs/csc_swscale/constants.txt
+++ b/xpra/codecs/csc_swscale/constants.txt
@@ -14,8 +14,8 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
+#AV_PIX_FMT_0RGB
+#AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/constants.txt
+++ b/xpra/codecs/dec_avcodec/constants.txt
@@ -14,8 +14,8 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
+#AV_PIX_FMT_0RGB
+#AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/decoder.pyx
+++ b/xpra/codecs/dec_avcodec/decoder.pyx
@@ -143,8 +143,6 @@
         "YUV422P"   : "AV_PIX_FMT_YUV422P",
         "YUV444P"   : "AV_PIX_FMT_YUV444P",
         "RGB"       : "AV_PIX_FMT_RGB24",
-        "XRGB"      : "AV_PIX_FMT_0RGB",
-        "BGRX"      : "AV_PIX_FMT_BGR0",
         "ARGB"      : "AV_PIX_FMT_ARGB",
         "BGRA"      : "AV_PIX_FMT_BGRA",
         "GBRP"      : "AV_PIX_FMT_GBRP",

And it successfully compiled.

However, I'm still seeing the server memory leak.

Let me know if there are any other tests you'd like me to perform or any other logs I can provide to help get to the bottom of this.

comment:11 Changed 8 years ago by Antoine Martin

Milestone: → 0.12
Owner: changed from jamie to Antoine Martin
Status: new → assigned

Confirmed here at last, took a few hours, using glxspheres made it happen quicker (lots of frames per second).

Will take care of it as a priority. Wiki link: wiki/Debugging

Last edited 8 years ago by Antoine Martin (previous) (diff)

comment:12 Changed 8 years ago by Antoine Martin

Output from top and running glxspheres with XPRA_XSHM=0:

20450 antoine   20   0 65.240g 101628  32916 R  97.2  0.8   0:06.47
20450 antoine   20   0 65.240g 111532  32976 S  94.5  0.9   2:39.92
20450 antoine   20   0 65.583g 110928  32984 S  92.6  0.9   7:59.86
20450 antoine   20   0 65.587g 126728  32984 S  94.7  1.0   9:54.56
20450 antoine   20   0 65.578g 113120  32984 S  95.0  0.9  10:20.34
20450 antoine   20   0 65.587g 133464  32984 S  93.2  1.1  12:33.52
20450 antoine   20   0 65.649g 156240  32984 S  92.2  1.3  17:20.23
20450 antoine   20   0 65.645g 177276  32984 R  88.7  1.4  18:57.47

Still leaking, so it isn't xshm refcount related.

comment:13 Changed 8 years ago by Antoine Martin

With XPRA_DETECT_LEAKS=1 support added in r6013, I see:

2014-04-04 12:01:46,738 print_leaks():
       55 : <type 'tuple'> : 2829 matches
       27 : <type 'cell'> : 762 matches
       22 : <type 'frame'> : 274 matches
        9 : <type 'list'> : 2883 matches
        7 : <type 'builtin_function_or_method'> : []
        4 : <type 'instancemethod'> : []
        2 : <type 'instance'> : 442 matches

Then a little while later:

2014-04-04 12:03:26,848 print_leaks():
       54 : <type 'tuple'> : 3383 matches
       34 : <type 'cell'> : 1173 matches
       17 : <type 'frame'> : 469 matches
       15 : <type 'builtin_function_or_method'> : []
        9 : <type 'list'> : 2982 matches
        6 : <type 'xpra.x11.bindings.ximage.XShmImageWrapper'> : 47 matches
        3 : <type 'instancemethod'> : []
        2 : <type 'dict'> : 3783 matches

2014-04-04 13:57:21,998 print_leaks():
       52 : <type 'tuple'> : 11378 matches
       45 : <type 'cell'> : 6503 matches
       35 : <type 'instancemethod'> : []
       23 : <type 'list'> : 6089 matches
       22 : <type 'frame'> : 9958 matches
       13 : <type 'dict'> : 3923 matches
       10 : <type 'collections.deque'> : 55 matches
        7 : <type 'builtin_function_or_method'> : []
        3 : <type 'xpra.x11.bindings.ximage.XShmImageWrapper'> : 129 matches

So, it looks like we're leaking:

  • tuples, dicts and lists (probably as part of other objects)
  • frames - no idea why we would leak those
  • cells (whatever they are...)
  • XShmImageWrapper is back in the picture (maybe more than one leak then...)
Last edited 8 years ago by Antoine Martin (previous) (diff)

Changed 8 years ago by Antoine Martin

Attachment: no-video-subregion.patch​ added

disable subregion code

comment:14 Changed 8 years ago by Antoine Martin

After some more improvements to the leak debugging code in r6017, r6021, r6022

I have narrowed it down to the video subregions code, using the attachment/ticket/535/no-video-subregion.patch seems to have stopped the leak.

Unless I can fix this reliably in the next 24 hours, I may disable video subregions (server side) for the next stable update.

@onlyjob and @jamie: can you please confirm that the patch does prevent the leak you were seeing?

comment:15 Changed 8 years ago by onlyjob

I had troubles reproducing the leak lately and I'm not sure why. It could be that it was leaking only while I was using some application (libreoffice?) that I don't use too often...

Last edited 8 years ago by onlyjob (previous) (diff)

comment:16 Changed 8 years ago by Antoine Martin

I had troubles reproducing the leak lately


Me too, and that makes me nervous... maybe the "fix" is only hiding the real cause, by making us follow a different codepath.

The debug output still points towards a very slow leak:

  • XShmImageWrapper: But the verbose debug code does not find anything referencing those objects!??!
  • <type 'frame'>: it could just be the debug code causing those.. heisenbug?

The other objects that cycle a lot are all fairly stable at around:

        3 : <type 'cell'>     : 459 matches
        1 : <type 'dict'>     : 3593 matches
        2 : <type 'list'>     : 2806 matches
        1 : <type 'instance'> : 427 matches
       12 : <type 'tuple'>    : 2365 matches

The two could be related. More investigation is needed.

I did find some other interesting leak issues along the way, like a fairly rare case when clients disconnect, fixed in r6023 - will need a backport.

comment:17 Changed 8 years ago by jamie

Unfortunately (or fortunately?) I'm still able to reproduce the memory leak, even with this patch. I tried applying the patch against the 12.x tag and against trunk itself. Here's the diff of my svn checkout that I most recently build a deb package from:

0 jamie@animal:xpra.svn.trunk$ svn diff
Index: src/xpra/codecs/csc_swscale/constants.txt
===================================================================
--- src/xpra/codecs/csc_swscale/constants.txt	(revision 6023)
+++ src/xpra/codecs/csc_swscale/constants.txt	(working copy)
@@ -14,8 +14,6 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
Index: src/xpra/codecs/dec_avcodec/constants.txt
===================================================================
--- src/xpra/codecs/dec_avcodec/constants.txt	(revision 6023)
+++ src/xpra/codecs/dec_avcodec/constants.txt	(working copy)
@@ -14,8 +14,6 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
Index: src/xpra/server/window_video_source.py
===================================================================
--- src/xpra/server/window_video_source.py	(revision 6023)
+++ src/xpra/server/window_video_source.py	(working copy)
@@ -245,6 +245,7 @@
 
 
     def identify_video_subregion(self):
+	return
         if self.statistics.damage_events_count < self.video_subregion_set_at:
             #stats got reset
             self.video_subregion_set_at = 0
0 jamie@animal:xpra.svn.trunk$

I've just applied one debian patch (without which it fails to compile) and the patch you attached.

However, still getting substantial memory leak every time I click a link in my web browser.

Here's a example - I ran the command once, then clicked on a link which loaded a new web page and then I ran it again:

0 jamie@animal:~$ ps -e -o rss,cmd | egrep '[X]pra' | grep ice
148928 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$ ps -e -o rss,cmd | egrep '[X]pra' | grep ice
151040 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$

Changed 8 years ago by jamie

Attachment: :201.2.log​ added

Server log with XPRA_DETECT_LEAKS=1

comment:18 Changed 8 years ago by jamie

I just attached a server log with XPRA_DETECT_LEAKS=1 - let me know if there is any more debugging I can do that would be helpful.

comment:19 Changed 8 years ago by Antoine Martin

@jamie: I have only just noticed that the process whose memory you are claiming is going up is the X11 vfb process and not xpra itself.

Even if it was the xpra process, the memory going up by just 3MB when doing something is not in itself evidence of a leak, only if the increase is persistent.
Because of python's garbage collection, the memory will fluctuate, it is the long term trend that is more telling.
We also collect statistics, delta buffers, video encoders, etc... and those all take time to initialize and populate fully.

Version 1, edited 8 years ago by Antoine Martin (previous) (next) (diff)

comment:20 Changed 8 years ago by Antoine Martin

Running for over 30 minutes with the 0.12.x branch as of r6036, and I saw no real change with mmap:

  • 11:44 xpra server rss: 246220, Xorg rss: 116036
  • 12:19 xpra server rss: 226732, Xorg rss: 116036

Or with plain rgb:

  • 12:21 xpra server rss: 98968, Xorg rss: 131432
  • 12:48 xpra server rss: 96932, Xorg rss: 131432

So... I just cannot reproduce any leaks. What am I missing?

Changed 8 years ago by Antoine Martin

Attachment: debux-xshmimage-leak.patch​ added

patch I've used to verify that we are not leaking xshm image wrappers

comment:21 Changed 8 years ago by Antoine Martin

Another idea, please disable (or even remove) OpenCL support, either:

  • build xpra with --without-opencl (and ensure you don't have an old copy installed)
  • use --csc-modules=cython,swscale at runtime (should work)
  • temporarily move /etc/OpenCL out of the way

Does that make any difference? Some of the OpenCL ICDs (IIRC, AMD's in particular) cause problems with threads and signals when loaded.

comment:22 in reply to:  19 Changed 8 years ago by jamie

Replying to totaam:

@jamie: I have only just noticed that the process whose memory you are claiming is going up is the X11 vfb process and not xpra itself. The X11 server will allocate memory whenever the client application (firefox or other, xpra too) requests any resources. xpra's allocations will go away if you exit the xpra server process (use "xpra exit") - problem with this is that we may also unmap some of the client windows on exit IIRC, so this will free some resources we should not really be accountable for..

Yes! That is definitely true. Sorry - I didn't realize it was not xpra itself. Should I file this against xvfb? Are there other steps you might suggest to help keep this memory usage under control or any other ways of running xpra to avoid it?

Even if it was the xpra process, the memory going up by just 3MB when doing something is not in itself evidence of a leak, only if the increase is persistent.

I'm so sorry to waste your time on this - there is minimal increase in memory in the actual xpra server that I am noticing (the xpra client does go up, but nothing like the X11 vfb process) but nothing worth reporting and I imagine it is fluctuating like you say.

I think we should close this bug.

comment:23 Changed 8 years ago by onlyjob

I agree, let's close this bug for now. I can't reproduce the leak any more. First thing I tried to do is to uninstall "python-webp" on server side. When leak did not manifest for a while I installed patched version "0.2.2-3" of "python-webp" (on Debian) which was also built against "libwebp5". Leak was there with "python-webp-0.2.2-2/libwebp4". If leak was related to libwebp4 or to python-webp prior to 0.2.2-3 then perhaps it was fixed somewhere...

comment:24 Changed 8 years ago by Antoine Martin

Should I file this against xvfb?


Not necessarily, it could still be xpra consuming those X11 server resources.

Does stopping the browser whilst xpra is running restore all the memory?
Does turning off xshm help (it will reduce the memory used, but does it affect the rate significantly?):

XPRA_XSHM=0 xpra start ...

comment:25 Changed 8 years ago by jamie

When closing the browser, it goes down a little (from 146064 kb to 145936) but doesn't restore all of it.

Adding XPRA_XSHM=0 doesn't seem to have any effect on the memory.

comment:26 Changed 8 years ago by Antoine Martin

When closing the browser, it goes down a little (from 146064 kb to 145936) but doesn't restore all of it.


I would have expected it to go down a lot more.
How much does it go down by if you exit the xpra server? (using xpra exit)

comment:27 Changed 8 years ago by jamie

Here's memory usage after three or four clicks:

0 jamie@animal:~$ ps -e -o rss,cmd | grep Xpra | grep iceweasel
182752 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$

Here's memory usage after xpra exit:

0 jamie@animal:~$ ps -e -o rss,cmd | grep Xpra | grep iceweasel
136308 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$
Last edited 8 years ago by jamie (previous) (diff)

comment:28 Changed 8 years ago by Antoine Martin

Owner: changed from Antoine Martin to onlyjob
Status: assigned → new

Sorry for taking so long to get onto this bug, should be fixed in r6874. (easily spotted with ​xrestop)
Further improvements in r6875 and r6876. (will backport at least the first one)

@onlyjob / @jamie: can I close this ticket?

Last edited 8 years ago by Antoine Martin (previous) (diff)

comment:29 Changed 8 years ago by onlyjob

Resolution: → fixed
Status: new → closed

Yes let's close please. I haven't seen this issue for a while.

comment:30 Changed 16 months ago by migration script

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/535

Note: See TracTickets for help on using tickets.