0.11.4 leaks memory on server side. After using attached session for several days I've noticed that xpra process substantially grew in memory (RES > 2.5 GiB; VIRT > 3 GiB). I'm not sure how to troubleshoot this issue but I'm fairly certain that it is not related to audio which I had disabled. For now I have to run xpra upgrade periodically to release RAM.
Could be related to #457.
Does it go away if you remove the webp codec from the system? (see #491)
(remove xpra/codecs/webm)
It may also be libav related...
Thank you for quick reply.
Replying to totaam:
Could be related to #457.
Does it go away if you remove the webp codec from the system? (see #491) (remove
xpra/codecs/webm)
I'll try without webp but it will take days to see the difference. Besides I have impression that those bugs are about leaks on the client side while mine is on server side.
I may also be
libavrelated...
Everything is possible. :) I'm still running libav-9.11 on client and server. It is the same libav that I was using for a while but I started to notice memory leak only recently...
I have impression that those bugs are about leaks on the client side while mine is on server side
True, but I suspect that the webp library may well be leaking when encoding as well as decoding.
I'm still running libav-9.11 on client and server
The reason why I suspect libav by default is twofold:
So, it would be interesting to see if the leak still occurs when using an encoding we know for sure does not leak: rgb or png have never shown signs of leaking - yet. (and to be certain video does not get used in any way, the best way is to not install the codec)
For what's it worth this leak is with RGB encoding (set as default in config)...
It's still worth deleting all the other codecs to ensure that only RGB can be used. More and more the code is using the chosen encoding only as a hint and will switch to other encodings as appropriate (though in theory, RGB is probably the only one that *should* be honoured at all times...).
Can you post the output of: xpra info | grep memory then things get bad?
Does minimizing the windows help? Re-connecting the client?
I seem to be experiencing the same issue. I can easily reproduce (the memory increases with every click of the web browser that I access via xrpa).
I'm running the client and server on the same machine (so I can run my web browser as an isolated user on my machine).
Question: how do I disable the other encodings? I don't seem to have a codecs/webm folder (maybe remove codecs/vpx?). I tried moving the entire codecs folder to codecs.off but it didn't seem to have any effect.
I've attached my /etc/xpra/xrpa.conf file and the server log file.
Here's my memory after starting xpra:
0 jamie@animal:cdtemp.5reEHd$ ps -e -o rss,cmd | egrep "[X]pra" | grep iceweasel 76784 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201 0 jamie@animal:cdtemp.5reEHd$
After launching iceweasel (Firefox)
jamie@animal:cdtemp.5reEHd$ ps -e -o rss,cmd | egrep "[X]pra" | grep iceweasel 110616 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201 0 jamie@animal:cdtemp.5reEHd$
Here's my command for starting xpra (run as iceweasel user):
/usr/bin/python /usr/bin/xpra --mmap-group --socket-dir=/home/iceweasel/xpra-socket --no-mdns start :201
Here's the attach command (run as jamie user):
/usr/bin/python /usr/bin/xpra --socket-dir=/home/iceweasel/xpra-socket --mmap-group --tray-icon=/usr/share/icons/hicolor/scalable/apps/iceweasel.svg --encoding=rgb24 attach :201
Here's the output of the info command:
0 jamie@animal:bin$ sudo -u iceweasel /usr/bin/python /usr/bin/xpra --mmap-group --socket-dir=/home/iceweasel/xpra-socket --no-mdns info :201 | grep memory memory.children.idrss=0 memory.children.inblock=0 memory.children.isrss=0 memory.children.ixrss=0 memory.children.majflt=0 memory.children.maxrss=30048 memory.children.minflt=19940 memory.children.msgrcv=0 memory.children.msgsnd=0 memory.children.nivcsw=54 memory.children.nsignals=0 memory.children.nswap=0 memory.children.nvcsw=261 memory.children.oublock=32 memory.children.stime=0 memory.children.utime=0 memory.server.idrss=0 memory.server.inblock=0 memory.server.isrss=0 memory.server.ixrss=0 memory.server.majflt=4 memory.server.maxrss=64392 memory.server.minflt=27163 memory.server.msgrcv=0 memory.server.msgsnd=0 memory.server.nivcsw=234 memory.server.nsignals=0 memory.server.nswap=0 memory.server.nvcsw=3989 memory.server.oublock=12024 memory.server.stime=0 memory.server.utime=0 0 jamie@animal:bin$
jamie-etc-xpra-xpra.conf
jamie-server-log
Also, minimizing the window causes the memory usage of the server to increase and maximizing the window causes the memory to increase.
Detaching has no impact on memory usage.
Attaching causes the memory usage to increase.
@jamie: what is your distro? What version are you using? Do you get the same results with 0.12.x? Please include the full xpra info as an attachment, not just the memory bits.
how do I disable the other encodings?
You can disable most of them at build time.
At runtime, it is more difficult.
I don't seem to have a codecs/webm folder (maybe remove codecs/vpx?).
Strange, did you build without webp (aka webm) support?
I tried moving the entire codecs folder to codecs.off but it didn't seem to have any effect.
It will have an effect and will certainly break everything. If it didn't, maybe you moved the wrong one.
minimizing the window causes the memory usage of the server to increase and maximizing the window causes the memory to increase
Interesting, maximizing is understandable. But minimizing less so!
Attaching causes the memory usage to increase
This one is expected as we allocate resources for each client.
Full output of xpra info
Thanks for the response. I just added the full output of xpra info.
I'm running Debian Jessie and xpra 0.11.4+dfsg-1.
I will try to compile latest xpra and see if the problem goes away.
Alas, no luck.
I tried installing using the Debian packaging with a 0.12.2 tar ball and slight modifications to the Debian-shipped patches:
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/disable_html5.patch debian/patches/disable_html5.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/disable_html5.patch 2014-02-03 11:23:26.000000000 -0500
+++ debian/patches/disable_html5.patch 2014-03-30 22:12:09.411752548 -0400
@@ -7,7 +7,7 @@
gtk3_ENABLED = client_ENABLED
qt4_ENABLED = client_ENABLED
opengl_ENABLED = client_ENABLED
--html5_ENABLED = True
+-html5_ENABLED = not WIN32 and not OSX
+html5_ENABLED = False
bencode_ENABLED = True
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/enable-x264-opencl.patch debian/patches/enable-x264-opencl.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/enable-x264-opencl.patch 2014-02-05 09:54:38.000000000 -0500
+++ debian/patches/enable-x264-opencl.patch 2014-03-30 22:12:09.411752548 -0400
@@ -6,18 +6,16 @@
--- a/xpra/codecs/enc_x264/encoder.pyx
+++ b/xpra/codecs/enc_x264/encoder.pyx
-@@ -69,8 +69,11 @@
- int i_bframe_pyramid #Keep some B-frames as references: 0=off, 1=strict hierarchical, 2=normal
+@@ -117,6 +117,9 @@
int b_open_gop
int b_bluray_compat
+ #not all builds support this:
+ int b_opencl #use OpenCL when available
+
+ rc rc #rate control
+
ctypedef struct x264_t:
- pass
- ctypedef struct x264_nal_t:
- int i_ref_idc
@@ -250,8 +253,9 @@
#we don't want IDR frames either:
param.i_keyint_min = 999999
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav10.patch debian/patches/libav10.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav10.patch 2014-03-03 07:33:40.000000000 -0500
+++ debian/patches/libav10.patch 2014-03-30 22:15:27.323962377 -0400
@@ -1,137 +1,24 @@
-Last-Update: 2014-03-03
+Description: libav10 compatibility
+Author: Antoine Martin <antoine@devloop.org.uk>
Forwarded: not-needed
-Author: Dmitry Smirnov <onlyjob@member.fsf.org>
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=739459
Bug-Xpra: https://www.xpra.org/trac/ticket/524
-Description: fix FTBFS with libav10
---- a/xpra/codecs/dec_avcodec/decoder.pyx
-+++ b/xpra/codecs/dec_avcodec/decoder.pyx
-@@ -86,12 +86,12 @@
- int thread_type
- int flags
- int flags2
-
-- AVPixelFormat PIX_FMT_NONE
-- AVCodecID CODEC_ID_H264
-- AVCodecID CODEC_ID_VP8
-- #AVCodecID CODEC_ID_VP9
-+ AVPixelFormat AV_PIX_FMT_NONE
-+ AVCodecID AV_CODEC_ID_H264
-+ AVCodecID AV_CODEC_ID_VP8
-+ #AVCodecID AV_CODEC_ID_VP9
-
- #init and free:
- void avcodec_register_all()
- AVCodec *avcodec_find_decoder(AVCodecID id)
-@@ -164,13 +164,13 @@
- global CODECS
- if CODECS is None:
- avcodec_register_all()
- CODECS = []
-- if avcodec_find_decoder(CODEC_ID_H264)!=NULL:
-+ if avcodec_find_decoder(AV_CODEC_ID_H264)!=NULL:
- CODECS.append("h264")
-- if avcodec_find_decoder(CODEC_ID_VP8)!=NULL:
-+ if avcodec_find_decoder(AV_CODEC_ID_VP8)!=NULL:
- CODECS.append("vp8")
-- #if avcodec_find_decoder(CODEC_ID_VP9)!=NULL:
-+ #if avcodec_find_decoder(AV_CODEC_ID_VP9)!=NULL:
- # CODECS.append("vp9")
- return CODECS
-
-
-@@ -366,24 +366,24 @@
- break
- if self.colorspace==NULL:
- error("invalid pixel format: %s", colorspace)
- return False
-- self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, PIX_FMT_NONE)
-- if self.pix_fmt==PIX_FMT_NONE:
-+ self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, AV_PIX_FMT_NONE)
-+ if self.pix_fmt==AV_PIX_FMT_NONE:
- error("invalid pixel format: %s", colorspace)
- return False
- self.actual_pix_fmt = self.pix_fmt
-
- avcodec_register_all()
-
- if self.encoding=="h264":
-- self.codec = avcodec_find_decoder(CODEC_ID_H264)
-+ self.codec = avcodec_find_decoder(AV_CODEC_ID_H264)
- if self.codec==NULL:
- error("codec H264 not found!")
- return False
- else:
- assert self.encoding=="vp8"
-- self.codec = avcodec_find_decoder(CODEC_ID_VP8)
-+ self.codec = avcodec_find_decoder(AV_CODEC_ID_VP8)
- if self.codec==NULL:
- error("codec VP8 not found!")
- return False
-
---- a/xpra/codecs/dec_avcodec2/decoder.pyx
-+++ b/xpra/codecs/dec_avcodec2/decoder.pyx
-@@ -83,12 +83,12 @@
- int flags
- int flags2
- int refcounted_frames
-
-- AVPixelFormat PIX_FMT_NONE
-- AVCodecID CODEC_ID_H264
-- AVCodecID CODEC_ID_VP8
-- #AVCodecID CODEC_ID_VP9
-+ AVPixelFormat AV_PIX_FMT_NONE
-+ AVCodecID AV_CODEC_ID_H264
-+ AVCodecID AV_CODEC_ID_VP8
-+ #AVCodecID AV_CODEC_ID_VP9
-
- #init and free:
- void avcodec_register_all()
- AVCodec *avcodec_find_decoder(AVCodecID id)
-@@ -155,13 +155,13 @@
- global CODECS
- if CODECS is None:
- avcodec_register_all()
- CODECS = []
-- if avcodec_find_decoder(CODEC_ID_H264)!=NULL:
-+ if avcodec_find_decoder(AV_CODEC_ID_H264)!=NULL:
- CODECS.append("h264")
-- if avcodec_find_decoder(CODEC_ID_VP8)!=NULL:
-+ if avcodec_find_decoder(AV_CODEC_ID_VP8)!=NULL:
- CODECS.append("vp8")
-- #if avcodec_find_decoder(CODEC_ID_VP9)!=NULL:
-+ #if avcodec_find_decoder(AV_CODEC_ID_VP9)!=NULL:
- # CODECS.append("vp9")
- return CODECS
-
-
-@@ -267,24 +267,24 @@
- break
- if self.colorspace==NULL:
- error("invalid pixel format: %s", colorspace)
- return False
-- self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, PIX_FMT_NONE)
-- if self.pix_fmt==PIX_FMT_NONE:
-+ self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, AV_PIX_FMT_NONE)
-+ if self.pix_fmt==AV_PIX_FMT_NONE:
- error("invalid pixel format: %s", colorspace)
- return False
- self.actual_pix_fmt = self.pix_fmt
-
- avcodec_register_all()
-
- if self.encoding=="h264":
-- self.codec = avcodec_find_decoder(CODEC_ID_H264)
-+ self.codec = avcodec_find_decoder(AV_CODEC_ID_H264)
- if self.codec==NULL:
- error("codec H264 not found!")
- return False
- else:
- assert self.encoding=="vp8"
-- self.codec = avcodec_find_decoder(CODEC_ID_VP8)
-+ self.codec = avcodec_find_decoder(AV_CODEC_ID_VP8)
- if self.codec==NULL:
- error("codec VP8 not found!")
- return False
+Index: xpra/codecs/dec_avcodec/decoder.pyx
+===================================================================
+--- a/xpra/codecs/dec_avcodec/decoder.pyx (revision 5906)
++++ b/xpra/codecs/dec_avcodec/decoder.pyx (working copy)
+@@ -109,7 +109,13 @@
+ int avcodec_default_get_buffer(AVCodecContext *s, AVFrame *pic)
+ void avcodec_default_release_buffer(AVCodecContext *s, AVFrame *pic)
+
++cdef extern from "libavcodec/old_codec_ids.h":
++ AVPixelFormat PIX_FMT_NONE
++ AVCodecID CODEC_ID_H264
++ AVCodecID CODEC_ID_VP8
++ #AVCodecID CODEC_ID_VP9
+
++
+ def get_version():
+ return (LIBAVCODEC_VERSION_MAJOR, LIBAVCODEC_VERSION_MINOR, LIBAVCODEC_VERSION_MICRO)
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav-no0RGB.patch debian/patches/libav-no0RGB.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav-no0RGB.patch 2014-02-02 22:17:00.000000000 -0500
+++ debian/patches/libav-no0RGB.patch 2014-03-30 22:12:09.411752548 -0400
@@ -36,12 +36,13 @@
AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/decoder.pyx
+++ b/xpra/codecs/dec_avcodec/decoder.pyx
-@@ -140,8 +140,6 @@
- "YUV422P" : "AV_PIX_FMT_YUV422P",
- "YUV444P" : "AV_PIX_FMT_YUV444P",
- "RGB" : "AV_PIX_FMT_RGB24",
-- "XRGB" : "AV_PIX_FMT_0RGB",
-- "BGRX" : "AV_PIX_FMT_BGR0",
- "ARGB" : "AV_PIX_FMT_ARGB",
- "BGRA" : "AV_PIX_FMT_BGRA",
- "GBRP" : "AV_PIX_FMT_GBRP",
+@@ -143,8 +143,6 @@
+ "YUV422P" : "AV_PIX_FMT_YUV422P",
+ "YUV444P" : "AV_PIX_FMT_YUV444P",
+ "RGB" : "AV_PIX_FMT_RGB24",
+- "XRGB" : "AV_PIX_FMT_0RGB",
+- "BGRX" : "AV_PIX_FMT_BGR0",
+ "ARGB" : "AV_PIX_FMT_ARGB",
+ "BGRA" : "AV_PIX_FMT_BGRA",
+ "GBRP" : "AV_PIX_FMT_GBRP",
+
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/series debian/patches/series
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/series 2014-03-03 16:49:16.000000000 -0500
+++ debian/patches/series 2014-03-30 22:12:09.411752548 -0400
@@ -19,4 +19,4 @@
libav10.patch
## Conditional patch, must be the last one.
-old-libav.patch
+#old-libav.patch
However, same result.
Next, I did an svn checkout of the 0.12.x branch with no debian patches. I first got a compile error:
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python2.7 -c xpra/codecs/dec_avcodec/decoder.c -o build/temp.linux-x86_64-2.7/xpra/codecs/dec_avcodec/decoder.o -Wall -Werror=implicit-function-declaration -fPIC
xpra/codecs/dec_avcodec/decoder.c: In function ‘initdecoder’:
xpra/codecs/dec_avcodec/decoder.c:10104:45: error: ‘AV_PIX_FMT_0RGB’ undeclared (first use in this function)
__pyx_t_1 = __Pyx_PyInt_From_unsigned_int(AV_PIX_FMT_0RGB); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 17; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
xpra/codecs/dec_avcodec/decoder.c:10104:45: note: each undeclared identifier is reported only once for each function it appears in
xpra/codecs/dec_avcodec/decoder.c:10116:45: error: ‘AV_PIX_FMT_BGR0’ undeclared (first use in this function)
__pyx_t_1 = __Pyx_PyInt_From_unsigned_int(AV_PIX_FMT_BGR0); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 18; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
dh_auto_build: python setup.py build --force returned exit code 1
make: *** [build] Error 1
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1364:
dpkg-buildpackage -rfakeroot -D -us -uc -b failed
29 jamie@animal:src$
So, I just applied this patch:
--- a/xpra/codecs/csc_swscale/constants.txt
+++ b/xpra/codecs/csc_swscale/constants.txt
@@ -14,8 +14,8 @@
AV_PIX_FMT_YUV422P
AV_PIX_FMT_YUV444P
AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
+#AV_PIX_FMT_0RGB
+#AV_PIX_FMT_BGR0
AV_PIX_FMT_ARGB
AV_PIX_FMT_BGRA
AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/constants.txt
+++ b/xpra/codecs/dec_avcodec/constants.txt
@@ -14,8 +14,8 @@
AV_PIX_FMT_YUV422P
AV_PIX_FMT_YUV444P
AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
+#AV_PIX_FMT_0RGB
+#AV_PIX_FMT_BGR0
AV_PIX_FMT_ARGB
AV_PIX_FMT_BGRA
AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/decoder.pyx
+++ b/xpra/codecs/dec_avcodec/decoder.pyx
@@ -143,8 +143,6 @@
"YUV422P" : "AV_PIX_FMT_YUV422P",
"YUV444P" : "AV_PIX_FMT_YUV444P",
"RGB" : "AV_PIX_FMT_RGB24",
- "XRGB" : "AV_PIX_FMT_0RGB",
- "BGRX" : "AV_PIX_FMT_BGR0",
"ARGB" : "AV_PIX_FMT_ARGB",
"BGRA" : "AV_PIX_FMT_BGRA",
"GBRP" : "AV_PIX_FMT_GBRP",
And it successfully compiled.
However, I'm still seeing the server memory leak.
Let me know if there are any other tests you'd like me to perform or any other logs I can provide to help get to the bottom of this.
Confirmed here at last, took a few hours, using glxspheres made it happen quicker (lots of frames per second).
Will take care of it as a priority. Wiki link: wiki/Debugging
Output from top and running glxspheres with XPRA_XSHM=0:
20450 antoine 20 0 65.240g 101628 32916 R 97.2 0.8 0:06.47 20450 antoine 20 0 65.240g 111532 32976 S 94.5 0.9 2:39.92 20450 antoine 20 0 65.583g 110928 32984 S 92.6 0.9 7:59.86 20450 antoine 20 0 65.587g 126728 32984 S 94.7 1.0 9:54.56 20450 antoine 20 0 65.578g 113120 32984 S 95.0 0.9 10:20.34 20450 antoine 20 0 65.587g 133464 32984 S 93.2 1.1 12:33.52 20450 antoine 20 0 65.649g 156240 32984 S 92.2 1.3 17:20.23 20450 antoine 20 0 65.645g 177276 32984 R 88.7 1.4 18:57.47
Still leaking, so it isn't xshm refcount related.
With XPRA_DETECT_LEAKS=1 support added in r6013, I see:
2014-04-04 12:01:46,738 print_leaks():
55 : <type 'tuple'> : 2829 matches
27 : <type 'cell'> : 762 matches
22 : <type 'frame'> : 274 matches
9 : <type 'list'> : 2883 matches
7 : <type 'builtin_function_or_method'> : []
4 : <type 'instancemethod'> : []
2 : <type 'instance'> : 442 matches
Then a little while later:
2014-04-04 12:03:26,848 print_leaks():
54 : <type 'tuple'> : 3383 matches
34 : <type 'cell'> : 1173 matches
17 : <type 'frame'> : 469 matches
15 : <type 'builtin_function_or_method'> : []
9 : <type 'list'> : 2982 matches
6 : <type 'xpra.x11.bindings.ximage.XShmImageWrapper'> : 47 matches
3 : <type 'instancemethod'> : []
2 : <type 'dict'> : 3783 matches
2014-04-04 13:57:21,998 print_leaks():
52 : <type 'tuple'> : 11378 matches
45 : <type 'cell'> : 6503 matches
35 : <type 'instancemethod'> : []
23 : <type 'list'> : 6089 matches
22 : <type 'frame'> : 9958 matches
13 : <type 'dict'> : 3923 matches
10 : <type 'collections.deque'> : 55 matches
7 : <type 'builtin_function_or_method'> : []
3 : <type 'xpra.x11.bindings.ximage.XShmImageWrapper'> : 129 matches
So, it looks like we're leaking:
tuples, dicts and lists (probably as part of other objects)
frames - no idea why we would leak those
cells (whatever they are...)
XShmImageWrapper is back in the picture (maybe more than one leak then...)
disable subregion code
After some more improvements to the leak debugging code in r6017, r6021, r6022
I have narrowed it down to the video subregions code, using the attachment/ticket/535/no-video-subregion.patch seems to have stopped the leak.
Unless I can fix this reliably in the next 24 hours, I may disable video subregions (server side) for the next stable update.
@onlyjob and @jamie: can you please confirm that the patch does prevent the leak you were seeing?
I had troubles reproducing the leak lately and I'm not sure why. It could be that it was leaking only while I was using some application (libreoffice?) that I don't use too often...
I had troubles reproducing the leak lately
Me too, and that makes me nervous... maybe the "fix" is only hiding the real cause, by making us follow a different codepath.
The debug output still points towards a very slow leak:
XShmImageWrapper: But the verbose debug code does not find anything referencing those objects!??!
<type 'frame'>: it could just be the debug code causing those.. heisenbug?
The other objects that cycle a lot are all fairly stable at around:
3 : <type 'cell'> : 459 matches
1 : <type 'dict'> : 3593 matches
2 : <type 'list'> : 2806 matches
1 : <type 'instance'> : 427 matches
12 : <type 'tuple'> : 2365 matches
The two could be related. More investigation is needed.
I did find some other interesting leak issues along the way, like a fairly rare case when clients disconnect, fixed in r6023 - will need a backport.
Unfortunately (or fortunately?) I'm still able to reproduce the memory leak, even with this patch. I tried applying the patch against the 12.x tag and against trunk itself. Here's the diff of my svn checkout that I most recently build a deb package from:
0 jamie@animal:xpra.svn.trunk$ svn diff
Index: src/xpra/codecs/csc_swscale/constants.txt
===================================================================
--- src/xpra/codecs/csc_swscale/constants.txt (revision 6023)
+++ src/xpra/codecs/csc_swscale/constants.txt (working copy)
@@ -14,8 +14,6 @@
AV_PIX_FMT_YUV422P
AV_PIX_FMT_YUV444P
AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
AV_PIX_FMT_ARGB
AV_PIX_FMT_BGRA
AV_PIX_FMT_GBRP
Index: src/xpra/codecs/dec_avcodec/constants.txt
===================================================================
--- src/xpra/codecs/dec_avcodec/constants.txt (revision 6023)
+++ src/xpra/codecs/dec_avcodec/constants.txt (working copy)
@@ -14,8 +14,6 @@
AV_PIX_FMT_YUV422P
AV_PIX_FMT_YUV444P
AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
AV_PIX_FMT_ARGB
AV_PIX_FMT_BGRA
AV_PIX_FMT_GBRP
Index: src/xpra/server/window_video_source.py
===================================================================
--- src/xpra/server/window_video_source.py (revision 6023)
+++ src/xpra/server/window_video_source.py (working copy)
@@ -245,6 +245,7 @@
def identify_video_subregion(self):
+ return
if self.statistics.damage_events_count < self.video_subregion_set_at:
#stats got reset
self.video_subregion_set_at = 0
0 jamie@animal:xpra.svn.trunk$
I've just applied one debian patch (without which it fails to compile) and the patch you attached.
However, still getting substantial memory leak every time I click a link in my web browser.
Here's a example - I ran the command once, then clicked on a link which loaded a new web page and then I ran it again:
0 jamie@animal:~$ ps -e -o rss,cmd | egrep '[X]pra' | grep ice 148928 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201 0 jamie@animal:~$ ps -e -o rss,cmd | egrep '[X]pra' | grep ice 151040 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201 0 jamie@animal:~$
Server log with XPRA_DETECT_LEAKS=1
I just attached a server log with XPRA_DETECT_LEAKS=1 - let me know if there is any more debugging I can do that would be helpful.
@jamie: I have only just noticed that the process whose memory you are claiming is going up is the X11 vfb process and not xpra itself. The X11 server will allocate memory whenever the client application (firefox or other, xpra too) requests any resources. xpra's allocations will go away if you exit the xpra server process (use "xpra exit") - problem with this is that we may also unmap some of the client windows on exit IIRC, so this will free some resources we should not really be accountable for..
Even if it was the xpra process, the memory going up by just 3MB when doing something is not in itself evidence of a leak, only if the increase is persistent. Because of python's garbage collection, the memory will fluctuate, it is the long term trend that is more telling. We also collect statistics, delta buffers, video encoders, etc... and those all take time to initialize and populate fully.
Running for over 30 minutes with the 0.12.x branch as of r6036, and I saw no real change with mmap:
xpra server rss: 246220, Xorg rss: 116036
xpra server rss: 226732, Xorg rss: 116036
Or with plain rgb:
xpra server rss: 98968, Xorg rss: 131432
xpra server rss: 96932, Xorg rss: 131432
So... I just cannot reproduce any leaks. What am I missing?
patch I've used to verify that we are not leaking xshm image wrappers
Another idea, please disable (or even remove) OpenCL support, either:
--without-opencl (and ensure you don't have an old copy installed)
--csc-modules=cython,swscale at runtime (should work)
/etc/OpenCL out of the way
Does that make any difference? Some of the OpenCL ICDs (IIRC, AMD's in particular) cause problems with threads and signals when loaded.
Replying to totaam:
@jamie: I have only just noticed that the process whose memory you are claiming is going up is the X11 vfb process and not xpra itself. The X11 server will allocate memory whenever the client application (firefox or other, xpra too) requests any resources. xpra's allocations will go away if you exit the xpra server process (use "
xpra exit") - problem with this is that we may also unmap some of the client windows on exit IIRC, so this will free some resources we should not really be accountable for..
Yes! That is definitely true. Sorry - I didn't realize it was not xpra itself. Should I file this against xvfb? Are there other steps you might suggest to help keep this memory usage under control or any other ways of running xpra to avoid it?
Even if it was the xpra process, the memory going up by just 3MB when doing something is not in itself evidence of a leak, only if the increase is persistent.
I'm so sorry to waste your time on this - there is minimal increase in memory in the actual xpra server that I am noticing (the xpra client does go up, but nothing like the X11 vfb process) but nothing worth reporting and I imagine it is fluctuating like you say.
I think we should close this bug.
I agree, let's close this bug for now. I can't reproduce the leak any more. First thing I tried to do is to uninstall "python-webp" on server side. When leak did not manifest for a while I installed patched version "0.2.2-3" of "python-webp" (on Debian) which was also built against "libwebp5". Leak was there with "python-webp-0.2.2-2/libwebp4". If leak was related to libwebp4 or to python-webp prior to 0.2.2-3 then perhaps it was fixed somewhere...
Should I file this against xvfb?
Not necessarily, it could still be xpra consuming those X11 server resources.
Does stopping the browser whilst xpra is running restore all the memory? Does turning off xshm help (it will reduce the memory used, but does it affect the rate significantly?):
XPRA_XSHM=0 xpra start ...
When closing the browser, it goes down a little (from 146064 kb to 145936) but doesn't restore all of it.
Adding XPRA_XSHM=0 doesn't seem to have any effect on the memory.
When closing the browser, it goes down a little (from 146064 kb to 145936) but doesn't restore all of it.
I would have expected it to go down a lot more.
How much does it go down by if you exit the xpra server? (using xpra exit)
Here's memory usage after three or four clicks:
0 jamie@animal:~$ ps -e -o rss,cmd | grep Xpra | grep iceweasel 182752 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201 0 jamie@animal:~$
Here's memory usage after xpra exit:
0 jamie@animal:~$ ps -e -o rss,cmd | grep Xpra | grep iceweasel 136308 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201 0 jamie@animal:~$
Sorry for taking so long to get onto this bug, should be fixed in r6874. (easily spotted with xrestop) Further improvements in r6875 and r6876. (will backport at least the first one)
@onlyjob / @jamie: can I close this ticket?
Yes let's close please. I haven't seen this issue for a while.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/535