After running the client for many hours with glxgears at full throttle, the client eventually ran out of memory and got killed by the oom-killer.
Unfortunately, I was looking for a server leak with "xpra info" (none to be found), so I don't have much data.
I'll have to try again with different encodings to see if this is encoding related or something else. This is not sound related as I had --no-speaker
.
The memory usage seems to go up slowly, at a rate of about 1MB per minute.
Good news is that this is encoding related and not some obscure python object leak:x264
leaks but none of the other encodings do (I've tried vpx
,rgb
and png
).
So the problem clearly is in the dec_avcodec
code. Will try to deal with this together with #415 (also in dec_avcodec
).
Temporary workaround is to use another encoding... sadly, x264 is the best one.
Thanks for looking into this.
Perhaps you could consider doing the sort of thing they do in mythtv which I believe is either statically linking the library or shipping their own version of the library. I believe they also sited a multitude of possible issues on having to support different versions of libraries from different distributions.
I think this would cut your support overhead as well as potentially making the application more stable which if you are expecting to use it for a few years without any major problems due to upgrades etc. would be great. I assume it would also make it easier to add any custom patches required for the library.
This also brings me back to the suggestion of having a test harness or some sort of automated testing that would attempt to look for issues like this or any regressions in the future. Easier said than done I know!
... statically linking the library or shipping their own version ...
No, not when the distributions in question already ship a version we can run against: there is no way we can shoulder the maintenance cost of static builds and bug-fix/security updates for those.
it would also make it easier to add any custom patches required for the library
We do not require any such patches.
What we should do is report the bug to vendor and hope that they fix it..
In this case, it does look like an Ubuntu bug once again, I cannot reproduce this on other distros (though I haven't tried them all) - so I am lowering the priority and will probably release 0.10.10 as it is.
r4872 fixes the memleak with ffmpeg2, the fix for the old libav versions found in Ubuntu is likely to be similar.
updated patch that doesn't seem to leak
found nothing with valgrind, but when I removed part of the old-libav
patch (see updated patch), it seems to not leak anymore.
Please test, there are new beta packages available which include this patch.
OK, that took ages to test properly and narrow down as repeated runs produced different results.
Here's how I ended up testing to mazimize the chances of reproducing the problems:
libav-nofree
patch (see r4955) or the plain old-libav
patch, plus whatever patches are needed to build (usually pixfmt
and no0RGB
)
xterm
and glxgears
(glxgears to generate enough frames to be able to spot the memory leak - assuming it is frame related..)
h264
(since that's the encoding that seemed to cause the leak - see comment:2)
glxgears
then unmaximize, repeat 20 times to be sure
The detailed per-distro per-build-type results can be found in #477
I eventually found that the memory would only leak if sound was enabled *and* there were sound overruns.
OTOH, potential solutions:
It looked like it was encoding related and distro specific, probably because:
oops, didn't mean to close: please help me find what OS leaks and under what conditions.
Trying to use glxgears on my fedora 19 server xpra session xterms produces the error Error: couldn't get an RGB, Double-buffered visual
Trying to find fix for glxgears
led down a rabbit hole (from which I eventually fled with no success).
Testing instead by running google-chrome playing a video:
Windows does not seem to have a leak. Starts at about 100 MB, after 20 minutes 112 MB. Maximize/un-maximize: ... when un-maximizing the memory shows 140-150 MB, which drops within a second to 120 (+/-) MB (presumably dropping memory use with un-maximized vs. maximized). No appreciable difference between the first and the 25th iteration.
(running 2 windows puts memory at 150 MB/ 170 MB when one is maximized)
OSX likewise seems to have no leaks. 0.11.0-r4940 client runs at about 96 MB both before and after 20 minutes (with a laptop, one displayed google-chrome running the above, one minimized background google-chrome running the same in the background). Memory only hopped between about 99 and 94 MB as I maximized/un-maximized 25 times. With 2 windows displaying the javascript, maximizing/un-maximizing one then the other 25 time memory ranged from 100-107 MB.
Still working on getting a suitable fedora client to test.
As per comment:7, the leak seems to be triggered by sound overruns:
I eventually found that the memory would only leak if sound was enabled *and* there were sound overruns.
Did you get any overruns? Did you try to trigger any?
You can use:
XPRA_SOUND_QUEUE_TIME=50 xpra attach...
To lower the size of the sound queue, which should make it easier to get overruns. (the default value is currently 450
)
With the window client (0.11.0-r4940), setting XPRA_SOUND_QUEUE_TIME=50
, there are regular speaker resets and the memory seems to climb rather surprisingly.
starts: 56 MB
10 minutes: 154 MB
20 minutes: 237 MB
30 minutes: 302 MB
50 minutes: 413 MB and the speakers turned off and wouldn't restart
memory kept rising, even with speakers refusing to restart
105 minutes: 591 MB
132 minutes connection lost - server crashed and had to be killed (kill -15
)
start: 80 MB 20 minutes: 256 MB 35 minutes: 321 MB 50 minutes: 414 MB 60 minutes: 459 MB
With osx client, I couldn't cause speaker re-starts. Even setting XPRA_SOUND_QUEUE_TIME=5
memory didn't rise significantly after 30 minutes (from 190 to 195 MB)
there are regular speaker resets and the memory seems to climb rather surprisingly
OK, so this tells us that restarts caused by overruns do cause a memory leak. Let's hope this is the only one.
Any sound debug info?
server crashed and had to be killed
How did it crash? It has to be alive to be killed.. Is this reproducible? What was in the log? Memory usage? What version was this running? Assuming it is just stuck, can we get a gdb backtrace of when it is stuck?
With osx client, I couldn't cause speaker re-starts
Any sound debug log? We used to get overruns reliably, what has changed?
Testing again with win client 0.11.0 r4940 I was unable to reproduce the speaker re-start failures or the server lock-ups, and thusly couldn't get any logs. (ISP was having problems the day of the initial tests, might have been contributing more to the issues than I'd expected.)
I discovered that I wasn't using the command line with the osx client correctly, hence the lack of speaker re-starts. Correctly lowering the XPRA_SOUND_QUEUE_TIME
to 50 does cause speaker restarts, and there is still no sign of a memory leak with osx.
Using 0.11.0 r4998 osx client and fedora 19 r5006 server with XPRA_SOUND_QUEUE_TIME=50
I got the following:
start 74 MB 10 minutes 84 MB 20 minutes 108 MB 30 minutes 120 MB 40 minutes 127 MB 50 minutes 102 MB 60 minutes 112 MB 70 minutes 112 MB 80 minutes 110 MB 90 minutes 119 MB
repeating with XPRA_SOUND_QUEUE_TIME=10
, to be extra thorough, I got about the same:
start 77 MB 10 minutes 95 MB 20 minutes 107 MB 30 minutes 116 MB 40 minutes 99 MB 50 minutes 100 MB 60 minutes 104 MB
It looks like osx is behaving, here at least.
I just noticed that my second round of tests with windows client (with a behaving ISP) didn't post. Correcting that oversight, I'll post them again. The memory still grew, but it was a far more reasonable rate of growth.
win 0.11.0r5016 test 1: connection sound_time=50 0 minutes 84 MB 10 minutes 109 MB 25 minutes 134 MB 35 minutes 163 MB 50 minutes 191 mb 60 minutes 196 MB 80 minutes 221 MB 95 minutes 256 MB 120 minutes 265 MB
test 2 0 minutes 86 MB 10 minutes 157 MB 25 minutes 176 MB 30 minutes 178 MB 40 minutes 193 MB 50 minutes 211 MB 60 minutes 219 MB 70 minutes 248 MB 80 minutes 258 MB
It leaks, but what we want to confirm is what causes the leak.
ie: Does it leak without sound? Does it leak with sound but without sound overruns? How many overruns were there during those tests? Did they coincide with the increase? Does stopping the sound stop the leak? etc
Testing is becoming really drawn out, erratic, and confusing.
It seems like low XPRA_SOUND_QUEUE_TIME
settings do indeed cause more memory growth, but yet it seems like the default sound setting is actually, inexplicably, more efficient than actually using the --no-speaker
setting.
With default settings I got the following:
0 minutes 116 MB 15 minutes 121 MB 30 minutes 127 MB
With XPRA_SOUND_QUEUE_TIME=50
I got:
0 minutes 116 MB 15 minutes 135 MB 30 minutes 156 MB
While with --no-speaker
(engaged from client-side) I got:
0 minutes 100 MB 20 minutes 130 MB 30 minutes 158 MB
Just out of curiosity I also tried --no-pulseaudio
server-side... which didn't disable the sound, surprisingly, and turned up the following:
0 minutes 92 MB 10 minutes 95 MB 20 minutes 99 MB 30 minutes 102 MB
Once I'd tried that though, I was no longer able to disable speakers on the client-side with --no-speaker
until after I'd rebooted the server VM.
Trying --no-speaker
from the server side gave the following:
0 minutes 85 MB 10 minutes 98 MB 20 minutes 105 MB 30 minutes 122 MB
The output was hard to reproduce however, as another attempt with --no-speaker --opengl=on
on the client-side gave the following:
0 minutes 87 MB 10 minutes 91 MB 20 minutes 99 MB 30 minutes 108 MB 40 minutes 113 MB
Open GL doesn't seem to be a large contributor long term though, trying with --no-speaker --opengl=off
client-side came up with this:
0 minutes 68 MB 10 minutes 81 MB 20 minutes 95 MB 30 minutes 107 MB 40 minutes 123 MB 50 minutes 127 MB 60 minutes 133 MB
which isn't hugely different than with the --opengl=on
.
Setting XPRA_SOUND_QUEUE_TIME=50 --opengl=off
came up with this:
0 minutes 70 MB 10 minutes 83 MB 20 minutes 111 MB 30 minutes 151 MB 40 minutes 179 MB 50 minutes 198 MB 60 minutes 218 MB
which is extremely similar to what I got with defaults (which, I think, is opengl=on... though now I might have to go back and double check that).
I did notice a little difference between memory when running one long video vs. running a player that loaded a number of shorter videos over the same period of time.
(--no-speaker
, long video)
0 minutes 100 MB
20 minutes 130 MB
30 minutes 158 MB
(--no-speaker
, lots of shorts)
0 minutes 100 MB
20 minutes 159 MB
30 minutes 174 MB
I was basically testing by streaming hulu.com (streaming tv) - and I did notice that the memory would often jump by a couple 2-5 MB whenever a commercial loaded... and SIGINTing the chrome browser from an xpra session xterm and leaving the xpra session to run with no video, just inactive xterms, I noted that the memory didn't grow - but it also didn't diminish.
With the XPRA_SOUND_QUEUE_TIME=50
I saw (& heard) a lot of speaker restarts. With the default setting I still saw them occasionally. As expected, I did not see any with --no-speaker
, at least not when the flag worked.
Anyway, does any of this give you a hint? Would you like me to try out other codecs? Try other flags? Let me know if anything comes to mind.
It seems like low
XPRA_SOUND_QUEUE_TIME
settings do indeed cause more memory growth, but yet it seems like the default sound setting is actually, inexplicably, more efficient than actually using the--no-speaker
setting.
That would be very odd.
The problem with the figures above is that I can't see how many overruns there were for each test, so we can't correlate overruns with leaks. My guess is that each overrun will cause a small leak, either a fixed amount of memory, or maybe the amount of buffers in the queue at the time, or both.
Just out of curiosity I also tried
--no-pulseaudio
server-side... which didn't disable the sound, surprisingly [...]
It isn't meant to, the man page says:
--no-pulseaudio Disables the starting of a pulseaudio server with the session.
If a pulseaudio server already exists, it makes no difference. In fact, in that case, you should see an error message when *not* using the flag. (see the pulseaudio warning note in the wiki/FAQ)
Once I'd tried that though, I was no longer able to disable speakers on the client-side with
--no-speaker
until after I'd rebooted the server VM.
I don't understand what that means: no longer able ..
. Also, why would rebooting a VM affect the process running inside it?
Trying
--no-speaker
from the server side gave the following:
Using the flag server side or client side should not make any difference, from the help page:
These options can be used to turn certain features on or off, they can be specified on the client or on the server, but the client cannot enable them if they are disabled on the server.
I did notice a little difference between memory when running one long video vs. running a player that loaded a number of shorter videos over the same period of time.
Could be caused by key frames in h264 decoding, which take up more space, and are meant to be kept around longer.
Anyway, does any of this give you a hint? Would you like me to try out other codecs? Try other flags? Let me know if anything comes to mind.
Yes, I think there may well be two different leaks:
So let's start by isolating each one. Please try a different picture encoding (ie: plain rgb or jpeg). First with --no-speaker
and --opengl=off
: I am hoping you won't be seeing any leak at all in that case.
Assuming this behaves ok, turn things on again one at a time to confirm which specific setting(s) really do cause a leak.
Another thing that is worth trying is sound on its own, with the video window minimized so we don't generate any video frame decoding at all. Bearing in mind that the lack of video traffic may cause fewer sound overruns, which makes the results harder to interpret if we don't show how many overruns there were in both cases.
Please also post the output of: Encoding_Info.exe
If the leak is h264 decoding related, it is then worth checking if the same leak occurs with vpx, and if not we have at least the option of trying ffmpeg2 decoding instead (#415). Later on, it may also be worth trying vpx decoding via ffmpeg instead of native: that would tell us if the leak is in the ffmpeg core (buffer handling), of the h264 decoding code (the easiest way of doing this will be to delete the vpx codec from library.zip
- or building the win32 installer with vpx turned off).
One reassuring fact is that I did not see any memory leaks on all the *nix client platforms that I tested before releasing 0.10.10
: 4 versions of Ubuntu, 3 versions of Debian, 3 versions of Fedora, some BSD
s - the most interesting ones amongst these are: Ubuntu 12.04 (no h264 decoding), Ubuntu 13.10 (latest libav), Fedora 19 (current ffmpeg1) and Fedora 20 or later (ffmpeg2). But then again, I only tested with glxgears
, you may want to verify this assumption by running your usual test process against some of those to confirm.
Just a couple quick clarifications for the moment (in case there's something more than mere pedantry involved).
Once I'd tried that though, I was no longer able to disable speakers on the client-side with --no-speaker until after I'd rebooted the server VM.
I was no longer able...
means that, once I had tried the --no-pulseaudio
then, after a control-c SIGINT to shut down the server xpra session on the given display, when I started a new xpra server session and then tried to attach with a client-side windows xpra session... the --no-speaker
option no longer had any effect (there was still sound).
Stopping and starting the server (and client) xpra sessions did not fix the issue, and neither did a fedora 19 server-side killall pulseaudio
. I finally had to reboot the fedora server, before the --no-speaker
flag would work, client-side, as expected. (I suppose I will have to repeat that some time and see if it is something reproducible, or if it was just a result of a (VM) server running too long.)
As for the number of speaker resets (a far more potentially useful bit of information), with the default sound queue setting, I would say there were resets maybe every 5 or 10 minutes at most... and often none for 20 minutes or more at a time. Meanwhile, with the queue set to 50 there were often speaker restarts every couple of seconds, sometimes multiple per second. Obviously, when --no-speaker
worked, then there were no speaker restarts.
Anyway, I'll dig in and try some permutations now.
Testing with --encoding=rgb
The memory usage stayed in the 30 - 80 MB range, over the course of about an hour of streaming video. The results were ball park similar with --no-speaker
, XPRA_SOUND_QUEUE_TIME=50
, or with the default sound setting. In all these cases the opengl was off, but the earlier tests showed no particular difference with opengl on (although I could test that explicitly to be sure if you think it might help).
With sound queue at default, I was getting 2 - 5 speaker restarts per hour, with sound queue set to 50 I was getting a speaker restart every 2 seconds or so (and it wasn't uncommon for the speakers to fail to restart for a few minutes). Again, there was no appreciable difference between the two (69 MB at 40 minutes with default sound, 72 MB at 40 minutes with XPRA_SOUND_QUEUE_TIME=50
).
Some testing with --encoding=vpx
showed about the same as with rgb, and there wasn't any appreciable difference with or without speakers (64 MB at 20 minutes with --no-speaker
vs. 63 MB at 20 minutes with speakers at default).
Just for completeness, the Encoding_Info.exe
produced this:
codecs/csc modules found: * PIL : True <module 'PIL' from 'C:\Program Files (x86)\Xpra\library.zip\PIL\__init__.pyc'> * enc_vpx : True <module 'xpra.codecs.vpx.encoder' from 'C:\Program Files (x86)\Xpra\xpra.codecs.vpx.encoder.pyd'> * dec_vpx : True <module 'xpra.codecs.vpx.decoder' from 'C:\Program Files (x86)\Xpra\xpra.codecs.vpx.decoder.pyd'> * enc_x264 : False * enc_nvenc : False * csc_swscale : True <module 'xpra.codecs.csc_swscale.colorspace_converter' from 'C:\Program Files (x86)\Xpra\xpra.codecs.csc_swscale.colorspace_converter.pyd'> * csc_cython : True <module 'xpra.codecs.csc_cython.colorspace_converter' from 'C:\Program Files (x86)\Xpra\xpra.codecs.csc_cython.colorspace_converter.pyd'> * csc_opencl : False * csc_nvcuda : False * dec_avcodec : True <module 'xpra.codecs.dec_avcodec.decoder' from 'C:\Program Files (x86)\Xpra\xpra.codecs.dec_avcodec.decoder.pyd'> * enc_webp : True <module 'xpra.codecs.webm.encode' from 'C:\Program Files (x86)\Xpra\library.zip\xpra\codecs\webm\encode.pyc'> * enc_webp_lossless : True <module 'xpra.codecs.webm.encode' from 'C:\Program Files (x86)\Xpra\library.zip\xpra\codecs\webm\encode.pyc'> * webp_bitmap_handlers : True <module 'xpra.codecs.webm.handlers' from 'C:\Program Files (x86)\Xpra\library.zip\xpra\codecs\webm\handlers.pyc'> * dec_webp : True <module 'xpra.codecs.webm.decode' from 'C:\Program Files (x86)\Xpra\library.zip\xpra\codecs\webm\decode.pyc'> codecs versions: * cython : (0, 1) * PIL : 1.1.7 * avcodec : (54, 92, 100) * vpx : v1.2.0 * webp : 0.2.2 * swscale : (2, 2, 100) Press Enter to close
disables threading in avcodec decoding
First, for comment:21
then tried to attach with a client-side windows xpra session... the --no-speaker option no longer had any effect (there was still sound).
I don't understand how that is possible. Are you sure the sound is coming from xpra and not some other sound forwarding system (VirtualBox
or other).
Please create a new ticket and include full command lines used, debug information from both xpra info and the session info dialog (showing sound codecs and sound status).
Now back to the leak.
Let me try to recap the previous comments. Please confirm.
To reproduce the leak, we need a combination of things:
XPRA_SOUND_QUEUE_TIME=50
)
Things to try next:
I'll confirm your summary.
I would also note, however, with h264 the memory leak is evident even with the --no-speakers
option.
Testing with 0.9.(various) and 0.10.(various) also shows no sign of the leak, with default sound queue or with XPRA_SOUND_QUEUE_TIME=50
. (It looks like it is a regression.)
I'll try to get some testing done with the threading as soon as I can.
Testing with the avcodec_nothreading.patch
with a 0.11.0 r5153 win client with default SOUND_QUEUE_TIME (but about 20 speaker re-starts in 40 minutes) it looks like the memory growth is rather stable, also.
0 minutes 103 MB 10 minutes 110 MB 20 minutes 116 MB 30 minutes 120 MB 40 minutes 123 MB
diff between v0.10 version of the codec and v0.11
partial revert of r4815: use frame.data instead of opaque field
And... the plot thickens, I am seeing the leak now on Fedora 19 (and I don't think I saw it before when I did a thorough testing of many distros, for example for #477, but maybe this leak is in ffmpeg and not in the libav fork used by Debian and Ubuntu.. or maybe it is too slow to notice?), without sound enabled, and with or without the noopaque
patch below. Taking ownership of the ticket now that I can reproduce it reliably (I hope). I am still puzzled by the effect that sound restarts seem to have on this bug (OTOH: a threading issue?, an existing issue made worse by the extra load?)
On Fedora 19, 0.10.11
is also affected by this leak (and maybe leaking a tad less).
So the next step will be to really identify which versions and which distros leak and by how much, because at this point it looks like all the assumptions from previous comments were wrong.
(probably not very relevant but still)
Before I found out that this is not a regression (at least on Fedora, probably others too, contrary to comment:24), I had spent a lot of time inspecting the code changes in avcodec: attachment/ticket/457/avcodec-v10-to-v11.diff shows the full diff between current v0.10.x branch and current trunk. Summary of noteworthy changes:
This patch: attachment/ticket/457/avcodec-noopaque.patch does not seem to make any noticeable difference.
Full list of changesets affecting this file since v0.10.0: r5105, r5004, r4997, r4990, r4989, r4933, r4851, r4816, r4815, r4812, r4810, r4786, r4707, r4667, r4666, r4661, r4657, r4652, r4533, r4480, r4345, r4292, r4245, r4232, r4196, r4180, r4178, r4175, r4174, r4172, r4122. (that's less than 32 changesets - and a fair few are clearly not particularly interesting - or broken, so it should take no more than 5 attempts to bisect)
Some resident memory usage statistics (RES
in MB) as seen with top
on Fedora 19, time is measured in CPU time (as shown by top
- which is probably a fairer comparison than wall clock time, and easier to collect too).
Running two windows: xterm + glxgears, using x264
encoding and --no-mmap
and --no-speaker
switches.
Xpra Version | SHR | 0 | 2 min | 4 min | 6 min | 8 min | 10 min | 20 min | 30 min | 40 min | loss rate in MB/hour | h264 Kiloframes/hour |
---|---|---|---|---|---|---|---|---|---|---|---|---|
0.7.8 | 27 | 61 | 61 | 61 | 0 | 890 | ||||||
0.8.9 | 27 | 67 | 68 | 68 | 0 | 796 | ||||||
0.9.8 | 27 | 73 | 74 | 74 | 74 | 74 | 74 | 74 | 74 | 74 | 0 | 810 |
0.10.11 | 37 | 101 | 103 | 107 | 113 | 118 | 125 | 160 | 192 | 216 | 182 | 810 |
0.10.11 OpenGL | 58 | 127 | 130 | 141 | 144 | 145 | 147 | 162 | 181 | 202 | 112 | 740 |
trunk r5155 | 39 | 96 | 100 | 102 | 105 | 111 | 115 | 156 | 210 | 260 | 228 | 826 |
trunk r5155 OpenGL | 66 | 136 | 146 | 153 | 158 | 162 | 169 | 196 | 228 | 262 | 189 | 780 |
Will continue to update this table. An easy way to get those numbers is to leave this command running in a terminal with a long history (or send it to a log file):
while true; do sleep 5; top -c -n 1 | grep attach; done
The rate per hour is simply extrapolated from the largest sample collected. The frames per hour can be obtained the same way with:
xpra info| grep frames | egrep "h264|x264"
Still TODO:
Not directly related to this ticket:
As per comment:28, bisecting 0.9.x to 0.10.x:
--no-speaker
)
--no-speaker
)
--no-speaker
So the "bad" changeset is r4311, but this only fixes the auto-refresh code, so the leak must be in the auto-refresh code..
Running with --auto-refresh-delay=0
prevents the leak and confirms it.
Since the auto-refresh uses a different encoding, I then looked at the encodings used for the auto-refresh.
Using webp
as primary encoding does leak, and fast too! 4GB of memory in just over 1 minute with glxgears
!
So r5159 ensures we don't use webp unless requested (will backport it), and I've created #491 to fix this particular leak.
Now, for this ticket afarr:
(if there is still a leak with any encoding and sound restarts, let's put it in a separate ticket pointing back to this one)
Meant to leave this yesterday, but I guess it didn't save.
Anyways:
I'm running r5178 in Windows and watched a twitch.tv stream for a while and took down the memory usage from Task Manager (top
isn't available in Windows) every 5 minutes. Here's the relevant data:
Default everything | |
Time | Memory Usage (in Megabytes) |
Start | 114.9mb |
5 minutes | 119.9mb |
10 minutes | 121.7mb |
15 minutes | 122.3mb |
20 minutes | 123.9mb |
25 minutes | 122.9mb |
30 minutes | 125.6mb |
35 minutes | 125.7mb |
40 minutes | 122.3mb |
45 minutes | 122.3mb |
50 minutes | 122.4mb |
55 minutes | 122.6mb |
60 minutes | 123.0mb |
I restarted Xpra and went back to the same stream and reran it with --no-speaker as one of the arguments.
With --no-speaker | |
Time | Memory Usage (in Megabytes) |
Start | 116.7mb |
5 minutes | 119.2mb |
10 minutes | 119.3mb |
15 minutes | 119.2mb |
20 minutes | 120.0mb |
25 minutes | 120.1mb |
30 minutes | 120.3mb |
35 minutes | 120.0mb |
40 minutes | 119.6mb |
I am most definitely seeing a leak, with mplayer
, firefox
or even just an xterm
running top
.
It goes up regularly, but does go back down if I minimize the window. So freeing the avcodec
context is enough to free the memory.
Can you please try the latest beta build to see if the problem can be reproduced on your system with the same build I have used (latest beta): http://xpra.org/beta/windows/
I ran it through a couple different things to test the memory usage:
first test:
twitch.tv stream in Firefox:
Time | Memory Usage (in Megabytes) |
Start | 108.8 |
5 minutes | 145.7 |
10 minutes | 145.4 |
15 minutes | 146.5 |
20 minutes | 145.7 |
25 minutes | 144.1 |
30 minutes | 144.6 |
I hit the fullscreen button on the twitch viewer and it caused some pretty awful looking screen flickering. It was bad enough I had to un-fullscreen it. So then I ran the browser as fullscreen and left it like that for another 15 minutes:
Time | Memory Usage (in Megabytes) | |
Start | 143.2mb | Quick note: CPU usage was very similar |
5 minutes | 143.2mb | |
10 minutes | 143.2mb |
At around 11 minutes the server disconnected me so we powered it off to give it some more memory and hard drive space to load on a longer video file to play. After an hour or two we got it up and running again so I loaded up an episode of Top Gear I had sitting on my hard drive. The file is supposed to be 1 hour and 2 minutes but about 15 minutes into it mplayer exits saying it hit the end of the file. I'm thinking it didn't convert properly from .mkv. Interestingly Windows Media Player won't open the file, but VLC will on my laptop.
I digress, but here are the logs from watching the same file in non-fullscreen, native 1280x720 window size, and "fullscreen". Interestingly Xpra wouldn't maximize the mplayer window to fullscreen, I'll attach a screenshot to demonstrate.
1280 x 720 window size | |
Time | Memory Usage (in Megabytes) |
Start | 118.0mb |
5 minutes | 132.0mb |
10 minutes | 131.5mb |
"Fullscreen" - Somewhere around 1700 x 1080 | |
Time | Memory Usage (in Megabytes) |
Start | 141.9mb |
5 minutes | 143.2mb |
10 minutes | 143.3mb |
In both tests, the memory started to climb regularly, but after around 5 minutes it holds steady to within a megabyte or two more or less memory. I haven't been able to produce a memory leak.
This is a crop of the Xpra window not quite going fullscreen.
maxmylyn: there is nothing wrong with fullscreen in this case, this is how mplayer
works unless you add the -zoom
command line argument (when hardware scaling is not available IIRC). Even if there was, I don't see what this has to do with this bug. FYI: there is a ticket for multihead and fullscreen #496, or create a new one if this is a different issue. Is this in a virtual machine by any chance? Is the "twitch viewer" a flash player or html5?
The server we are running off of is a virtual machine, but the client is my laptop, which runs Windows 7 natively. And the "twitch viewer" is Flash currently but they're working on switching over to HTML5 for performance reasons.
With a windows 7 client, running r5319 client, running an idle xterm and a second xterm with top
my memory only climbs from 74 MB to 76 MB after half an hour.
Starting glxgears
from the idle xterm bumps the memory up to 85 MB, but after another half an hour it only rose to 86 MB (glxgears
ran steadily at 525-535 FPS, and seemed to be running at about 115% cpu, according to the top.
Interestingly, accidentally maximizing the xterm running top
, despite promptly restoring its size, did bump the memory up to 96 MB... though another 10 minutes of running left it still at 96 MB.
I get the impression it is not video related. I'll see about running something with sound and the window minimized and see what happens.
Tested with r5383 Win7 client. I left the KDFC (a local classical radio station) webplayer running in the background for an hour playing music. The memory only changed by 32 kilobytes from 107.024mb to 107.056.
link for the interested:
http://www.kdfc.com/pages/15744854.php?
Note: as of r6046, the leak detection code can also be used from the client, documented on wiki: Debugging Memory Leaks
I believe this particular memory leak may well be race in dec_avcodec2
(which would explain why it only happened under specific circumstances), will follow up in #465.
Closing. Feel free to re-open if you have more information.
FYI: in the future, we may move to statically linked dec_avcodec2
decoder builds for all Debian / Ubuntu, as this reduces the maintenance cost of having 2 versions of the "libav" based decoder.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/457