Xpra: Ticket #271: Ubuntu 12.10 x86_64 with v0.8.6 deb installed from repo crashes hard in x264lib.c

(gdb) bt
#0  0x00007ffff68f4425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff68f7b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff693239e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff693cb96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007ffff36991a2 in set_decoder_csc_format ()
   from /usr/lib/python2.7/dist-packages/xpra/x264/codec.so
#5  0x00007ffff3690004 in ?? () from /usr/lib/python2.7/dist-packages/xpra/x264/codec.so
#6  0x0000000000497ea4 in PyEval_EvalFrameEx ()

First suspect:

Tue, 26 Feb 2013 16:06:51 GMT - Antoine Martin: status changed

confirmed as being caused by r2661, this comes from #244 and us moving to newer libav version for some builds - the code changes that have been suggested are not backwards compatible by the looks of things :(

Tue, 26 Feb 2013 17:33:55 GMT - Antoine Martin: status changed; resolution set

A better stacktrace:

*** glibc detected *** /usr/bin/python: free(): invalid pointer: 0x00007f5f70002010 ***
======= Backtrace: =========

Fixed in r2821 + r2823 for trunk and r2824 for v0.8.x. The API docs do state (taken from the Ubuntu version which has the crash):

 * Allocate an AVFrame and set its fields to default values.  The resulting
 * struct can be deallocated by simply calling av_free().
 * @return An AVFrame filled with default values or NULL on failure.
 * @see avcodec_get_frame_defaults
AVFrame *avcodec_alloc_frame(void);

But having tried very hard to make it work, this does not seem to be the case. Despite the same recommendation to use av_free() in 0.8 and earlier also found in #244 - this just does not work.

Sat, 23 Jan 2021 04:50:09 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/271