Improved timestamps on AV packets

[Xepol]

If the server send timestamps sound-data and draw packets that represent not just when the data comes out of the encoder, but also when the raw data goes into the encoder. this information could be used by the client to better track how far out of sync audio is compared to video by comparing when the data went into the respective encoders.

[Antoine Martin]

The "timestamps" we currently get from the gstreamer pipeline are already from the input element AFAICT (TBC), adding a "do-timestamp=True" to the "pulsesrc" doesn't change that. The problem is that they start from 0, with no way for us to convert them to absolute time.

Looks like we'll need to

• tell gstreamer to record and carry a unix time so that we can correlate the two?

I've tried:

sysclock = gst.SystemClock.obtain()
self.pipeline.use_clock(sysclock)

But that didn't make any difference.

• and obviously, also record the timestamp when we freeze the pixels and include this in the "client_options" picture metadata - this needs to be captured when the picture gets frozen (when we copy the pixels) and not when we start compressing it (which happens later and from a different thread)

Links:

• ​Clocks and synchronization in GStreamer
• ​GStreamer clocks and latency calculation

I think I'll need to ask the gstreamer devs. In any case, this is 2.0 material, let's remove gstreamer 0.10 first.

[Antoine Martin]

I've updated the doc links which had bit-rotted, and asked the ​gstreamer-devel mailing list for help since I don't see any easy way of doing this: ​absolute buffer timestamp from src element.

[Antoine Martin]

Judging by the lack of response, there is no easy way to do this.
We'll have to write a custom gstreamer audio filter element and expose the timestamp delta as a property we can query from our capture pipeline.

Pointers:

• ​gstidentity: Dummy element that passes incoming data through unmodified.
• ​audioinvert: Swaps upper and lower half of audio samples.
• ​Writing a Plugin
• ​gst-template: contains template code for possible GStreamer projects.

[Antoine Martin]

timestamp plugin

[Antoine Martin]

header file for timestamp plugin

[Antoine Martin]

use the timestamp plugin to get a real monotonic absolute timestamp we can also use for video frames

[Antoine Martin]

To use the new plugin attached:

gcc -I. -I.. \
    -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include \
    -pthread -Wall -g -O2 -Wall  -c gsttimestamp.c  -fPIC -DPIC -o gsttimestamp.o
gcc -shared  -fPIC -DPIC  gsttimestamp.o  \
    -lgstbase-1.0 -lgstcontroller-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 \
    -pthread -g -O2   -pthread -Wl,-soname -Wl,libgsttimestamp.so  -o libgsttimestamp.so
sudo cp libgsttimestamp.so /usr/lib64/gstreamer-1.0/

Then with the xpra patch above, we expose extra attributes in the audio packet metadata:

• monotonic timestamp as "ts"
• capture latency as "latency"

Still TODO:

• packaging of this plugin (RPM only for now)
• expose the same monotonic timestamp from the screen capture code path
• maybe add a capability for this so that clients that don't care about this feature won't incur the small extra cost

[Antoine Martin]

• r15985 adds support for these timestamps in the xpra audio and video paths, disabled by default for video unless the client specifies the capability "send-timestamps=True", which can be enabled with the env var: XPRA_SEND_TIMESTAMPS=1

The timestamp value is sent as part of the audio and video metadata under the key "ts". (the existing "timestamp" is still present for audio packets - but this is an absolute value which cannot be reconciled with the video packets metadata)

• r15986 uses this new data to calculate a better value for the audio capture pipeline latency, now exposed as:

  $ xpra info | grep speaker.latency
  client.sound.speaker.latency=76
  

Note: this is how long it takes from the moment the audio buffer is captured from the source element (typically "pulsesrc") until we receive it in a compressed form in our buffer capture element ("appsink"). It does not include the time it takes to capture from the sound card (usually very low anyway), or the time it takes to forward this buffer through our subprocess wrapper layer back to the server process (also quite low)

• r15987 adds packaging for this new gstreamer plugin (not done using the humongous autotools setup the code was originally based on: ​gst-template), but 2 simple gcc commands in the specfile)
• r15988 adds it as a soft "recommends" dependency to RPM builds

[Antoine Martin]

Beta builds with those changes are available here http://xpra.org/beta.

[Antoine Martin]

(edit milestone)

[Antoine Martin]

this is an important feature - please test

[Antoine Martin]

Not heard back, closing.