xpra icon
Bug tracker and wiki

Opened 12 months ago

Closed 2 months ago

#1370 closed enhancement (worksforme)

Improved timestamps on AV packets

Reported by: Xepol Owned by: Xepol
Priority: critical Milestone: 2.1
Component: sound Version: trunk
Keywords: Cc:

Description

If the server send timestamps sound-data and draw packets that represent not just when the data comes out of the encoder, but also when the raw data goes into the encoder. this information could be used by the client to better track how far out of sync audio is compared to video by comparing when the data went into the respective encoders.

Attachments (3)

gsttimestamp.c (7.4 KB) - added by Antoine Martin 6 months ago.
timestamp plugin
gsttimestamp.h (3.2 KB) - added by Antoine Martin 6 months ago.
header file for timestamp plugin
audio-use-timestamp-plugin.patch (3.2 KB) - added by Antoine Martin 6 months ago.
use the timestamp plugin to get a real monotonic absolute timestamp we can also use for video frames

Download all attachments as: .zip

Change History (12)

comment:1 Changed 12 months ago by Antoine Martin

Component: serversound
Milestone: 2.0
Priority: majorminor
Status: newassigned

The "timestamps" we currently get from the gstreamer pipeline are already from the input element AFAICT (TBC), adding a "do-timestamp=True" to the "pulsesrc" doesn't change that. The problem is that they start from 0, with no way for us to convert them to absolute time.

Looks like we'll need to

  • tell gstreamer to record and carry a unix time so that we can correlate the two?

I've tried:

sysclock = gst.SystemClock.obtain()
self.pipeline.use_clock(sysclock)

But that didn't make any difference.

  • and obviously, also record the timestamp when we freeze the pixels and include this in the "client_options" picture metadata - this needs to be captured when the picture gets frozen (when we copy the pixels) and not when we start compressing it (which happens later and from a different thread)

Links:

I think I'll need to ask the gstreamer devs. In any case, this is 2.0 material, let's remove gstreamer 0.10 first.

Last edited 9 months ago by Antoine Martin (previous) (diff)

comment:2 Changed 9 months ago by Antoine Martin

I've updated the doc links which had bit-rotted, and asked the gstreamer-devel mailing list for help since I don't see any easy way of doing this: absolute buffer timestamp from src element.

comment:3 Changed 9 months ago by Antoine Martin

Milestone: 2.02.2
Status: assignednew

Judging by the lack of response, there is no easy way to do this.
We'll have to write a custom gstreamer audio filter element and expose the timestamp delta as a property we can query from our capture pipeline.

Pointers:

Changed 6 months ago by Antoine Martin

Attachment: gsttimestamp.c added

timestamp plugin

Changed 6 months ago by Antoine Martin

Attachment: gsttimestamp.h added

header file for timestamp plugin

Changed 6 months ago by Antoine Martin

use the timestamp plugin to get a real monotonic absolute timestamp we can also use for video frames

comment:4 Changed 6 months ago by Antoine Martin

Status: newassigned

To use the new plugin attached:

gcc -I. -I.. \
    -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include \
    -pthread -Wall -g -O2 -Wall  -c gsttimestamp.c  -fPIC -DPIC -o gsttimestamp.o
gcc -shared  -fPIC -DPIC  gsttimestamp.o  \
    -lgstbase-1.0 -lgstcontroller-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 \
    -pthread -g -O2   -pthread -Wl,-soname -Wl,libgsttimestamp.so  -o libgsttimestamp.so
sudo cp libgsttimestamp.so /usr/lib64/gstreamer-1.0/

Then with the xpra patch above, we expose extra attributes in the audio packet metadata:

  • monotonic timestamp as "ts"
  • capture latency as "latency"

Still TODO:

  • packaging of this plugin (RPM only for now)
  • expose the same monotonic timestamp from the screen capture code path
  • maybe add a capability for this so that clients that don't care about this feature won't incur the small extra cost

comment:5 Changed 6 months ago by Antoine Martin

  • r15985 adds support for these timestamps in the xpra audio and video paths, disabled by default for video unless the client specifies the capability "send-timestamps=True", which can be enabled with the env var: XPRA_SEND_TIMESTAMPS=1

The timestamp value is sent as part of the audio and video metadata under the key "ts". (the existing "timestamp" is still present for audio packets - but this is an absolute value which cannot be reconciled with the video packets metadata)

  • r15986 uses this new data to calculate a better value for the audio capture pipeline latency, now exposed as:
    $ xpra info | grep speaker.latency
    client.sound.speaker.latency=76
    

Note: this is how long it takes from the moment the audio buffer is captured from the source element (typically "pulsesrc") until we receive it in a compressed form in our buffer capture element ("appsink"). It does not include the time it takes to capture from the sound card (usually very low anyway), or the time it takes to forward this buffer through our subprocess wrapper layer back to the server process (also quite low)

  • r15987 adds packaging for this new gstreamer plugin (not done using the humongous autotools setup the code was originally based on: gst-template), but 2 simple gcc commands in the specfile)
  • r15988 adds it as a soft "recommends" dependency to RPM builds
Last edited 6 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 6 months ago by Antoine Martin

Owner: changed from Antoine Martin to Xepol
Status: assignednew

Beta builds with those changes are available here http://xpra.org/beta.

comment:7 Changed 4 months ago by Antoine Martin

Milestone: 2.22.1

(edit milestone)

comment:8 Changed 4 months ago by Antoine Martin

Priority: minorcritical

this is an important feature - please test

comment:9 Changed 2 months ago by Antoine Martin

Resolution: worksforme
Status: newclosed

Not heard back, closing.

Note: See TracTickets for help on using tickets.