#1370 closed enhancement (worksforme)
Improved timestamps on AV packets
Reported by: | Xepol | Owned by: | Xepol |
---|---|---|---|
Priority: | critical | Milestone: | 2.1 |
Component: | sound | Version: | trunk |
Keywords: | Cc: |
Description
If the server send timestamps sound-data and draw packets that represent not just when the data comes out of the encoder, but also when the raw data goes into the encoder. this information could be used by the client to better track how far out of sync audio is compared to video by comparing when the data went into the respective encoders.
Attachments (3)
Change History (13)
comment:1 Changed 4 years ago by
Component: | server → sound |
---|---|
Milestone: | → 2.0 |
Priority: | major → minor |
Status: | new → assigned |
comment:2 Changed 4 years ago by
I've updated the doc links which had bit-rotted, and asked the gstreamer-devel mailing list for help since I don't see any easy way of doing this: absolute buffer timestamp from src element.
comment:3 Changed 4 years ago by
Milestone: | 2.0 → 2.2 |
---|---|
Status: | assigned → new |
Judging by the lack of response, there is no easy way to do this.
We'll have to write a custom gstreamer audio filter element and expose the timestamp delta as a property we can query from our capture pipeline.
Pointers:
- gstidentity: Dummy element that passes incoming data through unmodified.
- audioinvert: Swaps upper and lower half of audio samples.
- Writing a Plugin
- gst-template: contains template code for possible GStreamer projects.
Changed 4 years ago by
Attachment: | audio-use-timestamp-plugin.patch added |
---|
use the timestamp plugin to get a real monotonic absolute timestamp we can also use for video frames
comment:4 Changed 4 years ago by
Status: | new → assigned |
---|
To use the new plugin attached:
gcc -I. -I.. \ -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include \ -pthread -Wall -g -O2 -Wall -c gsttimestamp.c -fPIC -DPIC -o gsttimestamp.o gcc -shared -fPIC -DPIC gsttimestamp.o \ -lgstbase-1.0 -lgstcontroller-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 \ -pthread -g -O2 -pthread -Wl,-soname -Wl,libgsttimestamp.so -o libgsttimestamp.so sudo cp libgsttimestamp.so /usr/lib64/gstreamer-1.0/
Then with the xpra patch above, we expose extra attributes in the audio packet metadata:
- monotonic timestamp as "ts"
- capture latency as "latency"
Still TODO:
- packaging of this plugin (RPM only for now)
- expose the same monotonic timestamp from the screen capture code path
- maybe add a capability for this so that clients that don't care about this feature won't incur the small extra cost
comment:5 Changed 4 years ago by
- r15985 adds support for these timestamps in the xpra audio and video paths, disabled by default for video unless the client specifies the capability "send-timestamps=True", which can be enabled with the env var:
XPRA_SEND_TIMESTAMPS=1
The timestamp value is sent as part of the audio and video metadata under the key "ts". (the existing "timestamp" is still present for audio packets - but this is an absolute value which cannot be reconciled with the video packets metadata)
- r15986 uses this new data to calculate a better value for the audio capture pipeline latency, now exposed as:
$ xpra info | grep speaker.latency client.sound.speaker.latency=76
Note: this is how long it takes from the moment the audio buffer is captured from the source element (typically "pulsesrc") until we receive it in a compressed form in our buffer capture element ("appsink"). It does not include the time it takes to capture from the sound card (usually very low anyway), or the time it takes to forward this buffer through our subprocess wrapper layer back to the server process (also quite low)
- r15987 adds packaging for this new gstreamer plugin (not done using the humongous autotools setup the code was originally based on: gst-template), but 2 simple gcc commands in the specfile)
- r15988 adds it as a soft "recommends" dependency to RPM builds
comment:6 Changed 4 years ago by
Owner: | changed from Antoine Martin to Xepol |
---|---|
Status: | assigned → new |
Beta builds with those changes are available here http://xpra.org/beta.
comment:8 Changed 4 years ago by
Priority: | minor → critical |
---|
this is an important feature - please test
comment:9 Changed 3 years ago by
Resolution: | → worksforme |
---|---|
Status: | new → closed |
Not heard back, closing.
comment:10 Changed 4 days ago by
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1370
The "timestamps" we currently get from the gstreamer pipeline are already from the input element AFAICT (TBC), adding a "do-timestamp=True" to the "pulsesrc" doesn't change that. The problem is that they start from 0, with no way for us to convert them to absolute time.
Looks like we'll need to
I've tried:
But that didn't make any difference.
Links:
I think I'll need to ask the gstreamer devs. In any case, this is 2.0 material, let's remove gstreamer 0.10 first.