Xpra: Ticket #753: nvenc error at high resolution if image stride is higher than input pitch

How is this possible?

2014-12-03 12:26:20,226 error processing damage data: right operand length must match slice length
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/server/source.py", line 1586, in encode_loop
    fn_and_args[0](*fn_and_args[1:])
  File "/usr/lib64/python2.7/site-packages/xpra/server/window_source.py", line 1123, in make_data_packet_cb
    packet = self.make_data_packet(damage_time, process_damage_time, wid, image, coding, sequence, options)
  File "/usr/lib64/python2.7/site-packages/xpra/server/window_source.py", line 1429, in make_data_packet
    ret = encoder(coding, image, options)
  File "/usr/lib64/python2.7/site-packages/xpra/server/window_video_source.py", line 1209, in video_encode
    ret = self._video_encoder.compress_image(csc_image, options)
  File "xpra/codecs/nvenc4/encoder.pyx", line 1649, in xpra.codecs.nvenc4.encoder.Encoder.compress_image (xpra/codecs/nvenc4/encoder.c:17182)
    return self.do_compress_image(image, options)
  File "xpra/codecs/nvenc4/encoder.pyx", line 1702, in xpra.codecs.nvenc4.encoder.Encoder.do_compress_image (xpra/codecs/nvenc4/encoder.c:18155)
    self.inputBuffer.data[i*stride:(i+1)*stride] = pixels[i*image_stride:(i+1)*image_stride+stride]
TypeError: right operand length must match slice length

The code in question:

        #FIXME: we should copy from pixels directly..
        #copy to input buffer:
        if image_stride<self.inputPitch:
            stride = image_stride
            assert len(pixels)<=input_size, "too many pixels (expected %s max, got %s) image: %sx%s stride=%s, input buffer: stride=%s, height=%s" % (input_size, len(pixels), w, h, stride, self.inputPitch, self.input_height)
            self.inputBuffer.data[:len(pixels)] = pixels
        else:
            #ouch, we need to copy the source pixels into the smaller buffer
            #before uploading to the device... this is probably costly!
            stride = self.inputPitch
            for i in range(h):
                self.inputBuffer.data[i*stride:(i+1)*stride] = pixels[i*image_stride:(i+1)*image_stride+stride]
        log("compress_image(..) host buffer populated with %s bytes (max %s)", len(pixels), input_size)


Wed, 03 Dec 2014 20:44:42 GMT - Antoine Martin: status changed

Should be fixed in r8176, will backport just in case someone uses nvenc with v0.14


Wed, 03 Dec 2014 22:43:05 GMT - Antoine Martin: status changed; resolution set

Backport for v0.14.x in r8179 - note: nvenc should still not be used in v0.14.x, but for as long as it is in the tree, we might as well not have it completely broken!


Sat, 23 Jan 2021 05:04:58 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/753