nvenc error at high resolution if image stride is higher than input pitch

[Antoine Martin]

How is this possible?

2014-12-03 12:26:20,226 error processing damage data: right operand length must match slice length
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/server/source.py", line 1586, in encode_loop
    fn_and_args[0](*fn_and_args[1:])
  File "/usr/lib64/python2.7/site-packages/xpra/server/window_source.py", line 1123, in make_data_packet_cb
    packet = self.make_data_packet(damage_time, process_damage_time, wid, image, coding, sequence, options)
  File "/usr/lib64/python2.7/site-packages/xpra/server/window_source.py", line 1429, in make_data_packet
    ret = encoder(coding, image, options)
  File "/usr/lib64/python2.7/site-packages/xpra/server/window_video_source.py", line 1209, in video_encode
    ret = self._video_encoder.compress_image(csc_image, options)
  File "xpra/codecs/nvenc4/encoder.pyx", line 1649, in xpra.codecs.nvenc4.encoder.Encoder.compress_image (xpra/codecs/nvenc4/encoder.c:17182)
    return self.do_compress_image(image, options)
  File "xpra/codecs/nvenc4/encoder.pyx", line 1702, in xpra.codecs.nvenc4.encoder.Encoder.do_compress_image (xpra/codecs/nvenc4/encoder.c:18155)
    self.inputBuffer.data[i*stride:(i+1)*stride] = pixels[i*image_stride:(i+1)*image_stride+stride]
TypeError: right operand length must match slice length

The code in question:

        #FIXME: we should copy from pixels directly..
        #copy to input buffer:
        if image_stride<self.inputPitch:
            stride = image_stride
            assert len(pixels)<=input_size, "too many pixels (expected %s max, got %s) image: %sx%s stride=%s, input buffer: stride=%s, height=%s" % (input_size, len(pixels), w, h, stride, self.inputPitch, self.input_height)
            self.inputBuffer.data[:len(pixels)] = pixels
        else:
            #ouch, we need to copy the source pixels into the smaller buffer
            #before uploading to the device... this is probably costly!
            stride = self.inputPitch
            for i in range(h):
                self.inputBuffer.data[i*stride:(i+1)*stride] = pixels[i*image_stride:(i+1)*image_stride+stride]
        log("compress_image(..) host buffer populated with %s bytes (max %s)", len(pixels), input_size)

[Antoine Martin]

Should be fixed in r8176, will backport just in case someone uses nvenc with v0.14

[Antoine Martin]

Backport for v0.14.x in r8179 - note: nvenc should still not be used in v0.14.x, but for as long as it is in the tree, we might as well not have it completely broken!

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/753