Synchronous encoding

[CzinkeM]

Hi!

First of all thanks for @ThibaultBee for this awesome library!

Also I would like to start a discussion about the topic of encoding.

I have been tinkering with this repository for a while now but I still have not got all the knowledge about every little part of the code. I already modified the library to work with ARCore but I want to gain more control over the encoding and I want to switch from async encoding to synchronous encoding, where I feed each frame into the encoder instead of the input surface.

I would greatly appreciate any advice or help on the topic. 🙏

[ThibaultBee]

Hi,

I am glad that you appreciate the library :) and sorry, I know that internal stuff are not well documented.

What kind of control are you looking at? Only the surface/callback mode?

So, I made something like this a long time ago, not sure it still works!
The VideoMediaCodecEncoder takes a useSurfaceMode parameters (see https://github.com/ThibaultBee/StreamPack/blob/2c03ec3d28737d59f6f1c65ae770f20c9b5303f6/core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/VideoMediaCodecEncoder.kt#L44C3-L44C3)
Hopefully, if useSurfaceMode is true, the video encoder operates in Surface mode, otherwise it operates in callback mode (so what you are looking for).

In the BaseStreamer, the useSurfaceMode comes from the VideoSource.hasSurface here: https://github.com/ThibaultBee/StreamPack/blob/2c03ec3d28737d59f6f1c65ae770f20c9b5303f6/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseStreamer.kt#L165C20-L165C20

I guess, you don't even have to fork the repository.
You can create a CameraARCoreSource that contains the CameraSource and your ARCore implementation with hasSurface = false.
Then, you create your own Streamer that takes CameraARCoreSource as an input (and muxer and endpoint according to your usage).

I don't know anything about ARCore. It could fit as an extension of this library. Could you show me what you did achieve when it is done?

[CzinkeM]

Thanks for this great advice I will try it and I get back to you. 🙏

[reis756]

Hi @ThibaultBee.

First I want to thank you for this amazing library. You are doing an excellent job.

I need a similar feature. I need to stream images from DJI drone using their SDK. The SDK provides images from the drone to the controller via a surfaceview, but it doesn't work if I just start the preview and start streaming.

I was thinking along the same lines as @CzinkeM, capturing each image frame instead of using preview. But I couldn't identify where I could do that.

[ThibaultBee]

If you want to pass image to the encoder, you should create a class that extends BaseStreamer (just like CameraBaseStream).
I guess you don't need any audio so you can set audioSource to null.
The muxer and the endpoint depends if you want to stream SRT or RTMP or write to a file. (for SRT it is TSMuxer + SrtProducer see

StreamPack/extensions/srt/src/main/java/io/github/thibaultbee/streampack/ext/srt/streamers/CameraSrtLiveStreamer.kt

Line 57 in 2c03ec3

muxer = TSMuxer().apply { addService(tsServiceInfo) },

)
Now you will have to implement your own video source where you call DJI API. This source should implement the IVideoSource interface andhasSurface = false and hasFrames = true and implements getFrame (

StreamPack/core/src/main/java/io/github/thibaultbee/streampack/internal/sources/IFrameSource.kt

Line 24 in 2c03ec3

interface IFrameSource<T> : Streamable, Configurable<T>, Releaseable {

). getFrame must return the RAW video buffer in a Frame.
The encoder will call getFrame and encode the buffer returns in the Frame.
Unfortunately, there isn't such example for video in StreamPack but it kind of look like an audio source.

If I had a DJI drone I could definitely implement that :)

[reis756]

Returning to thank you for your help. I found a way to use the Surface to transmit the images provided by the drone using the SDK and linking the Surface from the encoder. Thanks!

[ThibaultBee]

Nice, even better that what I explained.
Could you share your code so I can have a look. It could bring new ideas 👍

[reis756]

Nothing special. I followed your explanation and created a DroneSource class similar to CameraSource, but with the attribute hasSourface = true and, in startStream, I passed encoderSurface in the parameter of this DJI SDK function. This function feeds a surface with the stream from the drone's camera. I'm still doing some tests. The image orientation is still incorrect but it's transmitting perfectly. You can count on me for any help as this library has helped me a lot. I'm at your disposal.

cameraStreamManager.putCameraStreamSurface(
    ComponentIndexType.LEFT_OR_MAIN,
    encoderSurface!!,
    1280,
    720,
    ICameraStreamManager.ScaleType.CENTER_INSIDE
)

[ThibaultBee]

If DJI SDK provides a Surface that is even better :)

I don't really understand your code putCameraStreamSurface. Is that in your DroneSource?

Thanks your proposal about helping. This is kind of new for me and I would be super happy to have little help :) Could you contact me on Linkedin so we discuss about what you expect, what's your interest,...

[reis756]

The cameraStreamManager is just an instance of the drone's camera provider. So, putCameraStreamSurface provides the drone's video stream on a surface. I took the BaseStream's encodesurface and passed it as a parameter of this function that returns the video stream. Similar to what camera2 does on a given surface. I used the entire code structure of their API, just changing this image provider.

I found you on Linkedin. The project I'm working on needs to transmit data between devices and server via SRT. We'll probably need help with that. I'll talk to the CTO to get in touch with you if necessary.