I’m wanting to take an MJPEG stream from an HTML connection and paint it to a Canvas. Wait, you ask, why not just use an HTMLViewer? Well, I could but for what I am doing, a canvas is much more efficient. I have instances where I would be displaying multiple streams at once along with other content and there’s just stuff you can do in canvases that you can’t with an HTMLViewer. Plus, I have other instances where I want to use the same windows/controls but not with an MJPEG stream over an HTML connection.
Wait - can’t you use the “DrawInto” method to draw the contents of the HTMLViewer into a Canvas? Nope. The Docs say, “Although the DesktopHTMLViewer has this method, it’s not supported.” And additionally, even if I could, an HTMLViewer is a subclass of Control and so I can’t instantiate it in code without having it on the window. This makes it a real pain. So two reasons not to…
I would really like to simply use a URLConnection to get the stream - I can but it’s processing the data that is giving me problems. I can get it to work a little bit but not smoothly and then things lock up.
I would approach this from the standpoint of “how do I convert this MJPEG stream into a movie that can be played via the MoviePlayer”?
If you’re looking for smooth, full-motion video - I’d stay away from the Canvas control. You’re unlikely to get the kind of control and quality you would get from the full-fledged MoviePlayer control.
There are a number of tools that can do this – VLC, FFmpeg – so start shopping around and see which of them best fits your needs!
That creates the same issue as the HTMLViewer. The stream works great in the HTMLViewer. It’s all the other drawing and stuff that I can’t do. In the canvas I can draw multiple images in the canvas plus some graphic overlays, etc. None of that can be done with the HTMLViewer or MoviePlayer, etc.
Hmm. In that case - look into using FFmpeg to take the MJPEG stream and break it down into individual frames, then load the frames into memory as pictures and paint them to the Canvas like a flipbook, adding whatever you need on top.
It’s hard to say without further details whether this will be performant enough for you. If the performance is poor, you may have to take the extracted frames, draw on top of them, and then reassemble them into a playable movie format and present that; if that will serve your needs, that is.
basically you collect all incoming data and cut each image part.
you convert data into picture with =Picture.FromData(data)
sub class a canvas, set the new image into an property and do a .Refresh where you paint the new image/picture.
That’s kinda what I had been doing but it would hang and stop working. My device has a way to grab still images instead and I did get that working. Today I started playing with preemptive threads and that makes a HUGE difference in speed and smoothness of the image processing. The biggest culprit in time is the Picture.FromData or using the similar method call from MonkeyBread. That operation itself takes about 10milliseconds. Everything else in my code is a fraction of that. But putting that operation into preemptive threads seems to really work well. Gotta play with it some more and maybe with preemptive threads I can make processing the images in the MJPEG stream more effective…
Using preemptive threading, I have had some measured success with extracting images from a MJPEG stream and painting to a canvas. It works very well with painting a single stream. As I add multiple streams though, performance starts to suffer. Once I hit 4 streams, it becomes unusable. And I am stumped on this.
Each URL connection is in its own object. All image processing is done in preemptive threads. Nothing is using shared/common code from the threads.
I am wondering if there’s some limitation with Xojo’s URLConnection when handling streams like this. I can grab individual JPEGs with the URLConnection and get very good performance regardless of how many devices I am connecting to to get images. It’s the stream where I have this slowdown problem. They use different events in the URLConnection. Grabbing still images uses the ContentReceived event while grabbing the stream uses the ReceivingProgressed event. I am wondering if there’s something about the ReceivingProgressed event that gets bogged down.
I don’t know why I didn’t think to suggest this before, but you CAN simply plop a Canvas on top of a HTMLViewer and draw whatever you want. I put together a very quick demo and attached it - the Canvas drawing is buttery smooth and didn’t seem to consume a large amount of CPU. You could play your MJPEG stream in the HTML viewer and draw whatever you like on top. You can also capture mouse events.
I have other places where I need to be able to draw a single image across a canvas or multiple (ie: a video wall control). So it’s not possible to do what I want to do with HTMLViewers.
If only the HTMLViewer could be instantiated in code and use the DrawInto method. That would solve everything but it supports neither of those…
And I appreciate your help. You are 100% correct in that you could put a canvas over an HTMLViewer and draw all kinds of stuff in there. It’s just I need more than that…
How about you create a container control that you could reuse? It could have an HTMLViewer to view MJPEG streams; a Canvas to show the static images; and a Canvas over the top of both of them to do any overlay drawing. You could hide the HTMLViewer or the static image canvas if they aren’t being used.
I thought about that. But how would I draw the grid I show above. I need multiple HTMLViewers side by side. However, there’s something that I didn’t tell you about the above. If I click one of those images, it gives a full screen of that image. There’s also an alternate layout a customer wanted as shown below:
Right now all of this is done with a single canvas. I can resize everything easily, etc.
Wish I knew why the URLConnection seems to bog down with the stream. It’s literally pulling a fraction of the data over the network that grabbing JPEGs requires. The JPEGs are full resolution while the MJPEG stream is reduced. But the URLConnection seems to choke.
Yes, you would create and position multiple instances of my proposed Containers - each of them would have its own HTMLViewer (or the static image Canvas) as well as the overlay Canvas.
When the user clicks on one of the Containers to enlarge it, as in your example, you either switch to a different layout (i.e., another set of Containers) or you rearrange the ones you already have.
This is all looking more and more doable. I don’t see any reason this couldn’t work.
