Avfoundations. Caption subtitles text.

Hi,

As anyone gotten the MBS, Avfoundation plugin to work to add captions, subtitles or text to a video, the export it.
I know the Avfoundatns supports a parent CALayer object and a CALayer automation object, That object can be used to add text/subtitles/captions. Not sure how to do tat with the MBS plugin.

Thanks Eric

I’ve done some work with AVFoundation (not via MBS though), I can render subtitles directly on the video stream, but not as a specific subtitle stream (that’s beyond my knowledge).

AVFoundation can use Core Image to process the video frames. Use a videoOutPutStream and convert the CMSampleBuffer into a CIImage. Once you have a CIImage, you can use a Composition filter to apply a CIImage on top, which you can generate by creating a CIImage from a CGImage from a Xojo picture object.

So you can use Xojo to create a picture placing the text exactly where you want it.

Well, there is a AVMediaTypeSubtitle constant.

see
https://stackoverflow.com/questions/11525342/subtitles-for-avplayer-mpmovieplayercontroller

I think you can create a track with text and add it.

How?

I am not 100% sure, but the link above shows someone adds a text track.

@Sam Rowlands - while that’s a little above my head, I do understand some of that. Thanks for the info.

@Markus Winter - I will look at that link from @Christian Schmitz . A text track is better because a user can turn off Subtitles and Captions on their player. I think text tracks are embedded without the option to turn them off. Let’s see what I find out.

Thanks All

AVFoundation has a steep learning curve, I’ve only been using it for a year now and there’s still a lot to learn. If I was to include to the code to do what I suggest, it would take pages.

There are some convenience functions with AVFoundation, but these don’t work with Xojo as it uses blocks on different threads, and then there’s always some reason as to why the convenience functions are too limited anyway, and you end up writing your own transcoder.

If you can just add a text track, then you can use a AVMutableComposition and copy the video/audio from your movie into a composition and then in theory you can copy in a text track. Or look into the new AVMovie object as I think that does the same, but without having to read-in video/audio and write out again.

Well, I do have almost all stuff from AVFoundation in the MBS AVFoundation Plugin including all those blocks stuff.
If you miss something, please let me know.

@Sam Rowlands - I think you are correct. The MBS plugin gives access to the parts, but not the whole framework. From what I’ve read, I’d need to create an AVFoudationAnimation frame object of some kind. Then create a AVFoundation Text track that is attacked to the frame object. Then I’d need to set the text of the track to the font, size and color. But the MBS plugin is not giving me access to the AVFoundation classes and their properties.

So let’s take a look at this option.
The MBS plugin examples have one where a picture is overlaid onto a movie. The movie is read in, and output with the picture centered over the video.

So what I think will work would be to convert some text into a picture using a canvas’ paint method to drawstring. Save these text pictures in an array. Each picture would have the text converted to a picture, And a second array with the time code for when each picture should appear on the movie. In theory, I should be able to write out the movie with the picture images overlayed at the right time.

… In Theory.

Let me see if I can understand you approach:
AVFoundation can use Core Image to process the video frames.
Use a videoOutPutStream and convert the CMSampleBuffer into a CIImage.
Once you have a CIImage, you can use a Composition filter to apply a CIImage on top, which you can generate by
creating a CIImage from a CGImage from a Xojo picture object.

Since the video has audio, I assume I would need to preserve the audio track and combine the composite video frames.

Eric

Well, I can check it again, but I think I have all of AVFoundation in the plugin.
Maybe not the lastest additions completely for 10.13 or 10.14, but at some SDK, it was complete.

@Christian Schmitz Do you have an example of setting a font for a text, caption or subtitle track. Because if that can be done, then I will see that the plugin as access to the properties of the classes for these track types.

After that, the movie could be saved and the added track will be exported with the movie.

Thanks Eric

I am trying and added new Class for text layer. We will see if it works.
If you find example for c or swift, we may be able to convert it.

I got an example working. It renders text on the images using the new CATextLayerMBS class.

if you need that, I can recompile and upload tomorrow some new plugins with the example.

Just to be clear we’re talking about adding a text TRACK to the Avfoundation movie. That said, I am interested in seeing what you come up with.

Thanks Eric

I still have to find a sample in C which does add a text track. Sorry. Did you find one?

There are some examples in Swift.