Display audio samples from a Video file

General
1

I want to display an audio waveform I generate from an audio track in a video file.
Right now I use the VLC library to playback the video. But I have a hard time getting the audio.

The reason why I use the VLC libraries (via the VLCplugin by MBS) is that I want to be flexible with the various video formats out there. Personally I use quicktime files a lot. But I want to be able to use MKV files also. MKV is not compatible with the Movieplayer control. Besides, the Movieplayer control doesn’t have enough properties I would like to make use of.

2

Extract the PCM data and then plot the values in a Canvas.
I use ffmpeg for this but I assume the VLC Plugin can extract PCM data.

Here are some examples using ffmpeg and gnuplot.
https://trac.ffmpeg.org/wiki/Waveform#UsingGnuplot

3

After a letting the audio samples to rest for a while I dove back in.
I can’t really figure out how to extract PCM data using the VLCplugin. I think it is not (yet) possible.

One way is to use ffmpeg to do the job, to getting the samples.
But I can’t really get ffmpeg to work the way I want. Have you worked with ffmpeg? Do you have a simple snippet that might help?

sh.Arguments = "-i " + f.ShellPath + " -ac 1 -filter:a aresample=8000 -map 0:a -c:a pcm_s16le -f data -" sh.Execute "ffmpeg" dim result as string = sh.Result
f is a folderitem that points to an video file, quicktime with audio and video in this case
The result that is returned says something like “no such file / directory”

4

I find it easier to play on the command line first until I have an ffmpeg command that works.
I use something like this to extract pcm data:

…/whatever/ffmpeg -i video.mp4 -ac 1 -filter:a “aresample=1000” -map 0:a -c:a pcm_s16le -f data - > output.bin

With this, you end up with a binary file ‘output.bin’ that has values (from -32767 to 32767) every 1 millisecond that you can read and use to plot.

Then you translate this into a Shell command. If that works, you’ll remove > output.bin and read it straight in.

You can also let ffmpeg create a waveform for you.
https://dl.dropboxusercontent.com/u/609785/Xojo/Forum/ffmpeg_png.xojo_binary_project
…but I think you’re more flexible if you handle the PCM data yourself so you can normalize the volume etc.

Also, since these are videos and things can take a while, I suggest using async shells.

5

Totally… actually, what I do to test my commands is making some kind of command line tool within Xojo.
The thing is, when I work in OS-X’s terminal, I actually get data. My plan was to tweak the code in xojo to select a file, and fiddle with it, while learning and playing with the restults…

But I have a hard time getting that translated to a shell class. I tried several things.
This works:

myShell.execute "ls"

But this doesn’t:

myShell.execute "ffmpeg -i video.mp4 -ac..... (etc)

I have no idea where to put the arguments. Even placing them in the myShell.arguments property won’t help.

Exactly. I have the Shell subclassed on a window, to play with the DataAvailable event. Eventually I might place everything in a Thread.

6

[quote=275679:@Edwin van den Akker]But this doesn’t:

myShell.execute "ffmpeg -i video.mp4 -ac… (etc)[/quote]

find the full path with ‘which ffmpeg’ in terminal and use this path

for example
myShell.execute "/usr/bin/ffmpeg -i video.mp4 -ac… (etc)

7

Ah, that explains a lot. But why is it working in Terminal, and not in Shell?
I thought it was basically the same thing.

By the way, I just noticed this in the Xojo Reference:

Shell.Execute ( Command as String [,Parameters as String] )

The command, is that supposed to be the “/user/bin/ffmpeg”?
And the parameters the rest of the command line?

Doesn’t seem to work :confused:

8

Yes it does. See the Shell.Execute in the example project I posted above.

9

And of course, using Shell, there is no way of monitoring the progress. Or is there?

Just “thinking out loud”…
What if I can find a way of getting the number of samples, multiply it by 2, and multiply that result by the number of channels…
The final result should be the length in bytes, right? Or close enough…
Then monitor the folderitem that represents the output.bin.

10

this is what I use in Linux to show waveform in Canvas
ffmpegpath is a string property

Function Mp3ToWavPic(MP3File As FolderItem) As Picture
  dim x as string=str(wfCanvas.Width)
  dim y as string=str(wfCanvas.Height)
  
  If MP3File <> Nil And MP3File.Exists Then
    dim HelperApp As String = ffmpegPath
    // Set Command Options
    Dim cmdOptions() As String
    cmdOptions = Array(HelperApp, "-i", MP3File.ShellPath, _
    "-filter_complex", "showwavespic=split_channels=1:s=" _
    + X +"x" + Y , "-vcodec png", "-v 0", "-f image2", "-")
    
    // Start a Shell
    Dim shShell As Shell
    shShell = New Shell
    
    shShell.Execute(join(cmdOptions, " "))
    
    // Set result png
    If shShell.ErrorCode = 0 And shShell.Result.Len > 0 Then
      Return Picture.FromData(shShell.Result)
    else
      MsgBox shShell.ReadAll
    End If
  End If
End Function
11

Didn’t work here. I even double checked the ffmpeg path. But maybe I had the parameters wrong.

This actually worked. But I would love to work with raw PCM data, so I can manipulate it, like Normalizing Marco mentioned.
Tomorrow I will continue my quest. It is very AM here. So I better hit the sack.