Image processing in a background-thread

Hi, I am trying to process an image in a low priority thread, so the processing can happen in the background and that the user interface performance is not affected. To be precise, I am generating a picture of audio waveforms using MBS AVFoundation and MBS Picture Plugin. But the image rendering raises a ThreadAccessingUIException. Even though the picture is not affecting the UI yet, but is simply a variable inside the thread.run. This is the code in the Thread.run:

dim error as NSErrorMBS
dim a as new AVAudioFileMBS(pFolderitem, error)

if error <> nil then
  MsgBox error.LocalizedDescription
end if

dim ff as AVAudioFormatMBS = a.fileFormat
dim duration as Double = a.Length/a.fileFormat.sampleRate
dim pf as AVAudioFormatMBS = a.processingFormat
Var Channel As Integer = 0

// new buffer to receive data
dim buf as new AVAudioPCMBufferMBS(a.processingFormat, a.Length)

// request data
if a.readIntoBuffer(buf, a.Length, error) then  

  // only one array will have data
  dim Waveform as picture
  dim memFloat as MemoryBlock = buf.floatChannelDataCopy(Channel)
  if memFloat <> nil then
    // RenderSamplesMBS in Picture plugin
    Waveform = RenderSamplesMBS(memFloat, memFloat.Size / 4, 2, pMW.TimeLine.Width*20,   pMW.TimeLine.Height*2, 0, waveformDarkBlue, waveformLightBlue, waveformDarkBlue, -32, true)
  end if
  
  dim MemInt16 as MemoryBlock = buf.int16ChannelDataCopy(Channel)
  if memInt16 <> nil then
    Waveform = RenderSamplesMBS(memInt16, memInt16.Size / 2, 1,   pMW.TimeLine.Width*20,pMW. TimeLine.Height*2, 1, waveformDarkBlue, waveformLightBlue,   waveformDarkBlue, 15, true)
  end if

end if

Is there a way to use a Thread to process the image? If so, how do you do it? Or are Xojo Threads for text processing only?

I just found out, that the Error was not generated by the picture rendering, but because I tried to access Timline.Width and .Height of the mainWindow… Delivering those values as variables to the thread solved the problem.

But the image processing is still blocking the UI interface. This is how I call the thread:

var m as new FolderItem(MoviePath, FolderItem.PathModes.URL)
WaveformThread.Priority = Thread.LowestPriority
WaveformThread.run(Self, m, Timeline.Width, Timeline.Height)
WaveformThread.pause
// do other stuff
WaveformThread.resume

All the “do other stuff” is blocked, until the image is processed. Any ideas, why this is happening?

Threads in Xojo are cooperative.
This only works well, if your thread can yield often.
So calculating big pictures with RenderSamplesMBS will not yield.

My I ask what “yield” means in this context?

Yielding is context switching so that the main thread can be executed. If you don’t yield the users see the beachball and will terminate any app.

Thanks Beatrix, I understand. Might the solution be to divide the AVAudioPCMBufferMBS into chunks and process it piece by piece? Would that yield? If so, I would be glad to get some help on that…

You can try to do chunks. Alternatives are to use a worker or to have Christian make a multi-threaded alternative.

if you split work into a for loop, it will yield.

I’d like to try that. But I don’t know, how I can split AVAudioPCMBufferMBS into chunks. I am guessing something lik this, but obviously that does not work as mid() is only for strings:

var bufArray() as AVAudioPCMBufferMBS(a.processingFormat, 2000000)
var chunkStart as Integer = 0
var BufChunk as integer = 0
for chunkStart to a.Length
  bufArray.ResizeTo(BufChunk)
  bufArray.Add(BufChunk, mid(buf, chunkStart, 2000000)
  chunkStart = chunkStart+2000000
  BufChunk = BufChunk+1
next

I am still at a loss here. How can I split the data into a for loop?

1 Like

After digging into MemoryBlock, I came up with this solution:

var chunkSize as Integer = 20000000
var chunkIndex as Integer 

if memFloat <> nil then
  for chunkIndex = 0 to memFloat.Size  
    if chunkSize >= memFloat.Size - chunkIndex then // for the last chunk, adjust the chunksize
      chunkSize = memFloat.Size - chunkIndex
    end
    // RenderSamplesMBS in Picture plugin
    pWaveformArray.Add( RenderSamplesMBS(memFloat.StringValue(chunkIndex, chunkSize), chunkSize / 4, 2, TimeLine.Width*20, TimeLine.Height*2, 0, waveformDarkBlue, waveformLightBlue, waveformDarkBlue, -32, true) )
    chunkIndex = chunkIndex + chunkSize
  next
end if

now I just have to find out, how to stitch together the Pictures in the pWaveformArray to the Timeline Canvas. Any suggestions are warmly welcomed.

Well, I claimed victory too early. The above code splitting the memoryBlock does only produce some correct images. pWaveformArray(0) is ok, but the next ones are painted full from the middle line up. No waveforms, just a block from beginning to end… :frowning:

It is not easy to help you.
So you got several equally sized MemoryBlocks in an array of MemoryBlocks there?

Now you can call in a loop RenderSamplesMBS for each and store the picture in a new array.

Later you can make one big picture and raw the portions into that or draw them in a canvas’ paint event, but probably scaled down.

To draw a picture smaller, you need to use DrawPicture in Graphics class with all parameters to define the input and output rectangle. See this scale function:

Function ProportionalScaled(extends pic as Picture, Width as Integer, Height as Integer) As Picture
  // Calculate scale factor
  
  dim faktor as Double = min( Height / Pic.Height, Width / Pic.Width)
  
  // Calculate new size
  dim w as Integer = Pic.Width * faktor
  dim h as Integer = Pic.Height * faktor
  
  // create new picture
  dim NewPic as new Picture(w,h,32)
  
  // draw picture in the new size
  NewPic.Graphics.DrawPicture Pic, 0, 0, w, h, 0, 0, Pic.Width, Pic.Height
  
  // return result
  Return NewPic
End Function

Well, I work with the same memFloat Variable from the sample code of the MBS Show Samples.xojo_binary_project. But instead of generating the image with the whole memFloat using RenderSamplesMBS(), I work through parts of it using MidB.
But only every other image is rendered properly.

I also adjusted the image size, so I guess I don’t need the scaling function above?:

pWaveformArray.Add( RenderSamplesMBS(memFloat.MidB(chunkIndex, chunkSize), chunkSize / 4, 1, chunkSize/2000 , TimeLine.Height*2, 0, waveformDarkBlue, waveformLightBlue, waveformDarkBlue, -32, true) )

Could it be you have memory blocks, which are half filled with zeros?
e.g. one of the calculations there produces too big memory blocks?

Can you check for zeros in debugger by inspecting memory blocks?

I got it working! The issue was counting up the chunkIndex. Faulty was:

chunkIndex = chunkIndex + chunkSize

working does:

chunkIndex = chunkIndex + chunkSize-1

Otherwise, the new Index is starting always 1 byte later like 20…01, 40…02, 60…03, and so on.

Now I am back to stitching the images together.

Could you please give me one more hint, on how to do that?

A For…Next loop automatically increments, that’s where the extra 1 comes from.

You are right, Tim. for “Step” Next works:

for chunkIndex = 0 to memFloat.Size-1 Step chunkSize
// do the stuff
next

Best would be to draw all pictures into a PixmapShape. So it can be scaled later on. But I cannot find out how that is done…

Forum for Xojo Programming Language and IDE. Copyright © 2021 Xojo, Inc.