Image processing in a background-thread

Well, calculate size of final picture.
Then a FOR loop to draw each part into the bigger picture at the right position, possibly scale down.

As a side note, if error is in fact not nil, this message box will bite you with a ThreadAccessingUI, too.

Still I am trying to get this done. But with no success so far. I am trying to understand how to work with the picture object. I am experimenting with the Picture.Graphics.DrawPicture() - method to draw the smaller images into one big picture.

to start with, there is a picture and a pictureArray():

Public Property Waveform as Picture
Public Property Waveforms() as Picture

The Waveforms() Array contains all the small chunks that have been created with the RenderSamplesMBS() method. Waveform should then contain all the smaller chunks combined to one big picture. Studying the Xojo-Documentation, I conclude that Graphics.DrawPicture(Image As Picture, x As Double, y As Double [,destWidth As Double ] [, destHeight As Double ] [, sourceX As Double ] [, sourceY As Double ] [, sourceWidth As Double ] [, sourceHeight As Double]) would be the method of choice. But how do you apply this? This loop does not work:

for i as Integer = 0 to Waveforms.Count-1
  Waveform.Graphics.DrawPicture(Waveforms(i), xPosCalc, 0) 
next

where xPosCalc would be the calculated X-Position of each smaller picture.

There is also a Picture.ImageAt(index as Integer) As Picture - method. But I cannot figure out, how this could be of any help in this caseā€¦ If someone could direct me to the right path here, I would appreciate a lot.

If you give them all the same xPosCalc, theyā€™ll all get drawn on top of each other. And what about the Y-Position?

You need to supply the correct x and y for each call within the loop.

I thought, my short explanation would be enough:

But here is what I mean in code-form:

var xPosCalc as Integer
for i as Integer = 0 to Waveforms.Count-1
  Waveform.Graphics.DrawPicture(Waveforms(i), xPosCalc, 0) 
  xPosCalc = xPosCalc + Waveforms(i).Width
next

Another way I am pursuing is to add the image together over MemoryBlocks. But also here I get stuck. For that I additionally to the variables mentioned above, I create a MemoryBlock:

Public Property WaveformMB as MemoryBlock

To add the images I try the following:

for i as Integer = 0 to Waveforms.Count-1
  var mb as MemoryBlock = Waveforms(i).ToData(Picture.Formats.PNG)
  // WaveformMB = WaveformMB + mb  -->> creates NILObjectException
  WaveformMB = WaveformMB.Operator_Add(mb)  // -->> creates NILObjectException
next

So, still not much furtherā€¦ Help?

Trying to append PNG data together will never work.

Your previous example using DrawPicture should work. Have you checked that Waveforms() and Waveform have been create correctly?

Thanks Keving. Yes, Waveforms() and Waveform work both fine, I can create pictures in them and display them in a canvas or imageWellā€¦

Waveform.Graphics.DrawPicture(Waveforms(i), xPosCalc, 0)
creates NILObjectException

Have I reached the limit of Xojoā€™s Graphics capabilities here?

There a lot here to process (for me). Some fundamentals are not clear.

  1. How are you creating the waveform tiles?

  2. The below should help you track down what is causing the NOE.

if waveForm = nil then
  break
end if

For i as integer = 0 to n
  if waveForms( i ) = nil then
    break
  else
    Waveform.Graphics.DrawPicture(Waveforms(i), xPosCalc, 0)
  end if
Next
  1. The maximum size for an image is about 32,000 pixels wide or 32,000 pixels high. If youā€™ve exceeded that, then Iā€™d recommend keeping your tiles in memory and only displaying them when needed.

  2. The following wonā€™t work, because it is converting the pixels into a format suitable for storing on a disk. What you want is the RAW pixels, and even then, you canā€™t really append chunks of image data like that. Most image formats read left to right, top to bottom (or bottom to top).

var mb as MemoryBlock = Waveforms(i).ToData(Picture.Formats.PNG)
  1. If you wanted to strip together several images without drawing them, Iā€™d recommend writing it in C code as a plugin. It can be really fast and you can then take advantage of Grand Central Dispatch, so the processing will spawn across all available CPU cores. Iā€™ve done this for our image editing applications.
1 Like

Thank you very much for the thorough answer, Sam! Iā€™ll try to dig into it:

Like this:

var chunkSize as Integer = 20000000
var chunkIndex as Integer 
var error as NSErrorMBS
var a as new AVAudioFileMBS(f, error)
var buf as new AVAudioPCMBufferMBS(a.processingFormat, a.Length)
var memFloat as MemoryBlock = buf.floatChannelDataCopy(Channel)

if memFloat <> nil then
  for chunkIndex = 0 to memFloat.Size Step chunkSize
    if chunkSize >= memFloat.Size - chunkIndex then // for the last chunk, adjust the chunksize
      chunkSize = memFloat.Size - chunkIndex
    end
    // RenderSamplesMBS in Picture plugin
    Waveforms.Add( RenderSamplesMBS(memFloat.StringValue(chunkIndex, chunkSize), chunkSize / 4, 2, TimeLine.Width*20, TimeLine.Height*2, 0, waveformDarkBlue, waveformLightBlue, waveformDarkBlue, -32, true) )
  next
end if

It breaks at this loop:

if WaveForm = nil then
  break
end if
1 Like

Okay, so the function ā€œRenderSamplesMBSā€ creates the separate tiles. I donā€™t know anything about this function, but I think that is okay.

This is telling you that the main image waveForm is NIL, the next step would be to double check the code that creates this image. Start by double checking that the waveForm image is being created, then walk it back. If it is not actually being created i.e. new picture( width, height) is failing, it is either because one of the dimensions is zero or it is too big.

A suggestion I would make would be to change the name of the large image and the array of tiles to something more descriptive. Your future self will thank you.

waveForm  = waveFormCompositeImage
waveForms = waveFormTileImages

Just an example, but they can be anything you want.

Ok, so it was not the limit of Xojoā€™s Graphics capabilities here, but the limit of my knowledge. It is not sufficient to just create the image before you can draw into it, as I expected. The picture needs to have a size (width and height) declared. Thanks again, @Sam_Rowlands for getting me to understand this.

I am running into similar issues, while trying to create waveform tiles.
I understand the tiling process, a little: creating tiles to avoid the picture size limits.

Generating the tiles would not be the biggest problem, in my case, I think.
But getting the right tiles back can be.

If I want to display the right tiles in a timeline, I would need to know the horizontal position of the tile.
Say, I want to display the waveforms on a timeline where only a certain range is visible.
My viewport (the visible area within a longer timeline) might be 0:01:00 - 0:02:30 (0h 1m to 2m 30s), I need to know which tiles are in this range.

My first approach was to generate only 30 second waveforms of the audio. But that is where I got stuck.
Another approach is what I found here, by splitting it into sample chunks. But than, I how do I know which chunk represents what time-range?