Redim entries - optimising code / ArraySlice

I have a folder array that contains up to +1.000.000 entries.
Dim folderStack() as FolderItem

Now I need to remove X entries at the start and Y entries at the end.

Cutting the end isn’t difficult:
If I need to cut the last 5.000 and total entries is 10.000)
Redim folderStack(5000)

For removing entries at the start, is a bit more work. You need to copy the ones you want to keep to another array and then copy it back.
When there are a lot of entries, this is slow when you have a lot of entries.

Anyone has a better way doing this?

FWW with Swift you have ArraySlice which works incredible fast (even for more than 1.000.000 entries is <0.1s). I don’t think Xojo has a similar

Why not just shift the entries inside the same array then redim?

Does ArraySlice operate on the given array, or return a new one?

Instead, could you create a class which has a boolean flag?

  class myFolderItem
       f as folderItem
       ignore as boolean

Then just set folderStack(i).ignore = true and in your processing loop, ignore those.

This might be much faster than slicing the array.

A few more things:

  • in tight loops, it’s really important to set #pragma DisableBackgroundTasks
  • folderItem operations are much slower in macOS 10.14 in general - I think the next Xojo version may improve this.

Another thought: how about using Array.SortWith() https://documentation.xojo.com/api/language/sort.htmlWith and sort the array so the to-be-deleted items are now at the end of the array, and then use Redim() to truncate the array? This might be faster (it might also be much, much slower).

actually its not

for i=min(5000,array.ubound) downto 0
array.remove i
next i

It may not be fast, but its not a “lot of work” :slight_smile:

If you were looking to remove the first 5000 entries, the value would have to be 4999, but we all know that. :slight_smile:

Good idea, I bet this would be quite fast:

sub FastSliceArray(extends a() as folderItem, startIndex as integer, endIndex as integer) 
  #pragma DisableBackgroundTasks
  dim i as integer = 0
  for j as integer = startIndex to endIndex
    a(i) = a(j) ' shift item
    i = i + 1 ' remember index of last shifted item
  next
 redim a(i-1) ' truncate
end sub

Edit: typos etc.

I was thinking:

[code]Public Sub removeXItemsAt(extends thearray() as integer, itemcount as integer, index as Integer)
#Pragma DisableBoundsChecking

for i as integer=0 to thearray.Ubound-index-itemcount
thearray(i+index)=thearray(i+index+itemcount)
next

Redim thearray(thearray.Ubound-itemcount)
End Sub
[/code]

I’d add an error check in there to make sure endIndex >= startIndex. And perhaps use count instead of endIndex?

As an aside, it’s a shame that you can’t declare a general array parameter, because this would work with any type of array.

Sub MyArrayFunction (Extends arr() As Array) // if only...

[quote=453952:@Kem Tekinay]
As an aside, it’s a shame that you can’t declare a general array parameter, because this would work with any type of array.

Sub MyArrayFunction (Extends arr() As Array) // if only... [/quote]
huh … :slight_smile:

In Go, a slice is a reference type, so the “sub set” sliced is just a view of the real complete array, and that’s why it is so fast to grab a slice of an array.

Wow ! Great and some clever ideas!
Will do some tests for sure.

BTW I do like the boolean flag method. This way I can even undo things. :slight_smile:

Thank you all for the valuable input!

API2 comes to mind … :wink:

I think this is the same with Swift ArraySlice.

[quote=453966:@Rick Araujo]A “sliceable array” could be a Xojo class with .append() .index() etc and .slice(start, end) which returns a new slice with that subset.
[/quote]

Again API2 comes to mind. :wink:

A “sliceable array” could be a Xojo class with .append() .index() etc and .slice(start, end) which returns a new slice with that subset.

Great idea. Operator_Subscript would help here too.

Go for it Kem and others, I would like to play doing it, but have no time, sadly.

And generics would be gold in this mission. :frowning:

Rick’s idea is worth a feature request: <https://xojo.com/issue/57406>

:slight_smile:

<https://xojo.com/issue/56919>
<https://xojo.com/issue/14699>
https://www.great-white-software.com/blog/2019/09/04/generics/