Here is a hardcore question for the programming department :
I have an 3-dimensional array of integers with 3 decimal (double) values in each cell. So I would use this:
Var arr_dec1(5000,5000,5000) as double
Var arr_dec2(5000,5000,5000) as double
Var arr_dec3(5000,5000,5000) as double
But I do not know if it will be 5000 or much more, and only a fraction of the cells will be used/filled. So my question is now: Does it make sense to take another solution than an array? Pairs, a dictionary or a collection? With arrays I can access the content directly with
res = arr_dec1(a,b,c)
But how much space will that take? Is the array filled at the beginning when it is build - or will it âgrow upâ every time a new value is assigned? May I use 5000000 for each dimension? (This works!) At least this does not sound very smart to me. And how would I access any other storage model compared to this simple line above?
If all values are eventually filled, then youâll need the full space (1 TB). Best to use a binary file in that case, on a disk that has enough free space, and ideally on an SSD.
Then calculate the offset into the file like this:
dim offset as Int64 = (x * (5000 * 5000) + y * 5000 + z) * 8
8 is the size of a Double value.
Then read or write a DoubleValue using a BinaryStream, after setting the streamâs Position to the offset.
If itâs not all filled, the use a Dictionary, with the offset (without the * 8 factor) as the key, like Christian suggests - thatâll be the most efficient (besides, Collections are super slow, unless Xojo has improved them using a map now?)
No, the array will not be filled. And I tried to make one with 2.000.000 entries per dimension and it worked. Does that mean the array âgrows upâ when filled? If this is the case I will stay with the array. It is max. filled 5% ⊠does anyone know how the memory usage of an array is built? Does it use up half the memory if i would store something in value 2500?
If this may (or will) cause trouble I like Christians idea most. But is reading a dictionary as fast as an array-access? Speed will be an issue here. And will it work the same way on iOS (and Android)? And I have one problem left: How to access all parts of one dimension sequentially? If I would use a dictionary I can not imagine how this would work.
So I can forget thinking about collections. And what about pairs? What is that good for?
memory pages allocated, but never touched, donât use physical memory, only virtual.
So it may work to allocate 1 TB of memory, but only use a few bytes.
Check also IntegerToVariantHashMapMBS class, which would allow you to use integer as key directly and be faster than Xojoâs dictionary with variant as key.
Beside the space concern, if all values of the same index are tied together, Iâd possibly use an array of classes (the class would hold your 3 values). Youâd have a single array of your class and unused âindexesâ would simply not appear in the array.
From there, if you need a quick way to know whether a specific index has been filled, a dictionary (key=index, value=object of your class) would be an option (use Dict.HasKey).
Just my 2 cents.
So now look at this. First I use a routine to âcompressâ the index count:
Public Property aCriterias() as Pair
Public Function Arr_set_crt(ptr as integer) as Integer
' look for ID in the array and append one if not found
Var i As Integer
For i=0 To 5000
If i > aCriterias.LastRowIndex Then ' not found yet, add one
aCriterias.AddRow( i : ptr )
Return i
End If
If aCriterias(i).Right = ptr Then Return i ' found, leave ....
Next
End Function
This works and the array of indexes is filled correctly after it ran through.
Public Property aCells(5000,5000,5000) as Variant
Public Property aTuple(3) as Double
Do
' fill the array - but only with currently selected ID
o = Arr_set_opt(rs.Field("rating_option").IntegerValue)
c = Arr_set_crt(rs.Field("rating_criteria").IntegerValue)
u = Arr_set_usr(rs.Field("rating_user").IntegerValue)
' filling my element of the array
aTuple(0) = 0.5
aTuple(1) = Val(rs.Field("rating_rank").StringValue)
aTuple(2) = rs.Field("rating_value").DoubleValue
aTuple(3) = 0
' assign my content to the appropriate array cell
aCells(o,c,u) = aTuple ' and here it crashes completely
rs.MoveNext
Loop Until rs.EOF
This does not work and I do not understand why. It does not end in the debugger but it stops the complete application with âquit unexpetedlyâ and wants to report to Apple . Anyhow: The first o,c,u values are zero. I suppose it is not possible to assign an array as a Variant to a cell of another array? I tried this here and it worked:
Thanks Norman - I will do that. Very nice. But even if I have that dictionary in place it will not work. It prevents only a bloated array ⊠the problem seems to be in the assignment of an array as a Variant to another array.
So now I âshrinkedâ it down to the point where it comes to the crash. I am using this code here:
Dim aTuple(3) As Double
Dim aCells(600,600,600) As Variant
'
aTuple(0) = 0
aTuple(1) = 0.4
aTuple(2) = 0.9
aTuple(3) = 3
aCells(10,5,2) = aTuple
aTuple = aCells(10,5,2)
MsgBox(Str(aTuple(2)))
This works and uses 1.63 GB in RAM - so the array is bloated up at the moment I fill only one element. So no âpay per useâ in the array.
If I go higher with the dimensions it crashes without any warning or message. On my Mac 650 is already too much.
For my application this will mean, that max. 15% of the original array (dimensioned 5000 each) can be filled. I am not sure if this will be a lethal limitation at the end.
Finally Normans tip for using the core dictionary did the job. The array is condensed now as good as possible - I can proceed with my work now. Thanks to all.
It only SEEMED to work. But it did not. If I add a second tuple at another call of the array it does not get the first one back again:
Dim i,c,o,u As Integer
'
Dim aTuple(3) As Double
Dim aCells(600,600,600) As Variant
'
aTuple(0) = 0
aTuple(1) = 0.4
aTuple(2) = 0.9
aTuple(3) = 3
aCells(10,5,2) = aTuple
aTuple(0) = 1
aTuple(1) = 2
aTuple(2) = 3
aTuple(3) = 4
aCells(11,6,3) = aTuple
aTuple = aCells(10,5,2)
MsgBox(Str(aTuple(2)))
This should reference the entry in 10,5,2 and give back 0.9 - but it shows 3, the entry of the second tuple. Finally I regard this as an error in XOJO now, not able to handle Variants.
Dim c As New Cells
Dim t As New Tuple
'
t.Heat = 0
t.Rank = 0.4
t.Rating = 0.9
t.Result = 3
c.Cell(10,5,2) = t
t.Heat = 1
t.Rank = 2
t.Rating = 3
t.Result = 4
c.Cell(11,6,3) = t
t = c.Cell(10,5,2)
MsgBox(Str(t.Rating))
It still shows 3 as the result. What is wrong here? Did I misunderstand something obvious? Is this really a bug?