Big data elaboration

Hi team, I’m facing a data quantity problem. That’s a lot of data to process.
I started by doing a test … I have to calculate possible combinations given by the formula (ECA)/(FDB), where the letters represent arrays with the same number of numbers … I started by trying with 4 5 elements, and it takes a lot of time … now that I have to insert another 20, will it take me a day? Is there a faster way?


Dim ArrayIngranaggi() as integer=array(40,44,48,56,64,66,72)
Dim ArraiIngranaggiRuotaA() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaB() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaC() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaD() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaE() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaF() as integer=ArrayIngranaggi()


Dim A as integer
Dim B as integer
Dim C as integer
Dim D as integer
Dim E as integer
Dim F as integer
Dim VisualizzaStringa as string

ListBox1.ColumnCount=8

For A=0 To ArraiIngranaggiRuotaA.LastIndex
  For B=0 To ArraiIngranaggiRuotaB.LastIndex
    For C=0 To ArraiIngranaggiRuotaC.LastIndex
      For D=0 To ArraiIngranaggiRuotaD.LastIndex
        For E=0 To ArraiIngranaggiRuotaE.LastIndex
          For F=0 To ArraiIngranaggiRuotaF.LastIndex
            'listbox1.AddRow(ArraiIngranaggiRuotaA(A).ToText,ArraiIngranaggiRuotaB(B).ToText,ArraiIngranaggiRuotaC(C).ToText,ArraiIngranaggiRuotaD(D).ToText,ArraiIngranaggiRuotaE(E).ToText,ArraiIngranaggiRuotaF(F).ToText)
            
            Dim R as currency
            R=(ArraiIngranaggiRuotaE(E)*ArraiIngranaggiRuotaC(C)*ArraiIngranaggiRuotaA(A))/(ArraiIngranaggiRuotaF(F)*ArraiIngranaggiRuotaD(D)*ArraiIngranaggiRuotaB(B))
            VisualizzaStringa=ArraiIngranaggiRuotaE(E).ToText+"*" +ArraiIngranaggiRuotaC(C).ToText+"*"+ArraiIngranaggiRuotaA(A).ToText + " / " +ArraiIngranaggiRuotaF(F).ToText+"*"+ArraiIngranaggiRuotaD(D).ToText+"*"+ArraiIngranaggiRuotaB(B).ToText
            
            'if R=1.55 then
            'listbox1.AddRow(R.ToText)
            listbox1.AddRow(ArraiIngranaggiRuotaA(A).ToText,ArraiIngranaggiRuotaB(B).ToText,ArraiIngranaggiRuotaC(C).ToText,ArraiIngranaggiRuotaD(D).ToText,ArraiIngranaggiRuotaE(E).ToText,ArraiIngranaggiRuotaF(F).ToText,VisualizzaStringa,R.ToText)
            'end if
            
          Next
        Next
      Next
    Next
  Next
  
Next

You are up against factorials. Each number in your array increases the number of permutations substantially.
Numbers = Permutations
1 = 1
2 = 2
3 = 6
4 = 24
5 = 120
6 = 720
7 = 5040
8 = 40320
9 = 362560
10 = 3628800
20 = to big form me to calculate

A substantial amount of time is also just loading the lisbox. Instead of that I suggest writing each line to a file.

Also many of the sub-calculations are unnecessarily repeated.

I tried this with about 20 numbers and it completed in about about 16 minutes.

dim start as integer
start = System.Microseconds
var output as string
Dim ArrayIngranaggi() as integer=array(40,44,48,56,64,66,72, 14, 16, 18, 33, 55, 37, 14, 9, 20, 3, 42, 20, 21)
Dim ArraiIngranaggiRuotaA() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaB() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaC() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaD() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaE() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaF() as integer=ArrayIngranaggi()

Dim A as integer
Dim B as integer
Dim C as integer
Dim D as integer
Dim E as integer
Dim F as integer
Dim VisualizzaStringa as string

Var file As FolderItem = SpecialFolder.Desktop.Child("testOutPut.csv")
Var t As TextOutputStream = TextOutputStream.Create(file)

ListBox1.ColumnCount=8
var ac, ace, bd, bdf as integer
var last as integer = 10
For A=0 To ArraiIngranaggiRuotaA.LastIndex
  For B=0 To ArraiIngranaggiRuotaB.LastIndex
    For C=0 To ArraiIngranaggiRuotaC.LastIndex
      ac = ArraiIngranaggiRuotaA(a) * ArraiIngranaggiRuotaC(c)
      For D=0 To ArraiIngranaggiRuotaD.LastIndex
        bd = ArraiIngranaggiRuotaB(b) * ArraiIngranaggiRuotaD(d)
        For E=0 To ArraiIngranaggiRuotaE.LastIndex
          ace = ac * ArraiIngranaggiRuotaE(e)
          For F=0 To ArraiIngranaggiRuotaF.LastIndex
            bdf = bd * ArraiIngranaggiRuotaF(f)
            Dim R as currency
            'R=(ArraiIngranaggiRuotaE(E)*ArraiIngranaggiRuotaC(C)*ArraiIngranaggiRuotaA(A))/(ArraiIngranaggiRuotaF(F)*ArraiIngranaggiRuotaD(D)*ArraiIngranaggiRuotaB(B))
            r = ace / bdf
            VisualizzaStringa=ArraiIngranaggiRuotaE(E).ToText+"*" +ArraiIngranaggiRuotaC(C).ToText+"*"+ArraiIngranaggiRuotaA(A).ToText + " / " +ArraiIngranaggiRuotaF(F).ToText+"*"+ArraiIngranaggiRuotaD(D).ToText+"*"+ArraiIngranaggiRuotaB(B).ToText
            
            'if R=1.55 then
            'listbox1.AddRow(R.ToText)
            'listbox1.AddRow(ArraiIngranaggiRuotaA(A).ToText,ArraiIngranaggiRuotaB(B).ToText,ArraiIngranaggiRuotaC(C).ToText,ArraiIngranaggiRuotaD(D).ToText,ArraiIngranaggiRuotaE(E).ToText,ArraiIngranaggiRuotaF(F).ToText,VisualizzaStringa,R.ToText)
            'end if
            
            output = str(ArraiIngranaggiRuotaA(A))  + ", " + str(ArraiIngranaggiRuotaB(B))  + ", " + str(ArraiIngranaggiRuotaC(C))  + ", " + str(ArraiIngranaggiRuotaD(D))  + ", " + str(ArraiIngranaggiRuotaE(E))  + ", " + str(ArraiIngranaggiRuotaF(F))  + ", " + VisualizzaStringa + ", " + str(R)  
            t.writeline(output)
            
          Next
        Next
      Next
    Next
  Next
  
Next
t.close
MessageBox str((System.Microseconds - start) / 1000000) + " seconds"

Thanks, it’s actually much faster this way

Now I have to find a way not to repeat the array value in the other arrays unless, there are two similar values. Ex:
Array 10,20,30,40,50

10,10,10,10,10 NO
10,20,30,40,50 YES
20,10,30,40,50 YES
20,20,30,40,50 NO

Array 10,20,20,30,40

10,20,20,30,40 YES
10,20,30,40,40 NO
10,20,30,40,20 YES

They are permutations without repetitions

Do you need every permutation or just every combination?
(Note the distinction between permutation and combination.)

It appears that you are repeating calculations using the same data many times. The order of variables in the numerator appears to make no difference since you’re just multiplying them together. Likewise for the denominator. For example:
ECA/FDB = AEC/FDB = CAE/FDB etc.
All that changes is the order of the arguments. There are 6 possible permutations of the numerator, and 6 possible permutations of the denominator, giving a total of 36 permutations of what is essentially the same calculation with the same result. Is there some reason why you need 36 copies of what is essentially the same thing?

If a single copy is sufficient then it would be more efficient to change the loops so that you’re not performing the identical calculation over and over again.

As a simple example:

for i = 0 to arr1.LastIndex-2
  for j = i+1 to arr2.LastIndex-1
    for k = j+1 to arr3.LastIndex
      R = arr1(i)*arr2(j)*arr3(k)
    next
  next
next

Notice that the loops are set up such that the array indices never overlap. They will cover every combination exactly once (but not every permutation).

Indeed for what I need
ECA/FDB = AEC/FDB = CAE/FDB … so I don’t need to repeat, on the contrary, it would be better because it would save me time.
I would like to get all possible combinations with the formula ECA/FDB without repetitions. At the moment there are repetitions, but I would like to be able to eliminate them, because they waste my time and slow down the process.
I try to do this indexes thing, so as not to repeat.
In the meantime, thanks for the help.

''OUTPUT FORMULA:   R= (A*C*E)/(B*D*F)
dim start as integer
start = System.Microseconds
var output as string
Dim ArrayIngranaggi() as integer=array(40,44,48,56,64,66,72,100)
Dim ArraiIngranaggiRuotaA() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaB() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaC() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaD() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaE() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaF() as integer=ArrayIngranaggi()

Dim A as integer
Dim B as integer
Dim C as integer
Dim D as integer
Dim E as integer
Dim F as integer
Dim VisualizzaStringa as string

Var file As FolderItem = SpecialFolder.Desktop.Child("testOutPut.csv")
Var t As TextOutputStream = TextOutputStream.Create(file)

var ace,bdf as integer
Dim R as currency




For A=0 To ArraiIngranaggiRuotaA.LastIndex
  For B=0 To ArraiIngranaggiRuotaB.LastIndex
    For C=0 To ArraiIngranaggiRuotaC.LastIndex
      For D=0 To ArraiIngranaggiRuotaD.LastIndex
        For E=0 To ArraiIngranaggiRuotaE.LastIndex
          For F=0 To ArraiIngranaggiRuotaF.LastIndex
            
            ace = ArraiIngranaggiRuotaA(a) * ArraiIngranaggiRuotaC(c) * ArraiIngranaggiRuotaE(e)
            bdf = ArraiIngranaggiRuotaB(b) * ArraiIngranaggiRuotaD(d) * ArraiIngranaggiRuotaF(f)
            
            R = ace / bdf
            VisualizzaStringa="("+ArraiIngranaggiRuotaA(A).ToText+"x" +ArraiIngranaggiRuotaC(C).ToText+"x"+ArraiIngranaggiRuotaE(E).ToText + " / " +ArraiIngranaggiRuotaB(B).ToText+"x"+ArraiIngranaggiRuotaD(D).ToText+"x"+ArraiIngranaggiRuotaF(F).ToText+")"
            
            output = str(ArraiIngranaggiRuotaA(A))  + "**" + str(ArraiIngranaggiRuotaB(B))  + "**" + str(ArraiIngranaggiRuotaC(C))  + "**" + str(ArraiIngranaggiRuotaD(D))  + "**" + str(ArraiIngranaggiRuotaE(E))  + "**" + str(ArraiIngranaggiRuotaF(F)) + "**" +"-------"+ "**" + VisualizzaStringa + "**" +" ------------"+"**"+ str(R)  
            t.writeline(output)
            
          Next
        Next
      Next
    Next
  Next
  
Next
t.close
MessageBox str((System.Microseconds - start) / 1000000) + " seconds"

I tried to change the code a bit and move the indexes as suggested, but in doing so I get incomplete combinations as a result, where for example one or more values are missing. I rewrote the original listing to try and understand the problem.
As a result I get many combinations, of which few are the useful ones. I can’t repeat a number if I’ve already used it elsewhere… I also thought about using remove used element from array… help.
For example, in the picture, 40 cannot be in both the numerator and denominator, unless it is a repeated number in the array, but in our case there is only one number 40.

I don’t understand why you have six copies of the same array:

Dim ArrayIngranaggi() as integer=array(40,44,48,56,64,66,72,100)
Dim ArraiIngranaggiRuotaA() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaB() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaC() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaD() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaE() as integer=ArrayIngranaggi()
Dim ArraiIngranaggiRuotaF() as integer=ArrayIngranaggi()

Unless I overlooked something, it doesn’t appear that they are being changed, so I think you could just work with a single array.

Check out how I modified the indices from your original code here:


For A=0 To ArraiIngranaggiRuotaA.LastIndex-2
  For C=A+1 To ArraiIngranaggiRuotaB.LastIndex-1
    For E=C+1 To ArraiIngranaggiRuotaC.LastIndex
      For B=0 To ArraiIngranaggiRuotaD.LastIndex-2
        For D=B+1 To ArraiIngranaggiRuotaE.LastIndex-1
          For F=D+1 To ArraiIngranaggiRuotaF.LastIndex
            
            ace = ArraiIngranaggiRuotaA(a) * ArraiIngranaggiRuotaC(c) * ArraiIngranaggiRuotaE(e)
            bdf = ArraiIngranaggiRuotaB(b) * ArraiIngranaggiRuotaD(d) * ArraiIngranaggiRuotaF(f)
            
            R = ace / bdf
            VisualizzaStringa="("+ArraiIngranaggiRuotaA(A).ToText+"x" +ArraiIngranaggiRuotaC(C).ToText+"x"+ArraiIngranaggiRuotaE(E).ToText + " / " +ArraiIngranaggiRuotaB(B).ToText+"x"+ArraiIngranaggiRuotaD(D).ToText+"x"+ArraiIngranaggiRuotaF(F).ToText+")"
            
            output = str(ArraiIngranaggiRuotaA(A))  + "**" + str(ArraiIngranaggiRuotaB(B))  + "**" + str(ArraiIngranaggiRuotaC(C))  + "**" + str(ArraiIngranaggiRuotaD(D))  + "**" + str(ArraiIngranaggiRuotaE(E))  + "**" + str(ArraiIngranaggiRuotaF(F)) + "**" +"-------"+ "**" + VisualizzaStringa + "**" +" ------------"+"**"+ str(R)  
            t.writeline(output)
            
          Next
        Next
      Next
    Next
  Next
  
Next
t.close
MessageBox str((System.Microseconds - start) / 1000000) + " seconds"

Done in a hurry. I hope I didn’t miss anything.

So two things…

First of all, arrays are objects, and sine you’re creating all six arrays from a single source, they all point to the exact same array. If you make any changes, they’ll all change.

Second, I see no indication of how you’re compiling your app. Mathematical equations are an area where the Aggressive optimization really shines. I suggest you try it because I’ve seen huge speed improvements. Unfortunately you’ll need to Build your app for this to work as debugging always uses the equivalent of “Default”.

I’m also going to suggest that instead of using a TextIOutputStream.Writeline on every loop that you cache them into an array and either write them periodically through the use of mod or wait until the end. Writing to disk is a bottleneck that is a performance killer in things like this… even on an SSD.

A further optimization…

Create a class to store the individual integer results and R and then do the string conversion when writing to disk. So all this should wait until you’re actually writing the data.

VisualizzaStringa=ArraiIngranaggiRuotaE(E).ToText+"*" +ArraiIngranaggiRuotaC(C).ToText+"*"+ArraiIngranaggiRuotaA(A).ToText + " / " +ArraiIngranaggiRuotaF(F).ToText+"*"+ArraiIngranaggiRuotaD(D).ToText+"*"+ArraiIngranaggiRuotaB(B).ToText
            
output = str(ArraiIngranaggiRuotaA(A))  + ", " + str(ArraiIngranaggiRuotaB(B))  + ", " + str(ArraiIngranaggiRuotaC(C))  + ", " + str(ArraiIngranaggiRuotaD(D))  + ", " + str(ArraiIngranaggiRuotaE(E))  + ", " + str(ArraiIngranaggiRuotaF(F))  + ", " + VisualizzaStringa + ", " + str(R)

t.writeline(output)

(Stupid forum won’t let me answer more than three in a row…)

Regarding this issue, I suggest assigning proxies within the arrays that are always unique, to which you assign the real values for calculation purposes.

So maybe the arrays are always A,B,C,D,E,F and they represent 10. 20, 20, 30, 30, 40. Because the array values are unique, you can check for uniqueness during the loop. It is only when doing the calculations that you need the actual values and you can re-associate them at that time.

Meanwhile, thanks for all the suggestions, I wrote the code as suggested, but there are many duplicates … in the table in the attached figure, values ​​such as line 55 or 56 should come out that do not have duplicates. A value if I use it I can’t reuse it, unless there are two equal values ​​in the array.

''OUTPUT FORMULA:   R= (A*C*E)/(B*D*F)
dim start as integer
start = System.Microseconds
var output as string
Dim ArrayIngranaggi() as integer=array(40,44,48,56,64,66,72,100)

Dim VisualizzaStringa as string

Var file As FolderItem = SpecialFolder.Desktop.Child("testOutPut.csv")
Var t As TextOutputStream = TextOutputStream.Create(file)

ListBox1.ColumnCount=8

Dim R as currency
Dim ace,bdf,F as integer


For F=0 To ArrayIngranaggi.LastIndex-5
  
  ace = ArrayIngranaggi(F) * ArrayIngranaggi(F+1) * ArrayIngranaggi(F+2)
  bdf = ArrayIngranaggi(F+3) * ArrayIngranaggi(F+4) * ArrayIngranaggi(F+5)
  
  R = ace / bdf
  VisualizzaStringa="("+ArrayIngranaggi(F).ToText+"x" +ArrayIngranaggi(F+1).ToText+"x"+ArrayIngranaggi(F+2).ToText + " / " +ArrayIngranaggi(F+3).ToText+"x"+ArrayIngranaggi(F+4).ToText+"x"+ArrayIngranaggi(F+5).ToText+")"
  
  output = str(ArrayIngranaggi(F))  + "**" + str(ArrayIngranaggi(F+1))  + "**" + str(ArrayIngranaggi(F+2))  + "**" + str(ArrayIngranaggi(F+3))  + "**" + str(ArrayIngranaggi(F+4))  + "**" + str(ArrayIngranaggi(F+5)) + "**" +"-------"+ "**" + VisualizzaStringa + "**" +" ------------"+"**"+ str(R)  
  t.writeline(output)
  
Next


t.close
MessageBox str((System.Microseconds - start) / 1000000) + " seconds"

I was thinking that actually with an array you could handle everything, you have to understand how to move the indexes. I did this test, and I get some combination without repetition, but I can’t figure out how to get them all…
Immagine

for example, other combinations with 40 could be:

404448 // 404456 // 404464 // 404472 // 4044100 // 405664 ecc…ec.c…

The code you showed in your post is NOT the code that I gave you in my previous post. Did you try my code?

Of course I tried it, unfortunately it doesn’t work, the numbers are repeated … the final part scales, but the first ones are repeated and it doesn’t work.

I’m trying other ways, but nothing.

I am attaching screenshot of the result of your code.

Then it’s not clear what you are trying to do.

The numbers don’t repeat in the numerator, and they don’t repeat in the denominator. But, I thought that you wanted all of the numbers to occur exactly once in the numerator and exactly once in the denominator. That is what my previous code does.

If you don’t want the numbers to repeat between the numerator and denominator at all then you can set the loop indices like this:

For A=0 To ArraiIngranaggiRuotaA.LastIndex-5
  For C=A+1 To ArraiIngranaggiRuotaB.LastIndex-4
    For E=C+1 To ArraiIngranaggiRuotaC.LastIndex-3
      For B=E+1 To ArraiIngranaggiRuotaD.LastIndex-2
        For D=B+1 To ArraiIngranaggiRuotaE.LastIndex-1
          For F=D+1 To ArraiIngranaggiRuotaF.LastIndex
            
            ace = ArraiIngranaggiRuotaA(a) * ArraiIngranaggiRuotaC(c) * ArraiIngranaggiRuotaE(e)
            bdf = ArraiIngranaggiRuotaB(b) * ArraiIngranaggiRuotaD(d) * ArraiIngranaggiRuotaF(f)
            
            R = ace / bdf
            VisualizzaStringa="("+ArraiIngranaggiRuotaA(A).ToText+"x" +ArraiIngranaggiRuotaC(C).ToText+"x"+ArraiIngranaggiRuotaE(E).ToText + " / " +ArraiIngranaggiRuotaB(B).ToText+"x"+ArraiIngranaggiRuotaD(D).ToText+"x"+ArraiIngranaggiRuotaF(F).ToText+")"
            
            output = str(ArraiIngranaggiRuotaA(A))  + "**" + str(ArraiIngranaggiRuotaB(B))  + "**" + str(ArraiIngranaggiRuotaC(C))  + "**" + str(ArraiIngranaggiRuotaD(D))  + "**" + str(ArraiIngranaggiRuotaE(E))  + "**" + str(ArraiIngranaggiRuotaF(F)) + "**" +"-------"+ "**" + VisualizzaStringa + "**" +" ------------"+"**"+ str(R)  
            t.writeline(output)
            
          Next
        Next
      Next
    Next
  Next
  
Next

If that doesn’t work for you, then you need to explain exactly how you want the numerator and denominator numbers to vary.

This is already much better, there are many combinations without repeating numbers, BUT, missing combinations… after the last line there would still be:
48x100x56 or 48x72x66 … or … 72x56x100

The numbers are arranged in order of increasing index values. So, the value
48x100x56 is present as 48x56x100
and
48x72x66 is present as 48x66x72
and so on.

100 or 72 in the numerator never sample

I would need all the possible combinations without repetitions, both in the numerator and in the denominator … sorry, but I don’t know how to explain it to you.
I have to get all the combinations I can with the values inside the array…in the result I posted of your code, it shows that for example 72 or 100 never end up in the numerator, but always remain in the denominator.

When I run the code I get 72 and 100 in the numerator. Here is a partial list:
(40 x 72 x 100) / (40 x 44 x 56)
(40 x 72 x 100) / (40 x 44 x 64)
(40 x 72 x 100) / (40 x 44 x 66)
(40 x 72 x 100) / (40 x 44 x 72)
(40 x 72 x 100) / (40 x 44 x 100)
(40 x 72 x 100) / (40 x 48 x 56)
(40 x 72 x 100) / (40 x 48 x 64)
(40 x 72 x 100) / (40 x 48 x 66)
(40 x 72 x 100) / (40 x 48 x 72)
(40 x 72 x 100) / (40 x 48 x 100)
(40 x 72 x 100) / (40 x 56 x 64)
(40 x 72 x 100) / (40 x 56 x 66)
(40 x 72 x 100) / (40 x 56 x 72)
(40 x 72 x 100) / (40 x 56 x 100)
(40 x 72 x 100) / (40 x 64 x 66)
(40 x 72 x 100) / (40 x 64 x 72)
(40 x 72 x 100) / (40 x 64 x 100)
(40 x 72 x 100) / (40 x 66 x 72)
(40 x 72 x 100) / (40 x 66 x 100)
(40 x 72 x 100) / (40 x 72 x 100)
(40 x 72 x 100) / (44 x 48 x 56)
(40 x 72 x 100) / (44 x 48 x 64)
(40 x 72 x 100) / (44 x 48 x 66)
(40 x 72 x 100) / (44 x 48 x 72)
(40 x 72 x 100) / (44 x 48 x 100)
(40 x 72 x 100) / (44 x 56 x 64)
(40 x 72 x 100) / (44 x 56 x 66)
(40 x 72 x 100) / (44 x 56 x 72)
(40 x 72 x 100) / (44 x 56 x 100)
(40 x 72 x 100) / (44 x 64 x 66)

The complete list is too long to post here.