Optimize filling a Dictionary

Targets Windows
41

[quote=256621:@Rob Egal]Checked on OSX in 32/64 bit. Difference is -1sec for 64bit.
Don’t worry. It’s the framework unfortunately. You should rather go with sqlite db instead of waiting for fixes I think.[/quote]

Yes, I will use databases whenever possible. But for these projects we have to process a lot of data, coming from all sorts of sources like XML and CSV files. Dictionaries are an important instrument in the total process.

This particular example was just one of the tests (processing bulk data) I’m doing to examine if those projects can be done in Xojo.

42

There is another possible way, if you do not need the data read in real time : quickly store it in an array, then use a thread to move that queue to the dictionary.

That way you are sure not to miss a bit upon getting data.

43

Text <> String. The old String is more like a MemoryBlock. If you don’t care about encodings, you might try using a MemoryBlock instead. Probably the mutable version.

44

I’m from Europe, so encoding is a thing to care about.

45

[quote=256668:@Michel Bujardet]There is another possible way, if you do not need the data read in real time : quickly store it in an array, then use a thread to move that queue to the dictionary.

That way you are sure not to miss a bit upon getting data.[/quote]

I need it in realtime, so threads aren’t an option.

46

Are you creating something involved in high frequency trading ?

More seriously, unless the flow of data is continuous, using a buffer is a perfectly acceptable way of dealing with overflow.

If indeed data input exceeds dictionary speed, I frankly do not see how you can do it in a Xojo dictionary without some buffering.

And, maybe you see threads as slow, but they don’t have to be.

47

[quote=256525:@Johan van Breemen]I’m converting some PureBasic projects to Xojo. In these projects I use dictionaries a lot. I’m experiencing some serious performance issues here.
[/quote]
Is the PB code compiled with Unicode support on or off ?
Even when using the older API’s you are ALWAYS using UTF-8 internally

48

[quote=256697:@Norman Palardy]Is the PB code compiled with Unicode support on or off ?
Even when using the older API’s you are ALWAYS using UTF-8 internally[/quote]

Unicode is on in PB project and tested on same machine.

49

Threads aren’t making the process faster (or slower). The processes I have to do are in sequence, so using threads is of no use.

50

FYI: I just did run my own test project on a 2012 MBP and it finished in +/- 11 seconds. PB time on the MBP is 0,6 second.

Something in the Windows environment acts differently. But don’t forget: the PB project was tested on the same Windows machine.

51

I’d clarify that to say that something in your Windows environment acts differently since others here on Windows are not experiencing the same. It would probably be helpful if you could work with Xojo to narrow it down.