multithreading with multiple core machines

  1. 8 months ago

    Perry P

    9 Oct 2019 Boston, MA

    In reading about threading in Xojo I see that it doesn't assign threads to different cores. Is there any way to do that? My use case is an application that needs to go through a folder that could contain tens of thousands of files, and generate a checksum of each file. When the application encounters a large file, such as a 500GB Quicktime movie, the md5 hash is very slow, and bogs down the app until it's done. When it hits a folder of TIFF files, though, it zips right through them. I'd like to spawn an arbitrary number of threads, based on the core count of the machine at hand (it would be a user preference), then have those do a checksum and report back to the main thread. It seems I can't do this with threading, so how would I?

    What are the strategies for taking advantage of multi-core machines with Xojo, if any?

    Thanks!

  2. Louis D

    9 Oct 2019 Testers, Xojo Pro QC, Canada

    Helper apps is the usual multi-core strategy. Helper apps that you spawn will generally use other cores than the main application. There is an example under console/multiprocessing.

  3. Paul L

    9 Oct 2019 Xojo Inc, Third Party Store

    You can start separate console apps on multiple cores and have each of them communicate results back to a main app. More information:

  4. Perry P

    9 Oct 2019 Boston, MA

    Ahh, ok. Well that certainly seems easier than dealing with threads in some ways...

    Thanks

  5. Perry P

    9 Oct 2019 Boston, MA
    Edited 8 months ago

    How would I go about creating (and then referencing) an arbitrary number of shells?

    Each machine this application runs on has highly variable hardware. My little iMac is a measly 6-core i7, but we have some linux boxes with dual 14-core Xeons, and some windows machines with 12 and 14 core CPUs as well.

    I'm having hard time wrapping my head around how I'd set up a user-definable number of shells and how I'd reference them. I would need to run a loop that looks for a shell that's finished, get its result and sends it another task. Hardcoding say, 8 shells, is no big deal, but I may not need 8 in some cases, or I may want more.

  6. Norman P

    9 Oct 2019 Testers, Xojo Pro outside LMAO !!!!!!!

    you could ask the CPU for the number of cores it has and create that number of shells (or maybe a couple less)

  7. Perry P

    9 Oct 2019 Boston, MA
    Edited 8 months ago

    That's not really the question though. (but it's not what I want to do here because I don't necessarily want to max out the capability of the machine - there are times when we might be doing this in the background so we don't want to bog it down).

    The question is - how do I programmatically set up X number of shells, and then once set up, how do I reference each one?

    What I'm looking to do is set up a "pool" of shells that I can call as needed until the overall job is done.

  8. Perry P

    9 Oct 2019 Boston, MA

    @Markus W How to get the number of available CPUs and cores has been explained in xDev Developer magazine 13.6 (Mac), 14.1 (Linux), 14.3 (Windows), 14.4 (Windows).

    This isn't what I'm asking.

  9. Christian S

    9 Oct 2019 Testers, Xojo Pro, XDC Speakers, Third Party Store Germany
    Edited 8 months ago

    MBS Xojo Plugins come with a lot of multithreaded methods, which can help you to use multiple cores.

    see blog post:
    Multithreaded plugin functions can increase speed of Xojo application

  10. Markus W

    9 Oct 2019 #JeSuisHuman New Zealand, Auc...
    Edited 8 months ago

    @PerryPaolantonio This isn't what I'm asking.

    If you don’t know how many cores you CAN use then how do you determine how many shells you should set up?

    But suit yourself. Answer deleted as “off the mark“.

  11. Kyryl P

    9 Oct 2019 Xojo Pro Europe (Ukraine)

    @PerryPaolantonio The question is - how do I programmatically set up X number of shells, and then once set up, how do I reference each one?

    Use an array of Shells

  12. Christian S

    9 Oct 2019 Testers, Xojo Pro, XDC Speakers, Third Party Store Germany

    You could use SystemInformationMBS ProcessorCount function to check the core count.

  13. Norman P

    9 Oct 2019 Testers, Xojo Pro outside LMAO !!!!!!!

    @PerryPaolantonio That's not really the question though. (but it's not what I want to do here because I don't necessarily want to max out the capability of the machine - there are times when we might be doing this in the background so we don't want to bog it down).

    The question is - how do I programmatically set up X number of shells, and then once set up, how do I reference each one?

    What I'm looking to do is set up a "pool" of shells that I can call as needed until the overall job is done.

    shells ? make an array of them ?
    add as many as you want
    they are instances of a class - the Shell class in fact - so you create as many as you want in a loop with "New Shell"

  14. Andrew L

    9 Oct 2019 San Francisco, CA, USA

    @PerryPaolantonio When the application encounters a large file, such as a 500GB Quicktime movie, the md5 hash is very slow, and bogs down the app until it's done.

    Are you using the MD5Digest class? If not then try it; it's better suited for processing large files than the MD5 method.

  15. Perry P

    9 Oct 2019 Boston, MA

    @Markus W If you don’t know how many cores you CAN use then how do you determine how many shells you should set up?

    It's a user preference in the application. It will vary depending on the machine. Some days you can afford to leave a machine running flat out, some days you can't. We've found (using a command line tool that does what we're building this custom app for) that this is very much dependent upon the I/O load on our SAN and on the speed of the available cores. It's a balancing act, but letting it go full tilt by using too many cores will definitely slow the overall process down - either by bogging down the machine or overloading the disk I/O capability of the system.

  16. Perry P

    9 Oct 2019 Boston, MA

    @Kyryl P Use an array of Shells

    Thanks - that never occurred to me. I'll look into this now.

  17. Perry P

    9 Oct 2019 Boston, MA

    @Andrew L Are you using the MD5Digest class? If not then try it; it's better suited for processing large files than the MD5 method.

    I am. While MD5 is the most common request from our clients, the specification we're complying with also supports sha1, sha256 and sha512. I haven't tried the other algorithms yet to test for speed. Is there an equivalent to md5digest for those?

  18. Christian S

    9 Oct 2019 Testers, Xojo Pro, XDC Speakers, Third Party Store Germany
    Edited 8 months ago

    Please check our classes in MBS Xojo Encryption Plugin :

    They all have multi threaded HashFile methods, which give time to other Xojo threads while hashing on a different CPU core. SO when you hash 8 files at the same time, you can get 8 cores busy!

  19. Markus W

    9 Oct 2019 #JeSuisHuman New Zealand, Auc...

    @PerryPaolantonio It's a user preference in the application. It will vary depending on the machine. Some days you can afford to leave a machine running flat out, some days you can't. We've found (using a command line tool that does what we're building this custom app for) that this is very much dependent upon the I/O load on our SAN and on the speed of the available cores. It's a balancing act, but letting it go full tilt by using too many cores will definitely slow the overall process down - either by bogging down the machine or overloading the disk I/O capability of the system.

    Again: how do you know how many cores there are? It might be a user preference on how many cores you WANT to use but there needs to be a maximum that you can't and should not exceed … or can a user set it for example to 100 on a 6 core machine? Because that is possible, just not recommended.

    Maybe the users know what hardware they are running on and can set it accordingly - but they should not HAVE TO know - the system should tell them.

    And in the future other users might not know - it is bad practise to depend on user knowledge.

    But that's just my personal opinion.

    Personally I always leave at least one core free on my machine for general tasks.

  20. Newer ›

or Sign Up to reply!