R2020 Workers - Question and suggestion

I can only guess.

From the Video conference today, it is likely considered to be the “minimum viable implementation”.

  • karen

But it’s not near complete so I expect this will evolve somewhat before release.

[quote]@Kem Tekinay Q&A just wrapped up. Just to be clear, in v.1 the Worker won’t have access to anything in the main project, including your classes, modules, or methods. That will have to be included in the ProcessJob (or whatever it was called) event.
At least, for now.[/quote]
This is massively disappointing and severely limits their use.

For example, I have an app that converts lots of Markdown text files to HTML. It uses my open source MarkdownKit Xojo module. It would be ideal for Workers to be able to use this module to take the contents of a file as a String (containing Markdown text) open a Markdown file, convert it to HTML and then return the HTML as a String. The issue is, the Workers seemingly won’t have access to the MarkdownKit module, even though it’s just vanilla Xojo code.

[quote=481205:@Garry Pettet]This is massively disappointing and severely limits their use.
[/quote]

So much so, with that limited an implementation, I would put it more in the realm of a feature that ticks a marketing box rather than significant multicore support 
 The problem is that a lot of Xojo starts feeling like that to longer term users, and it takes a LONG time (if ever) for such minimal features to “mature”.

  • Karen

I have a video processing app that already does all the individual frame processing in “workers” which in my case is an actually separate console app. The console app and the main app use the MBS Plugins shared memory functions to send the raw picture data to the background app for processing. There were several tricky bits in getting it all to sync up but it definitely works. I don’t see any reason that you couldn’t wrap the simpler worker class in a subclass that provided for that functionality. It would not be terribly difficult. The main reason I went with this technique years ago is that I was having trouble with the core image stuff that would sometimes hang if you asked it to decompress an ever so slightly corrupt jpeg image and it also tended to leak some memory. So the main app never hangs just whichever video stream sent bad data and I can regularly restart the background processes that are leaking the most memory. Simply was never able to find what was getting lost but it doesn’t matter with being able to do that.

The pipe between the 2 could be used to send an encapsulated dictionary of commands or information necessary to open the shared memory location and size and such and what you need to do to the data. And a completion message can be sent back over the same pipe to let the host know it’s ready.

We can add those sorts of things to it in a subclass the day after they include it into a Xojo pre-release :wink: Yes it would be very nice to have some shared data or some automatic sharing of variables and obejcts or something, but I can see the complexity of doing that to a totally separate process.

With the philosophy of it being a totally separate process accepted though, what is really to be gained by doing that as opposed to just a completely isolated truly MP thread? That would open the door for some easier sharing between the thread and the app. There could be some really formalized methods of talking to it as if it were a separate process, but it would still be easier to do it than serializing everything you needed to the pipe or managing our own shared memory.

That being said I also send a LOT of larger data through IPC sockets and in some cases TCP sockets often, but not always, connected to localhost. It is not slow. If you are sending multiple gigabytes than the transfer will be a significant portion of the processing time for sure. If you’re just sending a few meg it really can happen very quickly as long as your encapsulation system on the socket isn’t doing too much copying and multiple movings of the data in order to process it as it comes in.

After all that babbling :wink: @Christian Schmitz do you know if the MBS File Mapping stuff will be allowed in the App store? Do they use any of the dreaded Sys V calls?

Again, this is an unreleased feature still early in its development. I am optimistic it will evolve before release, and after too.

Since I also have console helper apps which use the MBS functions, I thought I’d run a small test. My code is code signed and notarized (thanks to Kem’s excellent App Wrapper!), but as it is not distributed via MAS, I did not previously enable sandbox and MAS restrictions. So I went back into App Wrapper and enabled those and rewrapped the master and helper apps.

The good news is that it did code sign, notarize, and launch without any warnings. The bad news is that the main app did not see progress updates in the shared memory from the helper apps. I’m sure this is due to the sandbox, and perhaps there is a simple way around it. But since I don’t need MAS compatibility, and it is off topic for the Worker class anyway, I didn’t persue it more.

Er, my what now?

Oops, sorry, Sam’s excellent App Wraper. My bad. However, I do use some of your other stuff which I really appreciate!