First of all, thank you @Julian S @Anthony C and everyone else spent time and efforts for helping me on finding a solution.
@Julian S your solution works pretty much well for this case.
However this solution has some major flaws for my prospected use that make it impossible or really hard to use it:
- As I initially stated, this is a cross platform solution. Setting the modal dialog as a Floating window prevent to show it on Mac as a Sheet, and this is not acceptable for my requirements. Moreover, the floating window disappear when the whole app goes in background, but most important, with this approach it would require a Mac solution to behave like on Windows due the standard way is broken. That is, would be like fixing something which is already working.
- The popup window I have to show is part of a component (actually many different components) which have no knowledge of the Base window that ultimately fired its execution. But this approach rely on disabling the base window before opening the popup and enabling it again from the popup on close. This means redesign everything to pass along the base window to modal dialogs and redesign as well the components to take this base window reference. This for something like 100+ applications. ugh...
- The fake ModalDialog is indeed only mocking the modal UI behaviour but it remain modeless, that is the base window still execute code after showing the (fake) modal dialog. Again this can be handled in some way, but again this would be a huge amount of work on refactoring how applications works.
In the end, this would require a major refactor of my codebase that I can't afford at this time. And still I feel there could be other unwanted side effects.
I guess at this point I will go with my solution of keeping a timer running and check/set the SetCapture method. Still far from perfect but at least easy to implement and effective.
Nonetheless, I do understand the problem (probably) lies on a limitation on the Xojo framework and/or the Win32 framework and this is probably one of the best solutions to solve the problem. I also do understand you guys tried hard to help me and spent time on it, so I wish to thank you all again for the efforts and please don't feel offended by my critics.