We did this around a year ago on macOS and MS-Windows. Our app had the need for a custom WYSIWYG font menu similar to the Adobe suite of application. We initially looked at using a modal dialog but it just didn’t look right.
We eventually ended up with a solution that works well enough and allows us to create other types of custom pop ups in the future. We even integrated it with the edit field part of the Xojo combo box so that users can filter the contents of the list while it is open.
Implementing the solution wasn’t easy though. Quite a bit of time was spent working out how the built-in controls respond, writing test code, investigating solutions to various problems we came across and writing a production quality control that could be re-used and extended. It also required the MBS plugins, declares and at least one custom wndproc.
For us, all of the effort was worth it as it was going into an application used by tens of thousands of users each month. If you don’t have the same requirements I would see if the much simpler solution of a modal dialog is good enough.