I think this is relying on the graphics context’s settings. Do you have any insight if there are already declares for it?
Also, the way the image is created might influence the quality. In this short video you see an OS X ImageWell with a Xojo Picture cast on it. When “High Quality” is clicked, the NSViews image property is set to another copy of this picture, only this time via a NSImage created via initWithData coming from OriginalPicture.GetData (Picture.FormatPNG). This takes incredibly much longer but is a lot smoother.
EDIT: Probably easier to see: On the left a NSImage created from the Xojo image via copyOSHandle.CGImage, in the middle via PNG data and on the right just the Xojo image cast to the ImageWell directly. Times for setting the images in the textarea.