Guess: Using Quit is probably what's causing the dialog to appear twice. Some kind of funny order of events that logically works itself out to CancelClose occurring twice.
If you are trying to ask about saving a document, using specific phrasing can get the classic keyboard shortcuts working (Cmd-D for Don't Save).
This is what I recommend for a "Would you like to save changes before closing?" CancelClose. Place it on the Window, not App, nor should you invoke Quit manually at any point. This example uses the phrasing that allows the user to control the dialog with shortcuts on Mac. While nothing special happens on Windows, no workarounds are necessary to use this example on Windows.
Function CancelClose(appQuitting as Boolean) Handles CancelClose as Boolean
// Don't bother the user if the document isn't dirty
if self.ContentsChanged = false then return false
dim oDlg as new MessageDialog
oDlg.Message = "Do you want to save changes to this document before closing?"
oDlg.Explanation = "If you don't save, your changes will be lost."
oDlg.ActionButton.Caption = "&Save"
oDlg.CancelButton.Caption = "Cancel"
oDlg.CancelButton.Visible = true
oDlg.AlternateActionButton.Caption = "&Don't Save"
oDlg.AlternateActionButton.Visible = true
dim oBtn as MessageDialogButton = oDlg.ShowModalWithin(self)
select case oBtn
// Cancel close
// Don't Save