Clipboard - macOS problem

[AlbertoD]

I made some tests after finding some inconsistencies on the way EndOfLine is handled on mac and windows (all my tests on Linux are consistent).

I made a simple project, 2 TextArea, 2 buttons to break and check values, a button to use clipboard to copy from TextArea1 to TextArea2 and a button to check if the TextArea values are the same. It looks like this:

Then I write:

1 2 3
on the left TextArea. The hex values are: 310A 320A 33

Then I select all and do a copy/paste to the right TextArea. The values stayed the same. If I click the ‘Equal?’ button it say the contents of both TextAreas are the same.

Then I click the ‘Clipboard’ button, to use this code:

Dim c As New Clipboard c.Text = TextArea1.Text TextArea2.Text = c.Text c.close

Now the right TextArea look exactly the same, but the hex values are: 310D 320D 33. Pressing the ‘Equal?’ button will say they are not the same.

Now the most interesting part. Select all on the right TextArea (with the 0D end of line), do a system copy/paste over the values on the left TextArea and check the hex values, the values are: 310A 320A 33. So the mac system is changing the chr(13) end of line characters to chr(10), that’s the opposite of what Xojo’s clipboard function is doing. If I click the ‘Equal?’ button it say they are not the same, but if I do a system copy/paste on the right TextArea over the values of the right TextArea, now the values are equal because the system change the ‘wrong’ end of lines.

This is on mac 10.12.6

Note: tested under Linux and the end of line value is not changed, that’s what I expected under mac.
Note2: tested under Windows and the end of line value is not changed, the problem here is that TextArea on Windows just uses 0D and 0D0A is expected

Should I file a Feature request or a Bug report? I expect Xojo’s clipboard behave like the system copy/paste (copy 0A and not change to 0D, and if it finds a 0D to change to 0A)

[Greg_O_Lone]

if You are concerned about line endings when bringing data into your app, yes, even from the clipboard, you should run ReplaceLineEndings on the text.

[AlbertoD]

What I’m concerned is that 0A is changed to 0D when using Xojo’s clipboard on mac and no change on Linux.

And that mac’s system edit-copy/edit-paste keep 0A and if there is 0D it changes to 0A. So if mac system prefers and uses 0A over 0D, why Xojo’s clipboard change 0A to 0D.

[Eduardo_Gutierrez_de_Oliveira]

[quote=408503:@Alberto De Poo]What I’m concerned is that 0A is changed to 0D when using Xojo’s clipboard on mac and no change on Linux.

And that mac’s system edit-copy/edit-paste keep 0A and if there is 0D it changes to 0A. So if mac system prefers and uses 0A over 0D, why Xojo’s clipboard change 0A to 0D.[/quote]

The reasons for the different are historical but, like character encodings, impossible to have a universal rule for everything.

If you don’t convert, things will look wonky in at least one platform. If you convert, then you’re really changing the content and it might break somewhere else.

My assumption is that Xojo internally uses linefeeds (0A) because that’s what classic mac used and tries to convert as necessary but may be missing some cases. LInux uses carriage returns (0D) because that’s what old terminales used and DOS (and by extension Windows) used both a linefeed and a carriage return (because CP/M).

In the end you’ll need to take a decision on what you need to do, and decide to whether always convert, never convert or leave it as a user preference. The traditional stance is to always convert (define the system’s newline custom and convert the others whenever they’re not this).

This was more of a problem when Mac used both characters for different things but nowadays they’re all somewhat safe to replace.

(This is still the reason why pasting from some places into Outlook or HTML WYSIWYG editors ends up adding tons of lines inbetween paragraphs, since HTML also distinguides linefeed breaks
from carriage returns between paragraphs

).

No matter what decision you take, unless you use styled text you’ll lose information at some point (especially if people have learned that shift-enter creates a linefeed that is different from a paragraph break in some editors and you try to copy that)

[AlbertoD]

Thank you Eduardo.

Sorry I mentioned Linux, let my rephrase my previous comment:
What I’m concerned is that 0A is changed to 0D when using Xojo’s clipboard (code) on mac, while the mac system (using Edit - Copy and Edit - Paste), do the opposite.

If you select something from Xojo’s TextArea that use 0D and do an Edit - Copy then Edit - Paste into other TextArea and check the hex values, the mac system changed 0D for 0A.

Why Xojo change 0A to 0D on mac, while the macOS change 0D to 0A?

Edit: could I open a feature request to change this behavior to match what macOS is doing?
Edit2: Eduardo, you say "Linux uses carriage returns (0D), but my tests and the docs say Linux uses linefeeds (0A)

[Eduardo_Gutierrez_de_Oliveira]

@408602:@Alberto De Poo

Edit2: Eduardo, you say "Linux uses carriage returns (0D), but my tests and the docs say Linux uses linefeeds (0A)

I wrote from memory. I could have mixed them up. I’m as confused as Xojo’s textarea

[AlbertoD]

No problem Eduardo, sorry if what I wrote looked harsh. I’m learning and I get confused (easily). It was just to make sure that is one or the other.

[Tanner_Lee1 (Tanner Lee)]

For reference, here’s more exhaustive information information on Newlines.

[Thomas_Tempelmann]

Being the co-author of the clipboard recorder iClip (macOS), which is written in Xojo, I have some more insights to share:

It’s the macOS code for clipboard handling that converts the text as necessary.

In fact, when you place text into the clipboard, it’ll end up in different forms in the clipboard, so that apps that expect a certain format will have a more likeliness to find their expected format in there. For instance, you may place text in there in the Classic Mac format, with MacRoman encoding, while a newer app may expect UTF-16. Therefore, macOS will convert your text into some other formats automatically.

Same happens with Images: You can put in a GIF, and macOS will make availble them as JPG, TIFF, PNG and some other rare formats, even.

And HTML will also be made as RTF, and vice versa.

So, with all these conversions, the line ending will also get “adjusted”.

Compare that to Linux, where nothing of this happens, AFAIK. Because, well, Linux sucks for UI apps

BTW, you can use my program “ShowClipboards” to look into the contents of the macOS clipboard’s flavors, if you like to explore them in detail:

http://files.tempel.org/iClip-public/ShowClipboards-Xojo.rbp.zip
http://files.tempel.org/iClip-public/ShowClipboards.app.zip

[AlbertoD]

Thank you @Thomas great program.

I think the problem: Xojo’s clipboard using data in:
com.apple.traditional-mac-plain.text // I guess the macOS auto-converts 0A to 0D here
and I think it should use
public.utf8-plain-text // here the value of 0A is preserved

I did several tests with TextAreas Styled ON and Styled OFF.

Styled ON

I think this happen because macOS is using the styled information in RTF format and it changes 0D to 0A. Found the information on Thomas app in public.rtf

even when public.utf8-plain-text has 0D

Styled OFF

As you can see, if I edit-copy and edit-paste, macOS uses what is in public.utf8-plain-text if 0A is there, then it paste that, if 0D is there it paste that.
Thomas app:

I hope this is clear.

[Thomas_Tempelmann]

So, do you still need help or do you have a plan to solve your question?

[AlbertoD]

Well, now I know what Xojo’s clipboard is doing, but still don’t know why.

I still expect the clipboard to use the utf8 value just as the system Edit-Copy and Edit-Paste work with Styled OFF TextAreas.

For my little program I can do a ReplaceLineEndings copying back the data from using Xojo’s clipboard.Text the problem is with EndOfLine.Windows on a Mac, that the original hex will be 310D0A32, when using clipboard.text will be copied back as 310D0D32, then using REplaceLineEndings(clipboard.Text, EndOfLine.Windows) I will end with 310D0A0D0A32.

I don’t think I will use EndOfLine.Windows but still, this may cause issues with someone’s code.

[Thomas_Tempelmann]

Why would you expect to get Windows line delimiters in a Mac clipboard? I’d say that won’t happen, so you do not have to worry about that.

[Emile_Schwarz]

http://documentation.xojo.com/index.php/EndOfLine

says in more details why Thomas is right. I too do not understand what the trouble was.

EdOfLine use the correct character for the running platform (as the LR says).

[AlbertoD]

@408889:@Emile Schwarz

EdOfLine use the correct character for the running platform (as the LR says).

Yes EndOfLine use the correct character for the running platform, the problem is that the clipboard (on a Mac) have several things, as Thomas pointed out and I’m finding that information thanks to ShowClipboards.

@408887:@Thomas Tempelmann

Why would you expect to get Windows line delimiters in a Mac clipboard? I’d say that won’t happen, so you do not have to worry about that.

Because if I use mac’s Edit-Copy the Windows line delimiter is copied to the clipboard, correct in one place and in “traditional” form in another (converting 0A to 0D).

My guess is that the clipboard behavior was not changed to match what Notes on https://documentation.xojo.com/index.php/EndOfLine on macOS say:

[quote]EndOfLine on macOS
NOTE: Starting with 2013r1, EndOfLine returns EndOfLine.Unix on all macOS apps.
This change was necessary because 2013r1 creates Cocoa apps by default and Cocoa works with the Unix line endings.
If you require the old behavior, you should explicitly use EndOfLine.Macintosh.[/quote]

I’m sorry that I can’t make it clear. I’ll try again, here I go:

• sample code that put an EndOfLine.Windows between 1 and 2
  

• now we can see the hex on TextArea, as you can see there is 0D0A because that’s the windows line ending
  

• I do an Edit - Copy, to see if the Windows line ending is in a Mac clipboard
  

• thanks to ShowClipboards, I can see that the correct value is available
  

  

• also I can see there is another “traditional” value that auto convert 0A to 0D (no matter if it is Windows, Unix or OSX line endings)
  

  

I think Xojo is using the “traditional” clipboard value when I use clipboard.Text and I think they should use the public plain text utf8, that’s it.

I hope is more clear now. Yes, maybe I’m making more trouble than this deserves. Sorry for waiting your time with long never-ending posts. It makes me sad when I can’t explain/make an idea clear. For me is very clear but I see that my message is not getting thru.

[AlbertoD]

I did more tests.

Dim s As String = "1" + EndOfLine + "2" // hex value 310A32 Dim c As New Clipboard c.Text = s s = c.Text // hex value 310D32
Thanks to ShowClipboards app I was able to learn that the correct value was put into public.utf8-plain-text, so to make it work as expected, we can use the following code:

Dim s As String = "1" + EndOfLine + "2" // hex value 310A32 Dim c As New Clipboard c.Text = s s = DefineEncoding(c.RawData("public.utf8-plain-text"), Encodings.utf8) // hex value 310A32

Edit: you can use c.RawData("utf8") instead the full name

[AlbertoD]

Another test with EndOfLine.Windows on Mac:

Dim s As String = "1" + EndOfLine.Windows + "2" // hex value 310D0A32 Dim c As New Clipboard c.Text = s s = c.Text // hex value 310D0D32, not what I expect s = DefineEncoding(c.RawData("utf8"), Encodings.utf8) // hex value 310D0A32