# Encoding problem with Sonoma? **URL:** \ **Category:** macOS\ **Created:** [May 31, 2024, 2:45pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510 "2024-05-31T14:45:54Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [May 31, 2024, 2:45pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/1 "2024-05-31T14:45:54Z")
I’m having an encoding problem with Sonoma, at least that’s where I think the problem is. I have time strings, e.g. “5/12/24 11/54/27 AM” coded as “String”. To save them to a file, I first write them to a MemoryBlock using StringValue: myblock.StringValue(offset,mytime.Length) = mytime This has worked fine up until I run it under Sonoma. For some reason, the last 3 characters are now encoded differently. I’ll get “5/28/24 10/46/57���”, with " AM" corresponding to hex “E2 80 AF” instead of “20 41 4D”. What’s going on here and how can I fix it? ---
**Author:** ![AlbertoD](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/dbc845/32.png) [@AlbertoD](https://forum.xojo.com/u/AlbertoD)\ **Post date:** [May 31, 2024, 2:51pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/2 "2024-05-31T14:51:42Z")
I think I read somewhere that now the space before ‘AM’ is a non-breaking space (or something like that). Maybe you need to define UTF-8 as the encoding? ---
**Author:** ![Rick\_Araujo](https://forum.xojo.com/user_avatar/forum.xojo.com/rick_araujo/32/251_2.png) [@Rick\_Araujo](https://forum.xojo.com/u/Rick_Araujo)\ **Post date:** [May 31, 2024, 3:08pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/3 "2024-05-31T15:08:01Z")
The unicode specifies that before “AM/PM” the space separator is not the usual " " it is a specific “narrow space”, Xojo unicode : &u202f → " " 11:22:33 PM // normal space, breaks lines 11:22:33 PM // narrow non-breaking one, a little bit closer, it acts more like glue than a divider. As Alberto said, your encoding seems improper. ---
**Author:** ![TimStreater](https://forum.xojo.com/user_avatar/forum.xojo.com/timstreater/32/586_2.png) [@TimStreater](https://forum.xojo.com/u/TimStreater)\ **Post date:** [May 31, 2024, 6:16pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/4 "2024-05-31T18:16:50Z")
From the very useful Unicode/UTF-8 reference at: [https://www.utf8-chartable.de/unicode-utf8-table.pl](https://www.utf8-chartable.de/unicode-utf8-table.pl) we have this entry: U+202F e2 80 af NARROW NO-BREAK SPACE So, nothing to do with Sonoma. ---
**Author:** ![AlbertoD](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/dbc845/32.png) [@AlbertoD](https://forum.xojo.com/u/AlbertoD)\ **Post date:** [May 31, 2024, 7:36pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/5 "2024-05-31T19:36:41Z")
> [@TimStreater](#): > > So, nothing to do with Sonoma. Are you sure? use this code: ```auto Var dd As DateTime = DateTime.Now Var test As String = dd.ToString ``` ![image](https://forum.xojo.com/uploads/default/original/3X/b/4/b4240c201dc664d8c0b10b3e7f92f37fd53a3bfa.png) macOS 13.6.7 (Intel), Xojo 2024r1.1 * * * And this is the same code with macOS 14.5 (M2), Xojo 2024r1.1 ![image](https://forum.xojo.com/uploads/default/original/3X/2/1/21d52fa8b24951fda6e047f0d3ad2573f2350144.png) Edit: looks like Sonoma now adheres to the unicode standard (what Rick posted above) while previous versions use a normal space. ---
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [May 31, 2024, 7:44pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/6 "2024-05-31T19:44:06Z")
Thanks for the comments. I’ll fix it with encoding. What’s strange to me is that this has been working for years with all MacOS versions up through Ventura. The only other difference in my testing is that the Sonoma machine is an M2 versus other test machines having M1 or Intel, but I can’t see how that would make a change like this. Thanks. ---
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [May 31, 2024, 7:45pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/7 "2024-05-31T19:45:35Z")
Nailed it. ---
**Author:** ![TimStreater](https://forum.xojo.com/user_avatar/forum.xojo.com/timstreater/32/586_2.png) [@TimStreater](https://forum.xojo.com/u/TimStreater)\ **Post date:** [May 31, 2024, 8:45pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/8 "2024-05-31T20:45:05Z")
Depends on your PoV, I suppose (he said, splitting hairs). I’d call it an encoding problem which Sonoma happens to highlight. ---
**Author:** ![Rick\_Araujo](https://forum.xojo.com/user_avatar/forum.xojo.com/rick_araujo/32/251_2.png) [@Rick\_Araujo](https://forum.xojo.com/u/Rick_Araujo)\ **Post date:** [May 31, 2024, 10:38pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/9 "2024-05-31T22:38:04Z")
Technically there’s no PoV involved. There are 2 things in this report: There’s a bug in John’s code he found due to a bad encoding receiving a non-ASCII UTF-8 codepoint. AND Sonoma (or Xojo 2024r1.1+ under Sonoma, not sure how this pair handle ICU) updated the ICU/CLDR changing the behavior. It is advisable that if you “parse” such string, never trust if the separator is &u0020 or &u202f, accept the possibility of being anyone of those. ---
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [June 1, 2024, 1:28pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/10 "2024-06-01T13:28:44Z")
I was mistaken about the origin of this problem (writing to memoryblock). This unicode character (3 bytes) is output by the DateTime.ToString function. The reason I’m not seeing the AM or PM (just the unknown characters) is that I’m using String.Length for how many bytes to write to the memoryblock and it’s now 2 bytes longer (I guess I should have used String.Bytes). But I want to save it as simple string anyway. I couldn’t figure out how to change those bytes back to a space character with Encodings. When I tried to use ConvertEncoding, I got a “?”. I ended up using ReplaceAllBytes to do it: mytime = mytime.ReplaceAllBytes(&u202f, " ") Maybe there’s a more eloquent way to do it, but this works for now. ---
**Author:** ![TimStreater](https://forum.xojo.com/user_avatar/forum.xojo.com/timstreater/32/586_2.png) [@TimStreater](https://forum.xojo.com/u/TimStreater)\ **Post date:** [June 1, 2024, 1:31pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/11 "2024-06-01T13:31:55Z")
> [@John\_Eidsvoog](#): > > I couldn’t figure out how to change those bytes back to a space character with Encodings. When I tried to use ConvertEncoding, I got a “?”. I ended up using ReplaceAllBytes to do it: I don’t see that you could accomplish that with ConvertEncoding, unless you tried to convert to ASCII. But then it would depend how clever ConvertEncoding is. ---
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [June 1, 2024, 1:51pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/12 "2024-06-01T13:51:43Z")
Fixing this seemed to me to be a big waste of time, but I’m sure all those people who lost sleep over the huge amount of wasted space before “AM” or “PM” will be happy now that Sonoma has come along. 😃 ---
**Author:** ![Rick\_Araujo](https://forum.xojo.com/user_avatar/forum.xojo.com/rick_araujo/32/251_2.png) [@Rick\_Araujo](https://forum.xojo.com/u/Rick_Araujo)\ **Post date:** [June 1, 2024, 1:54pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/13 "2024-06-01T13:54:57Z")
Do you really need memoryblocks to do whatever you are doing? Are you just appending strings? Sometimes users use shotguns to kill flies, that’s why I ask. ---
**Author:** ![Rick\_Araujo](https://forum.xojo.com/user_avatar/forum.xojo.com/rick_araujo/32/251_2.png) [@Rick\_Araujo](https://forum.xojo.com/u/Rick_Araujo)\ **Post date:** [June 1, 2024, 2:59pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/14 "2024-06-01T14:59:23Z")
Sometimes more direct things like these are enough: ```auto Var someDateTime As DateTime = DateTime.Now Var tf As TextOutputStream = TextOutputStream.Open(SpecialFolder.Desktop.Child("dates_test.txt")) For offsetDays As Integer = 1 to 10 tf.WriteLine someDateTime.AddInterval(0,0,offsetDays) _ // next day .ToString(new Locale("en-us"), DateTime.FormatStyles.Short) _ // m/d/y h:m:s am/pm .ReplaceAll(&u202f, " ") // user prefer normal space Next tf.Close Quit ``` ---
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [June 1, 2024, 10:59pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/15 "2024-06-01T22:59:42Z")
> [@Rick\_Araujo](#): > > Sometimes users use shotguns to kill flies, that’s why I ask. Ha ha. I’m saving dozen or even hundreds of “songs” which are MIDI performances containing possibly thousands of 3-byte MIDI events including timestamps. I found it to be way too slow to save it all writing to a stream so I build my file in a memory block. What took 25 seconds before now takes about 1 second. Thanks for your more direct example. I try to do that but often forget. ---
**Author:** ![Rick\_Araujo](https://forum.xojo.com/user_avatar/forum.xojo.com/rick_araujo/32/251_2.png) [@Rick\_Araujo](https://forum.xojo.com/u/Rick_Araujo)\ **Post date:** [June 1, 2024, 11:22pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/16 "2024-06-01T23:22:17Z")
> [@John\_Eidsvoog](#): > > [Binary data] I found it to be way too slow to save it all writing to a stream so I build my file in a memory block. 25x slower is weird, a BinaryStream is supposed to be fast and use buffers (a memoryblock) internally to speed things up, flushing its contents over time. Should not be so slow. Something seems not well tuned in the Xojo framework. ---
**Author:** ![John\_Eidsvoog](https://forum.xojo.com/user_avatar/forum.xojo.com/john_eidsvoog/32/1798_2.png) [@John\_Eidsvoog](https://forum.xojo.com/u/John_Eidsvoog)\ **Post date:** [June 2, 2024, 2:08pm UTC](https://forum.xojo.com/t/encoding-problem-with-sonoma/80510/17 "2024-06-02T14:08:10Z")
> [@Rick\_Araujo](#): > > 25x slower is weird, a BinaryStream is supposed to be fast and use buffers (a memoryblock) internally to speed things up, flushing its contents over time. Should not be so slow. Something seems not well tuned in the Xojo framework. Part of my optimization was achieved by using a memoryblock in each song to hold the MIDI event data instead of an array of thousands of MIDI event classes. When those events are needed for playback, the time to parse it into an array is negligible. That might explain much of the speed difference. The 25x difference was extreme when testing with hundreds of thousands of songs resulting in a file of several hundred MBs. To use it as an example may have been a bit dramatic. Maybe it took 2 seconds. 😃