# XOJO 2019R3 Encodings Issues

**URL:** <https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973>\
**Category:** Databases\
**Created:** [January 13, 2020, 3:18pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973 "2020-01-13T15:18:41Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 3:18pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/1 "2020-01-13T15:18:41Z")

</div>

Greetings,

No idea what happened honestly but since I started to port the project in R3 I have only issues and I start to feel sorry that I did that .

So I do a query on the MariaDB database , I get the data , I know that it is UTF8 , I open a rtf file and I write the data into it and when I open the document my data is messed up .

In the Database and in XOJO ide in Debug and everywhere I have [quote]Stphanie[/quote] in the document I have [quote]St?phanie[/quote] no matter what I do, I tried converting it to UTF8, ISOLatin1 interface is perfect , DB is perfect , when it gets written in the file it is messes up .

Any ideas what happened and where to look ? previous versions with identical code works perfect, R3 is messed up.

Is there a way to detect the actual encoding ? apparently the data that comes from the DB always has no encoding and I have to define it. Do I have to do same when I write it to the file ?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Emile\_Schwarz](https://forum.xojo.com/letter_avatar_proxy/v4/letter/e/90ced4/32.png) [@Emile\_Schwarz](https://forum.xojo.com/u/Emile_Schwarz)\
**Post date:** [January 13, 2020, 3:29pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/2 "2020-01-13T15:29:45Z")

</div>

did you define the encoding in the rtf file ?

Example (ansi):

```auto
{\\rtf1\\ansi\\ansicpg1252
```

---

<div class="post-metadata">

**Author:** ![Sascha\_S](https://forum.xojo.com/user_avatar/forum.xojo.com/sascha_s/32/22328_2.png) [@Sascha\_S](https://forum.xojo.com/u/Sascha_S)\
**Post date:** [January 13, 2020, 3:33pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/3 "2020-01-13T15:33:37Z")

</div>

R u using [TextOutputStream](http://documentation.xojo.com/api/files/textoutputstream.html)?

```auto
output.Write(ConvertEncoding(StringValue, Encodings.UTF8))
```

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 3:34pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/4 "2020-01-13T15:34:48Z")

</div>

[quote=471246:@Emile Schwarz]did you define the encoding in the rtf file ?

Example (ansi):

```auto
{\\rtf1\\ansi\\ansicpg1252
```

First line it has that so I believe that the encoding is there `{\\rtf1\\adeflang1025\\ansi\\ansicpg1252\\uc1`

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 3:35pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/5 "2020-01-13T15:35:37Z")

</div>

[quote=471247:@Sascha S]R u using [TextOutputStream](http://documentation.xojo.com/api/files/textoutputstream.html)?

```auto
output.Write(ConvertEncoding(StringValue, Encodings.UTF8))
```

I already tried that, same result , no change at all in the document. I"m starting to think that ConvertEncoding is broken or so .

---

<div class="post-metadata">

**Author:** ![Sascha\_S](https://forum.xojo.com/user_avatar/forum.xojo.com/sascha_s/32/22328_2.png) [@Sascha\_S](https://forum.xojo.com/u/Sascha_S)\
**Post date:** [January 13, 2020, 3:36pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/6 "2020-01-13T15:36:53Z")

</div>

Can you show us some code please? 🙂

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 3:58pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/7 "2020-01-13T15:58:34Z")

</div>

> [@471250:@Sascha S](#):
>
> Can you show us some code please? 🙂

while I cannot share all the code I extracted bits of it that are related to what I need with the needed comments on what I found .

[code]Var TIStream As TextInputStream  
Var t As TextOutputStream  
Var fContent As String  
Var rows As RowSet

If file \<\> Nil Then  
Try  
TIStream = TextInputStream.Open(file)  
TIStream.Encoding = encodings.windowsLatin1 ’ Imported from some WindowsGeneratedRTF File and always worked this way until now.  
courrierContent = TIStream.readAll

Catch e As IOException  
…  
End Try  
TIStream.close

End If

…

Try  
rows = app.sqlBase.SelectSQL(“SELECT \* FROM patients WHERE id\_patient=?”, superWin.patientID)

If rows \<\> Nil Then  
For Each row As DatabaseRow In rows

```
  Var lName As String
  Var fName As String
  
  ' For test purposes, apparently if i define ISOLatin1 the name it shows properly in XOJO Debug
  
  lName = row.Column("lName").StringValue.DefineEncoding(Encodings.ISOLatin1) 
  lName = lName.ConvertEncoding(Encodings.UTF8)
  
  fName = row.Column("fName").StringValue.DefineEncoding(Encodings.ISOLatin1)
  fName = fName.ConvertEncoding(Encodings.UTF8)
  
  fContent=fContent.ReplaceAll("<#PGender#>", row.Column("gender").StringValue.DefineEncoding(Encodings.UTF8))
  fContent=fContent.ReplaceAll("<#PlName#>", lName)
  fContent=fContent.ReplaceAll("<#PfName#>", fName)
  
  
Next

rows.Close

```

End If  
Catch e As DatabaseException  
…  
End Try

…

If file \<\> Nil Then  
Try  
file.Remove

```
Try
  t = TextOutputStream.Create(file)
  't.Write(fContent) ' Functional in R2
  t.Write(ConvertEncoding(fContent, Encodings.UTF8)) ' Same result no matter what i put here.
  t.Close
  
Catch e As IOException
  ....
  
  
End Try

```

Catch error As IOException  
…  
End Try

End If[/code]

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:08pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/8 "2020-01-13T16:08:01Z")

</div>

I did found this commented in the code, I guess it was replaced on the way but so far it is not related :

[code]'Dim t as TextOutputStream  
’  
'file.delete()  
''t = file.CreateTextFile()  
't = TextOutputStream.Create(file)  
’  
't.Write ConvertEncoding(fContent, Encodings.macRoman)  
't.close()  
’

'This part we had to remove as it was not working anymore and i did not found any replacement for it .  
'file.macCreator = “MSWD”  
'file.macType = "RTF "  
[/code]

So far for me it seems that ConvertEncoding is not working at all, no matter what I put there I get always same result, or at least this is how it looks.

---

<div class="post-metadata">

**Author:** ![MarkusR](https://forum.xojo.com/user_avatar/forum.xojo.com/markusr/32/685_2.png) [@MarkusR](https://forum.xojo.com/u/MarkusR)\
**Post date:** [January 13, 2020, 4:10pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/9 "2020-01-13T16:10:01Z")

</div>

> [@471248:@Aurelian Negrea](#):
>
> First line it has that so I believe that the encoding is there {\rtf1\adeflang1025\ansi\ansicpg1252\uc1

seems you put utf8 in a document with ansi / codepage windows 1252.  
you can convert your string into ansi or you need a utf8 template file.

---

<div class="post-metadata">

**Author:** ![Sascha\_S](https://forum.xojo.com/user_avatar/forum.xojo.com/sascha_s/32/22328_2.png) [@Sascha\_S](https://forum.xojo.com/u/Sascha_S)\
**Post date:** [January 13, 2020, 4:12pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/10 "2020-01-13T16:12:21Z")

</div>

I can’t see an issue in your code.

But i see you are reading data from the database as ISOLatin1 and UTF8. Then you combine them using replace statements.

What happens if you convert f+lName to UTF8 before you combine them in an UTF8 string?

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:12pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/11 "2020-01-13T16:12:40Z")

</div>

> [@471256:@Markus Rauch](#):
>
> TIStream.Encoding = encodings.windowsLatin1

Well I believe that that was the [quote]TIStream.Encoding = encodings.windowsLatin1[/quote] part for if I’m not mistaking and it was always working until R3 so either something was broken before or something is broken now. I’ll try to do a fresh document and try again to see what result I get.

---

<div class="post-metadata">

**Author:** ![Michael\_Hußmann](https://forum.xojo.com/user_avatar/forum.xojo.com/michael_hu%C3%9Fmann/32/438_2.png) [@Michael\_Hußmann](https://forum.xojo.com/u/Michael_Hu%C3%9Fmann)\
**Post date:** [January 13, 2020, 4:17pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/12 "2020-01-13T16:17:17Z")

</div>

> [@471255:@Aurelian Negrea](#):
>
> So far for me it seems that ConvertEncoding is not working at all, no matter what I put there I get always same result, or at least this is how it looks.

This is usually due to the string’s encoding not being properly defined. ConvertEncoding can only convert from some known encoding into another. The general rules are actually quite simple: Whenever you fetch some text (like from a database) you define its encoding. Whenever you export some text you convert its encoding (if necessary) to whatever encoding is expected. Everything else is taken care of automagically.

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:19pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/13 "2020-01-13T16:19:13Z")

</div>

[quote=471258:@Sascha S]I can’t see an issue in your code.

But i see you are reading data from the database as ISOLatin1 and UTF8. Then you combine them using replace statements.

What happens if you convert f+lName to UTF8 before you combine them in an UTF8 string?[/quote]  
Well apparently the data supposed to be UTF8 and it was always like that but I did put Latin1 to see if maybe that is the issue as we had that in the past when importing from another app and when I did define the encoding as Latin1 I did not get the weird characters in the IDE Debug part while if I do replace the ISOLatin1 with UTF8 I get [quote]St?hanie[/quote] in the debug window so that seems that some data is Latin1 as I might suspected and only when we get it from another app, but still ConvertEncdoding supposed to Convert it properly once the data was correct or at least supposed to look correct . So my understanding was I get the data Latin1 I know it is Latin1, I convert it to UTF8 and I work with it, maybe I’m doing it wrong in the process. While Define works well convert does not seem to have any effect .

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:21pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/14 "2020-01-13T16:21:40Z")

</div>

> [@471260:@Michael Hußmann](#):
>
> This is usually due to the string’s encoding not being properly defined. ConvertEncoding can only convert from some known encoding into another. The general rules are actually quite simple: Whenever you fetch some text (like from a database) you define its encoding. Whenever you export some text you convert its encoding (if necessary) to whatever encoding is expected. Everything else is taken care of automagically.

Well that was the purpose of the tests that I did earlier, as I mentioned in the previous post, some fields were ISOLatin1 so I define them that way, it was showing well in the interface and debug, then converted them in UTF8 and it should do the job but apparently it does not, no idea honestly where else to look and I checked all the code and all the documentation so apparently all is ok but still does not work.

---

<div class="post-metadata">

**Author:** ![Sascha\_S](https://forum.xojo.com/user_avatar/forum.xojo.com/sascha_s/32/22328_2.png) [@Sascha\_S](https://forum.xojo.com/u/Sascha_S)\
**Post date:** [January 13, 2020, 4:24pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/15 "2020-01-13T16:24:30Z")

</div>

Can you please try the following?

[quote] lName = row.Column(“lName”).StringValue.DefineEncoding(Encodings.ISOLatin1)  
lName = lName.ConvertEncoding(Encodings.UTF8)

```
  fName = row.Column("fName").StringValue.DefineEncoding(Encodings.ISOLatin1)
  fName = fName.ConvertEncoding(Encodings.UTF8)
  
  fContent=fContent.ReplaceAll("<#PGender#>", row.Column("gender").StringValue.DefineEncoding(Encodings.UTF8))
  fContent=fContent.ReplaceAll("<#PlName#>", lName.ConvertEncoding(Encodings.UTF8))
  fContent=fContent.ReplaceAll("<#PfName#>", fName.ConvertEncoding(Encodings.UTF8))[/quote]
```

---

<div class="post-metadata">

**Author:** ![Sascha\_S](https://forum.xojo.com/user_avatar/forum.xojo.com/sascha_s/32/22328_2.png) [@Sascha\_S](https://forum.xojo.com/u/Sascha_S)\
**Post date:** [January 13, 2020, 4:26pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/16 "2020-01-13T16:26:28Z")

</div>

> [@471261:@Aurelian Negrea](#):
>
> so that seems that some data is Latin1 as I might suspected and only when we get it from another app

This “other App” is writing ISOLatin1 (or similar) into a UTF8 defined db field. Which is totally possible with SQL.

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:27pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/17 "2020-01-13T16:27:27Z")

</div>

> [@471264:@Sascha S](#):
>
> This “other App” is writing ISOLatin1 (or similar) into a UTF8 defined db field. Which is totally possible with SQL.

Apparently yes and it was supposed to be fixed to be all UTF8 so I’ll have to check to that side as well .

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:33pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/19 "2020-01-13T16:33:51Z")

</div>

[quote=471263:@Sascha S]fContent=fContent.ReplaceAll("\<#PlName#\>", lName.ConvertEncoding(Encodings.UTF8))  
fContent=fContent.ReplaceAll("\<#PfName#\>", fName.ConvertEncoding(Encodings.UTF8))[/quote]

When I do that , the whole word disappears, so nothing is showing anymore in the document the double converting part.

---

<div class="post-metadata">

**Author:** ![Sascha\_S](https://forum.xojo.com/user_avatar/forum.xojo.com/sascha_s/32/22328_2.png) [@Sascha\_S](https://forum.xojo.com/u/Sascha_S)\
**Post date:** [January 13, 2020, 4:43pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/20 "2020-01-13T16:43:08Z")

</div>

I guess fContent goes into an unknown encoding state if you combine it with different encodings.

---

<div class="post-metadata">

**Author:** ![Aurelian\_N](https://forum.xojo.com/letter_avatar_proxy/v4/letter/a/b4bc9f/32.png) [@Aurelian\_N](https://forum.xojo.com/u/Aurelian_N)\
**Post date:** [January 13, 2020, 4:46pm UTC](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973/22 "2020-01-13T16:46:46Z")

</div>

> [@471270:@Sascha S](#):
>
> {\rtf1\adeflang1025\ansi\ansicpg1252\uc1

Well I did tried

`TIStream.Encoding = encodings.windowsLatin1 ' Imported from some WindowsGeneratedRTF File and always worked this way until now.
    fContent = TIStream.readAll.ConvertEncoding(Encodings.UTF8)` but with the same result.

[Next page](https://forum.xojo.com/t/xojo-2019r3-encodings-issues/51973.md?page=2)