Hashed password values saved with 2017r2 don't work when checked with 2018r11

General
1

So my software saves passwords as hash data values using Xojo.Crypto.PBKDF2. A password saved with the app built in 2017r2 cannot be matched using the same exact function when the app is built in 2018r11.

Here’s the ToHash code:

[code]Protected Function ToHash(password as text, withSalt as text) as Text

Dim passwordData As Xojo.Core.MemoryBlock
passwordData = Xojo.Core.TextEncoding.UTF8.ConvertTextToData(password)

Dim salt As Text

if withSalt = “” then
salt = RandoID() //makes random 6 letter string
else
salt = withSalt
end if

Dim saltData As Xojo.Core.MemoryBlock
saltData = Xojo.Core.TextEncoding.UTF8.ConvertTextToData(salt)

Dim combinedData As New Xojo.Core.MutableMemoryBlock(passwordData)
If Not salt.Empty Then
combinedData.Append(saltData)
End If

Dim hashData As Xojo.Core.MemoryBlock

hashData = Xojo.Crypto.PBKDF2(saltData, passwordData, 500, 32, Xojo.Crypto.HashAlgorithms.SHA256)

// Convert hashData to hex for display
Dim hex As Text

For b As Int8 = 0 To hashData.Size - 1
hex = hex + hashData.Int8Value(b).ToHex(2)
Next

Return salt+hex
End Function
[/code]

The salt is a random 6 letter code generated when it is saved and easily accessed when checking against the saved hash value. The value for the password “sample” with the salt “QNACSXR” will produce the following:

2017: QNACSXR2D20117D0522635C090E7905450F1C0C4709060C0B010119087601130E480D0F 2018: QNACSXR2D20117DE522635CE9BE79B5450F1C8C47B9D6CC9B010119E876C1130E489D8F

The values are very close, but obviously different, which means that using 2018r11 users couldn’t get into their saved files because the passwords don’t match. None of my code for this has changed and it has worked for quite some time now, so something in Xojo had to have changed that’s affecting this.

2

Hello Tom,

I had the same experience a few months ago. I also had to change my routine.
Not so much of a problem, because in my case it is for personal use only.

I vaguely remember that this issue had been entered in the forum earlier
already then.

Jan

3

Thanks Jan. My problem is that many people use the app and I can’t just ignore the stored passwords. How do I update to a newer Xojo version (as is very often necessary) without breaking all stored authentication information?

Also any chance you have a link to the other topic? I searched but it’s always so hard to find specific posts on forums.

4

I don’t have experience with hash, but I found one previous report:
https://forum.xojo.com/48196-pbkdf2-changes-from-2017r3-to-2018r1/0
don’t know if Thom’s answer can help you (I don’t understand it).

I found Paul’s blog entry:
https://blog.xojo.com/2015/10/09/tips-dealing-with-the-problem-of-passwords/
but I ran the sample project in 2017r3 and 2018 and all hash values are different (md5, sha1, sha256, sha512, PBKDF2).

It is strange that there is too little information about this. I hope you can find a way to make it work. Please post what you find here.

5

Thanks appreciate it. The second link is (I think) what I used when creating the original code. The first link is definitely encountering the same issue as me. The problem is that it seems like something changed in the bowels of Xojo that’s affecting these very important functions in minute ways.

I think there isn’t too much talk about it because that level of forced security with password storage isn’t that common, but I have had issues before where different Xojo or MBS versions have broken the encryption/etc with user stored data between versions when I haven’t changed anything within the code itself.

It’s sort of like if you try to do the right thing and be more secure you run into much more severe problems. Hopefully there will be some workaround or variable I can magically adjust in 2018rXX so that I can have the functions match up with the previous versions.

6

Have you tried 2018r2? What if you use the Crypto instead of Xojo.Crypto?

As an aside, you should be generating random salts as bytes using Crypto.GenerateRandomBytes, and they should be a minimum of 8 bytes. As it is, you seem to have limited it to 6 uppercase ASCII characters, or 26^6 (308,915,776) possible combinations. If you’re using Random to generate them, that makes it less secure. (An 8-byte salt made of bytes has 256^8, or 1.8E19, combinations.)

7

Yeah it’s actually different again in 2018r2 vs 2018r11 and 2017r2. And I do agree it could be more secure using those methods, but if you just build using a new Xojo version and can’t get in even with the correct password then it’s the most secure!

8

And what of using Crypto rather than Xojo.Crypto. Is that an option?

9

Could it be that this code is the problem?

For b As Int8 = 0 To mb.Size - 1 hex = hex + mb.Int8Value(b).ToHex(2) Next
Maybe there was a bug on previous versions of Xojo and it was fixed on latest release?

This after testing (putting Break) and checking hashData contents (same for 2017 and 2018):

This is the hash result with 2017:

This is the hash result with 2018:

Using Paul’s code from blog post. Default values an MD5.

10

Just visually looking at it it seems like those similar differences between the two versions for my data as well. The problem is what can you really do about it to fix it on my end if with live user data it was a bug that was fixed that affects all existing data?

I wrote a quick method to compare the two strings using mid() and figure out the percentage difference between the two. Seems like it’s only about 15-25% of chars that are different in this situation. So that could work but that would obviously be a security flaw to accept hash values that match at a 75% level just because the software was build wit a new version of Xojo.

11

I used this code to test in 2018r2 and could not detect a difference:

const kIterations as integer = 500
const kLength as integer = 32

for i as integer = 1 to 1000
  dim mb as MemoryBlock = Crypto.GenerateRandomBytes( 8 )
  dim p as ptr = mb
  dim nmb as new Xojo.Core.MemoryBlock( p, mb.Size )
  
  dim mbSalt as MemoryBlock = Crypto.GenerateRandomBytes( 6 )
  p = mbSalt
  dim nmbSalt as new Xojo.Core.MemoryBlock( p, mbSalt.Size )
  
  dim mbHash as MemoryBlock = Crypto.PBKDF2( mbSalt, mb, kIterations, kLength, Crypto.Algorithm.SHA256 )
  dim nmbHash as Xojo.Core.MemoryBlock = Xojo.Crypto.PBKDF2( nmbSalt, nmb, kIterations, kLength, Xojo.Crypto.HashAlgorithms.SHA256 )
  
  if not MBAreSame( mbHash, nmbHash ) then
    MsgBox "Different!"
    return
  end if
next

Private Function MBAreSame(mb As MemoryBlock, nmb As Xojo.Core.MemoryBlock) as Boolean
  if mb is nil and nmb is nil then
    return true
  end if
  
  if mb is nil or nmb is nil then
    return false
  end if
  
  if mb.Size <> nmb.Size then
    return false
  end if
  
  if mb.Size = 0 then
    return true
  end if
  
  dim lastBytePosition as integer = mb.Size - 1
  for pos as integer = 0 to lastBytePosition
    if mb.Byte( pos ) <> nmb.Data.Byte( pos ) then
      return false
    end if
  next
  
  return true
End Function
12

I see (my images posted):
AE with 2018, 0E with 2017
B0, 00
F8, 08
E0, 00
C3, 03
F8, 08
so it looks like high HEX values in 2017 removed the first letter and changed to 0.

I don’t know much about all this, but it looks like that was wrong and now it is correct.

The problem is that if you have the HEX hash with wrong values, I don’t think you can easily fix this. Unless you force your users to change the password and create a new correct hash. Maybe there are other options.

13

Two ideas come to mind:

Release a version built in 2017 first and force your users to upgrade to it. That version will use the old hex encoding to compare passwords, then write the new values with some version indicator so you know those have been fixed.

In addition (or instead), compare only every other character for the old hash, then write the correct value with a version indicator moving forward.

14

[quote=404850:@Kem Tekinay]Two ideas come to mind:

Release a version built in 2017 first and force your users to upgrade to it. That version will use the old hex encoding to compare passwords, then write the new values with some version indicator so you know those have been fixed.

In addition (or instead), compare only every other character for the old hash, then write the correct value with a version indicator moving forward.[/quote]

We actually already have a version indicator included in the stored data and this could possibly work. But I’m not sure about the security of finality of it. It would help to hear from Xojo if this was in fact a bug that has been fixed and won’t be changed again, or if it is something different.

15

Maybe try the MBS crypto classes?

16

If the hex encoding was wrong, and now it’s right, I think that answers both questions. :slight_smile:

As for the security of it, the stored hash is already wrong, and thus insecure, in the existing version, no?

17

Tom, it looks like if you change:

For b As Int8 = 0 To hashData.Size - 1 hex = hex + hashData.Int8Value(b).ToHex(2) Next
to:

For b As Int8 = 0 To hashData.Size - 1 hex = hex + hashData.UInt8Value(b).ToHex(2) Next
you will get the correct hash with 2017 (same as 2018).

Hope this help for something.

My guess is that the ToHex had a bug when the value to convert is negative:
-82 ToHex(2) = 0E with 2017 but really is AE (as you get with 2018) using UInt instead of -82 I get
174 ToHex(2) = AE with 2017 and that’s the correct value.

18

I was thinking, maybe you don’t need to ask the users to change the password, but you will need a new 2017 app, one that check the hash as normal and also create a new (correct) hash with UInt. Then change the old hash to the correct hash. When every hash is updated you can create a new 2018 app.

19

Problem: you went with int8 instead of uint8

In 2017r2 you had these results: -128 => “00”, -127 => “01”, -126 => “02”, …, -113 => “0F”, -112 => “00”, -111 => “01”, …
See a pattern?

This is a securityrisk since these are a LOT of collisions.

In 2018r1 this problem is fixed, but you get stuff like -127 => “FFFFFFFFFF01”, -111 => “FFFFFFFFFF11”, …
So here you see the differences, but its still way too large

The simple sollution: change int8 by uint8 and this fixes your code, but breaks every password.

The “autopatch” sollution: calculate both hashes and change the wrong hash in the right hash whenever it’s not fixed yet. (Save this for each record or so)

Private Sub CalcHash(strPass As Text, strSalt As Text, ByRef strHash As Text, ByRef strWrongHash As Text)
  Dim mbPasswordData As Xojo.Core.MemoryBlock
  Dim mbSaltData As Xojo.Core.MemoryBlock
  Dim mbCombinedData As Xojo.Core.MutableMemoryBlock
  Dim mbHashData As Xojo.Core.MemoryBlock
  Dim strHexRight As Text
  Dim strHexWrong As Text
  Dim t As Integer
  Dim iVal As Integer
  Dim ch As Text
  
  mbPasswordData = Xojo.Core.TextEncoding.UTF8.ConvertTextToData(strPass)
  mbCombinedData = New Xojo.Core.MutableMemoryBlock(mbPasswordData)
  
  If Not strSalt.Empty Then
    mbSaltData = Xojo.Core.TextEncoding.UTF8.ConvertTextToData(strSalt)
    mbCombinedData.Append(mbSaltData)
  End If
  
  mbHashData = Xojo.Crypto.PBKDF2(mbSaltData, mbPasswordData, 500, 32, Xojo.Crypto.HashAlgorithms.SHA256)
  
  For t = 0 To mbHashData.Size - 1
    iVal = mbHashData.Int8Value(t)
    If iVal < 0 Then
      iVal = ((128 + iVal) mod 16)
      ch = "0" + iVal.ToHex(2).Right(1)
    Else
      ch = iVal.ToHex(2)
    End If
    strHexWrong = strHexWrong + ch
    
    iVal = mbHashData.UInt8Value(t)
    ch = iVal.ToHex(2)
    strHexRight = strHexRight + ch
  Next t
  
  strHash = strSalt + strHexRight
  strWrongHash = strSalt + strHexWrong
End Sub
20

Thanks for all of the replies and help everyone. Especially Gino for going above and beyond and writing up all the code. Always great to have a community that can help out with obscure issues such as this!