Failure creating JSONItem

I’ve been trying to figure out why my program would crash after retrieving a large amount of data from a website. I finally tracked it down to creating a JSONItem from it. I’ve filed a bug report and supplied a test app.

feedback://showreport?report_id=54695

I’m currently using 2018r4 on Windows 10.

(For what it’s worth, the data received is a bit over 1 million characters.)

Have you tried my JSONItem_MTC, a drop-in replacement for JSONItem? It solves some problems that still exist in JSONItem and might do the trick for you in the short term. (It helps that it’s also faster than JSONItem.)

https://github.com/ktekinay/JSONItem_MTC

Note the project also includes functions to convert JSON to/from Dictionary/Variant array.

[quote=421742:@Kem Tekinay]Have you tried my JSONItem_MTC, a drop-in replacement for JSONItem? It solves some problems that still exist in JSONItem and might do the trick for you in the short term. (It helps that it’s also faster than JSONItem.)

https://github.com/ktekinay/JSONItem_MTC

Note the project also includes functions to convert JSON to/from Dictionary/Variant array.[/quote]
Actually, Kem, that’s on my agenda for this afternoon. I have it and have used it in a different project. The only “complaint” I have with it is that in the debugger, the dictionary keys show as hex strings rather than words. Not serious but sometimes annoying.

Necessary since JSON keys are case-sensitive, so that’s the only good way to do that. But I thought I had added some debugging property to list the keys? (I might be thinking of something else.)

Hi again, Kem. I just dropped the module into the test project and it seems to work properly in creating the JSONItem_MTC from the data.

@Kem Tekinay — I respectfully disagree with you. I think a better way to implement case-sensitive keys would be to generate your own hash table and override some functions so the user can have the right (unchanged) keys and the right behavior at the same time. Though it may be nitpicking :slight_smile:

Interesting. Potential performance issues aside, how would that work if the JSON object has, for example, the keys “my key” and “MY KEY”?

#Kem Tekinay — With the Xojo’s standard Variant.Hash function, both would have the same hash. But if you use your own case-sensitive hash function, the 2 would have different hashes. Dictionaries don’t compare keys but their hash, for speed.

If you are interested, you should have a look at Jenkins hash function or, more generally, to non-cryptographic hash functions

I understand how hashes work. :slight_smile: My question is, what would you end up storing as a value in the Dictionary to alleviate the problem of not being able to see the original keys? How would you link the key to its hash or vice-versa?

I just looked at the project, and I misremembered somewhat. I did add a DebugKeysMap property to JSONDictionary, but that is only used by M_JSON.ParseJSON_MTC, and has nothing to do with JSONItem_MTC.

I suppose I could change the key encoding so it tacks the hex encoding onto the original key, something like “my key-ABCDEF” (not a real encoding), but I’d have to check the performance hit of that.

I could also do something like this: “mykey” stays “mykey” while “myKey” becomes “my Key” and “my key” turns into “my key” (one space turns into two) and “my Key” becomes “my Key” (each space turns into two, each capital letter gets a space before it). Something like “MYKEY” would become " M Y K E Y".

Again, I’d have to see what the performance hit of something like that might be.

I ran some some tests and if I convert the name to a Dictionary key like this:

key = name + "-" + EncodeHex( name )

I don’t see an impact on performance and, if you examine the Dictionary in the debugger, you will see keys like “A-41”, “a-61”, “myKey-6D794B6579”. This will keep it unique for case and make it easier to debug, yes?

[quote=421855:@Kem Tekinay]I ran some some tests and if I convert the name to a Dictionary key like this:

key = name + "-" + EncodeHex( name )

I don’t see an impact on performance and, if you examine the Dictionary in the debugger, you will see keys like “A-41”, “a-61”, “myKey-6D794B6579”. This will keep it unique for case and make it easier to debug, yes?[/quote]
What effect will that have on methods like HasKey()? If I currently have a Dictionary that has a key ‘696E666F’ (‘info’) I can have a statement like

If myDictionary.HasKey("info") then ...
Will my statement as written fail? Will I have to revise it? Or will the new key format be only for the debugger?

You won’t see a difference until to try to manually examine the object Dictionary in the debugger.

Sounds good to me!

@Kem Tekinay — Sounds good

Updated, see https://forum.xojo.com/conversation/post/422355 and continue any conversation about JSONItem_MTC in that thread please.

This is really fantastic Kem! I had the same problem and you solved it!
Thank you for sharing your hard work!

Tim