As I mentioned awhile ago I thought using a memoryblock to Serialize a recordset for transfer to ca lient app Xojo would be smaller and faster than using JSON… though the data transfer would be binary rather than Base64 encoded, which is not an issue for what I am doing.
I started working on doing just that…and it look like I was both right and wrong.
I think I have the serialization part done.(won’t know for sure until the deserialization part is finished and I get the data back out)
It is all pure Xojo code with no plugins… I wanted to see if what i was doing was worthwhile, so I compared the Serialization time and string size of my memoryblock code vs JsonItem, JsonItem_MTC and Xojo.Data.GenerateJson ( Doing implicit conversions both ways between string and text).
As I have never used JSON, I essentially used the Luna Express RecordSetToJSON method for JSONItem And JASONItem_MTC, and based the Xojo.Data version on the language reference code…
These may be naive implementations of using JSON to serialize a record set, and someone knowledgeable might be able to create code to get smaller strings and/or faster performance … but it gave me a baseline to compare against.
The RecordSet I used for testing was based on an SQLite View. It had 37 fields and 20,000 records and records averaged (as binary) about 350 bytes - I did not look at individual JSON record sizes.
The data shown below is from running in the IDE… When compiled, the MemoryBlock code runs 2.5-3X faster. I did not test the other methods compiled
While this data only applies to this recordset as different record composition/field data sizes will obviously matter, I found it encouraging!
Method StringSize (MB) Time Sec GZipedSize (MB)
MemoryBlock 7.18 1.85 1.44
JSONItem 14.95 124.67 1.55
JSONItem_MTC 14.95 12.42 1.55
Xojo.Data 14.95 1.67 1.58
So uncompressed the memoryblock string is about half the size of the JSON string, but compressed there is not much difference in size
Speed wise the pure Xojo memoryblock code is much faster than either JSONItem method. The new framework is sightly faster… but my memoryblock code lets me add more features and I expect deserialization might faster than JSON as well.
Here is the JSONItem code I used
[code]Dim RecordsJSON As New JSONItem_MTC
Dim Startms as Double = Microseconds
// Loop over each record…
While Not Records.EOF
Dim RecordJSON As New JSONItem_MTC
// Loop over each column…
For i As Integer = 0 To Records.FieldCount-1
// Add a name / value pair to the JSON record.
RecordJSON.Value( Records.IdxField(i+1).Name ) = Records.IdxField(i+1).StringValue
Next
// Add the JSON record to the JSON records object.
RecordsJSON.Append(RecordJSON)
// Go to the next row.
Records.MoveNext
Wend
Dim StrVal As String = RecordsJSON.ToString
Dim Duration as Double = Microseconds - Startms
Dim Size As UInt32 =StrVal.LenB
Dim ZipedSz As UInt32 = GZip(StrVal).LenB
Dim ZDuration as Double = Microseconds - Startms
// Close the recordset.
If Close Then
Records.Close
End If
Dim C as New Clipboard
C.text = Str(Size) + Chr(9)+ Str(Duration) + Chr(9) + Str(ZipedSz)+ Chr(9) + Str(ZDuration)
C.Close[/code]
This is teh Xojo.Data.GenerateJSON codeL
[code]Dim dictArray() As Xojo.Core.Dictionary
Dim Startms as Double = Microseconds
// Loop over each record…
While Not Records.EOF
Dim d As New Xojo.Core.Dictionary
// Loop over each column…
For i As Integer = 0 To Records.FieldCount-1
// Add a name / value pair to the JSON record.
d.Value( Records.IdxField(i+1).Name ) = Records.IdxField(i+1).StringValue
Next
// Add the JSON record to the JSON records object.
dictArray.Append(d)
// Go to the next row.
Records.MoveNext
Wend
Dim StrVal As String = Xojo.Data.GenerateJSON(dictArray)
Dim Duration as Double = Microseconds - Startms
Dim Size As UInt32 =StrVal.LenB
Dim ZipedSz As UInt32 = GZip(StrVal).LenB
Dim ZDuration as Double = Microseconds - Startms
// Close the recordset.
If Close Then
Records.Close
End If
Dim C as New Clipboard
C.text = Str(Size) + Chr(9)+ Str(Duration) + Chr(9) + Str(ZipedSz)+ Chr(9) + Str(ZDuration)
C.Close[/code]