MongoDB Xojo Driver (maybe)

Databases
21

Hello, Alwyn!

I think I have solved my issue from above. I added some code to your BSONSerializer.encodePair method to format an ObjectId. I assume that if the pairName is “_id” and the pairValue is a 24 character hex sequence, then it’s an ObjectId. I’ve included the code I added to the encodePair method as well as two new private methods for the BSONSerializer. These are formatObjectId and isObjectId.

If these changes pass your testing and I’ve implemented them properly, please feel free to add it to your driver. They work for me in my application.

Have a great day!

BSONSerializer.encodePair:

case 34, 39 ’ " OR ’ = string

pairValue = encExtractString(jsonMB, pos, jsonMB.Byte(pos))

  • if pairName = “_id” and isObjectId(pairValue) then

  •  bson.Append Chr(7)

  •  bson.Append pairName

  •  bson.Append Chr(0)

  •  bson.Append formatObjectId(pairValue)

  • else

    bson.Append Chr(2)
    bson.Append pairName
    bson.Append Chr(0)
    bson.Append formatInt32(Len(pairValue) + 1)
    bson.Append pairValue
    bson.Append Chr(0)

  • end if

BSONSerializer.formatObjectId:

dim tmpMB As new MemoryBlock(12)
dim i, j As Integer

j = 1

for i = 0 to 11
tmpMB.Byte(i) = val("&h" + s.Mid(j,2))
j = j + 2
next

Return tmpMB

BSONSerializer.isObjectId:

dim i, c, l As Integer

l = s.LenB

if l <> 24 then Return False

for i = 1 to l
c = s.Mid(i, 1).Uppercase.Asc
if not (( c >= 48 and c <= 57 ) or ( c >= 65 and c <= 70)) then Return False
next

Return True

22

Thanks for sharing Mark… much appreciated.

I plan to continue work on the MongoDB driver next week, and I’ll definitely patch the driver with your fix.

23

I had to give some attention to other projects the past week, but my focus is now back at the MongoDB driver.

Your fix works perfectly Mark, and I’ve updated the driver on GitHub with the fix. I plan to do a new release as soon as possible so that everyone else may also benefit from the changes. A big thanks for passing through the information and code, for it turned out to be a major issue.

Patrick also supplied a new test app for the driver that was comitted this morning. The test app is a great place to start if you want to familiarize yourself with how to use the MongoDB driver in your own Xojo applications.

24

Great job, Alwyn.

I can insert a document fine, but not sure of the best way to get the object ID of the document I just inserted?

25

Thanks Luis.

One way might be to query the document directly after the insert and then read the _id value… for example:

db.data.insert({somedoc:1})

and then

doc = db.data.find({somedoc:1}}

You can then read the doc._id value to get the ID. The driver doesn’t generate the ID and relies on the MongoDB server to generate a unique ID.

26

Thanks, that looks similair to what I’m already doing.

cursor = collection.find(myDoc.ToString) oid = "" while cursor.hasNext dim jsonRecord as new JsonItem(cursor.getNext()) oid = jsonRecord.Value("_id") wend

It seems to work fine, but like I said, I’m not exactly if this is the best way.

Also, I read in the mongodb documents that the driver sometimes supplies the _id, if not, mongodb will add it to the document, which is what I’ve been doing.

I’m just not sure if I should just generate my own id, but I’m thinking this just might add unnecessary complexity (i guess maybe I could make an MD5 hash?). On the other hand, I’m not liking the idea of searching for the record after insert (what if my app gets busy?).

27

I have to concur that searching after the insert is probably not the best way to do it. (E.g. what happens when there are duplicate documents).

Generating your own id is probably also not the best way. If you have different connections inserting documents at the same time, how do you prevent duplicate id keys? Probably best that the server generate the id to ensure uniqueness.

Let me get back to you on this one Luis… I’m not sure myself how one would reliable retrieve the id after insert other than selecting the document.

28

Not a solution to our current problem of obtaining the last inserted id, but this article gives good motivation why one should let the MongoDB server create the ID, instead of creating it oneself…

http://www.javacodegeeks.com/2013/06/mongodb-primary-keys-are-your-friend.html

29

Note - non-DBA logic that is probably too simplistic follows WRT getting last _id field

What if you return the _id value after the insert?

30

great article about learning MongoDB

http://www.rackspace.com/blog/how-i-learned-mongodb/

31

[quote=86847:@Tim Jones]Note - non-DBA logic that is probably too simplistic follows WRT getting last _id field

What if you return the _id value after the insert?[/quote]

I thought about this, but such an implementation would be inconsistent with the documentation provide by MongoDB.

Nowhere in the documentation for the db.collection.insert are there provisions made to return the with the insert. I suspect that drivers that do return the ID after an insert probably made their own customizations to their drivers?

Also, I cannot find any information on how to retrieve the id other than with the method that Luis illustrated with his code. For all technical purposes the MongoDB driver is simply a CRUD component, that doesn’t get involved with the “logic” of the data flow. I haven’t been able to find an built-in instruction for MongoDB to retrieve the last insert id with, like you might do with a MySQL database.

Given this, I’m starting to wonder if one should be concerned with ID’s at all, while working with MongoDB. From my limited experience with MongoDB, it seems that MongoDB is best suited for non-normalized data. IDs usually plays a more important role in normalized data system (e.g. relational databases)… perhaps using NoSQL requires a different approach to data handling?

32

Like you said, Alwyn, it seems like they don’t care about the id too much. I read that page before posting here. It suggests that some drivers come up with the id, and some don’t. But it doesn’t really mention any requirement to return the id on insert.

I guess my best bet here is to generate it client side. This is not a bad thing for me, because I want to limit their ability to create documents.

i.e. I can create an _id of joe-document1 which is ideal for me, because user Joe should only be able to insert one document. I can also just see first if there is a document with an id of joe-document1 and present options for him to edit it if necessary. I’m also letting these documents expire. This seems like the best path for my needs.

33

It definitely sounds like a good way forward for your scenario then.

I’m still in two minds whether the driver should create the _id’s and return them with the insert (like some other drivers), or whether this choice should be left to the user (e.g. give the user the choice to exclude an _id tag and let the server generate the _id, or self-generate the _id and include it with the insert, like you’re planning to do).

The advantage of the first option is obviously convenience… but I’m leaning towards option 2 though. Let’s see, if more users request that the driver should generate the _id with good motivation why, I might look into adding such ability.

34

This.

35

Alwyn, I’m not sure if this is a bug or me just not knowing how to upsert, but if I do this:

  collection.update("{'_id': "+ myExistingDocumentID +"}", "{ $set: {'_id': "+ myExistingDocumentID +", 'someNewField':  'TRUE' }", TRUE, FALSE)

I get these errors:
{“err”:“Mod on _id not allowed”,“code”:10148,“n”:0,“connectionId”:872,“ok”:1}
Mod on _id not allowed

But if I remove the id from the document, it’s inserted with an ID of 0.

I see somebody doing it this way here , so let me know if I need to report this as a bug.

EDIT: I had to quote the _id value. It was being cast as integer 0. By quoting the _id value, it was cast as a string:
collection.update("{’_id’: ‘"+ myExistingDocumentID +"’}", “{ $set: {’_id’: “+ myExistingDocumentID +”, ‘someNewField’: ‘TRUE’ }”, TRUE, FALSE)

36

Looks like you found the solution while I was sleeping… :wink:

37

I can’t figuire out how to sort records I find. I’m trying to display the last record inserted, but it’s pulling up the 1st record inserted, which obviously will never change…

Right now, I’m just doing collection.findOne(document) because I don’t see a way to chain the sort command…

38

The driver does not support chaining of commands.

Your best option will probably be to sort the results using the $orderby command. Have a look at this link:

http://docs.mongodb.org/manual/reference/operator/meta/orderby/

and in particular…

db.collection.find( { $query: {}, $orderby: { age : -1 } } )

Maybe it is possible to use this technique with the findOne method to return the correct document?

39

Thanks. Any plan to support GridFS… looks complicated. Sounds like I can store binary data up to 16 MB at least even without GridFS.

I personally think this driver is one the best things to happen to Xojo, but people just haven’t picked up on it yet.

40

I’m glad to hear you are finding the driver useful. There are a few people I know of that are using the driver extensively.

I suspect MongoDB is still a relatively new data technology when compared to mature relational databases that’s been around for decades. MongoDB definitely provides a different approach to big data management.

The plan is to continually grow the driver with time, and I trust I’ll get to GridFS at some point. I’m working on the driver in my spare time, and had to place developments a bit on hold due to the huge slice of time that a new born baby commands.

When I pick up development again I’ll look into features such as GridFS. The driver should be complete enough for most data needs at this point, but I’ve created a feature request for GridFS in the meantime though.

https://github.com/alwyn1024/mongodb-xojodriver/issues/39