XMLReader Parse XML File in Chunks

What’s the best way to break up a very large (~500 MB) file and send it to XMLReader.Parse?

Thank you

You shouldn’t need to
Note that Parse has 2 forms - one takes a string the other a folderitem

Yes, I did see that. However…
I’ve been successfully parsing XML files using:

[code] dim TIS_XML as TextInputStream = TextInputStream.Open(XML_RawFile)
dim XML_Code as String = TIS_XML.ReadAll

dim xdoc as XmlDocument = new XmlDocument

try
xdoc.LoadXml(XML_Code)
catch xe As XmlException
d.Message = “XML Exception.”
d.Explanation = xe.Message + chr(13) + chr(13)
b=d.ShowModal
return false
end try[/code]

But, when I try to load the 500 MB file, it crashes on the xdoc.LoadXml(XML_Code)

So I thought I needed to use XMLReader instead and feed it only chunks at a time?

Thank you

No need for you to chunk it - the reader will do that for itself
You just need to respond to the events

Thanks Norman. In the future, if I use LoadXML, is there any benefit to passing it a FolderItem instead of reading the xml file into a string and then passing LoadXML a string?

Use the folder item as you end up having fewer copies of the data in memory

BUT if you have no idea how large the file will be use the reader as it can handle pretty much any size of file

It’s more work for you but it’s less likely to have out of memory issues on really big files

Thanks Norman.

Does anyone have RealBasic Developer Magazine Jan/Feb 2006 Issue 4.3? There is an article called, Extending XML Reader: Applying the Chain of Responsibility Pattern by Charles Yeomans. I purchased the article and I believe there is a typo in the code every few lines. I was wondering if someone could confirm.

Thank you

Marc from XDev Magazine contacted me to help analyze the possible typos.

However I have another question regarding XMLReader. If I want to have a progress bar while my application parses the XML, is there any way to get a total count on the number of children in the document, so I can initialize my progress bar?

This was trivial using LoadXML, but I assume since XMLReader takes the data in chunks there is no way to do this?

Thank you.

[quote=46482:@Mark S]Marc from XDev Magazine contacted me to help analyze the possible typos.

However I have another question regarding XMLReader. If I want to have a progress bar while my application parses the XML, is there any way to get a total count on the number of children in the document, so I can initialize my progress bar?

This was trivial using LoadXML, but I assume since XMLReader takes the data in chunks there is no way to do this?

Thank you.[/quote]

No since it would have to load the entire document to know this and thats NOT how it works
Its why it can work for extremely large files

I have my parse method that uses XMLReader in it’s own thread and it’s still locking up the GUI. Is there any way around this?

Also, is there any way to cancel XMLReader.Parse midway through if , say one of my database insertions fail?

Thank you

You might want to make sure you have loops or YieldToNextThread calls in there
I’m not sure how much the XML reader itself Yields to other threads

And you can just stop reading with the reader at any point (How you do that with how your code is up to you)
No need to read tip the end

I’ll try adding some YieldToNexThread calls, thanks.

How exactly do you stop XMLReader though? I don’t see any methods in the language reference that would cancel a parse.

Thank you.

You COULD throw an exception & catch it

Thanks Norman.