File Writing Efficiency

I’m working on writing code for exporting large amounts of delimited data. As I’m working on it, an optimization question occurred to me.

Presently I’m storing the contents of each line to a FileData() As String variable. At the end I take the OS line ending the file is intended for (user selects), join the array, and use the return to write the whole thing in bulk to the file. All done.

Normally I’d write to the file line by line via a TextOutputStream on desktop, but for WE I only have the WebFile.Data property to work with. I suppose I could write a line as:

Dim File as New WebFile File.Data = "Some content" + EndOfLine.Windows File.Data = File.Data + "More Content" + EndOfLine.Windows // etc...

Don’t take that code too literally; I’m just looking appending to the property. This still means the contents will be in memory, but at least not redundantly. I’m still trying to wrap my mind around the best way to do things in WE.

Thoughts? :slight_smile:

Maybe I missed something. Why can’t you do:

File.Data = join( FileData, EndOfLine.Windows )

Hi Kem,

Sorry if it wasn’t clear in my second paragraph, but that’s exactly what I’m currently doing and precisely what I’m questioning. :wink:

My file may be 20-100 mb in size (a lot of data). If I use that line of code, then for a moment I effectively have twice that much use of memory while the method is executing.

I’m probably making a mountain out of a mole hill here, but in WE I’m trying to be especially mindful of memory use as a single server is supporting multiple sessions.

Join is a 1000 times faster on large data sets compared to plus operator.
The memory for the temp buffer of join and your array are probably released quickly after that code finished, so no big memory problem.

I understand now, and agree with Christian that 20-100 MB is probably not a big deal, and that probably the fastest way to do it.

But if you’re really concerned, write the contents to a file, then use WebFile.Open to assigned that file to a WebFile instance. If you use tos.WriteLine instead of the array, you will never have a duplicate of data in RAM, at least not one that you can control (there may be some caching involved). But this might end up to be substantially slower, so you should test.

That’s what I was thinking/hoping. Thanks, Christian! :smiley: