File ready

Hello,

I have a ‘watch’ directory and if files show up in there, my App goes to work.

Things work fine except when files are not ‘ready’.
Somehow I need to wait with processing because files can be copied (or downloaded) into the directory. The files can be rather large and it is important that the file is ‘ready’ to be processed. (I need a complete file because I need the file size and a hash)

Is there a way to determine if a file is done copying? For example in Finder, the file is grey and cannot be clicked until the copy is done.
Except folderitem.length (filesize), things like locked, iswritable/readable lasterror fileinuse all give the same result when the copy is in progress and when the copy is done.

Instead of processing any file immediately, add it to a queue and then start a 5 second (or whatever) timer. If the file length is the same after 5 seconds, it’s probably ready for processing.

Beyond that, I think you’d have to get into declares or plugins to see if the file is “in use”.

Do you have control of the file delivery?
In a similar situation, I once had an arrangement where the main file was accompanied by a control file.
The control file was written after the main file, and contained a row count among other things.

In your situation, the second file would need to exist before you consider opening the first, even if you dont care whats in the second.
Then when you are done, delete them both?

To expand on Jeff’s point, if you have that control, you can also copy to a different directory, then move the copied file into the scanned directory.

No, unfortunately not.
The files can be copied in, saved from another program, downloaded etc. but I have no control over the delivery.

I’d hoped that things like opening or renaming the file would raise an error (LastErrorCode) but it seems that everything is allowed in the middle of a copy operation.
Tomorrow I will try if I can move the file back and forth to some temp directory to see if that raises an error. If not, I’m afraid that I have to go for the timer solution checking the file size.

…or change the file extension until it’s done. That’s how Safari does downloads.

Try to open a binaryStream. You will get an error if the file is locked.

Oh, that should work, as long as you try to create the BinaryStream with write privileges.

No, nothing :frowning:
I can even normally read from it while it’s copying.
Somehow the file isn’t locked although Finder shows the file ‘grey’ and the filesize shows the progressbar.
Maybe I’m missing something. I’m copying a 1GB file from one folder to the ‘Watch’ folder.
During the copy process, I try this:

Try Dim bs As BinaryStream = BinaryStream.Open(FolderItem.Item(i), True) TextArea1.Text = bs.Read(1024) Catch e As IOException msgbox("Catch") End Try
…but no error.

Even moving the file while it’s being copied doesn’t give an error.
The file is moved and the copy just finishes at the new location.

[quote=211409:@Marco Hof]No, nothing :frowning:
I can even normally read from it while it’s copying.
Somehow the file isn’t locked although Finder shows the file ‘grey’ and the filesize shows the progressbar.
Maybe I’m missing something. I’m copying a 1GB file from one folder to the ‘Watch’ folder.
During the copy process, I try this:

Try
    Dim bs As BinaryStream = BinaryStream.Open(FolderItem.Item(i), True)
    TextArea1.Text = bs.Read(1024)
Catch e As IOException
    msgbox("Catch")
End Try[/code]
...but no error.[/quote]

I wanted to get the length of the file, but it appears simply trying to open the binarystream while the file is written does trigger an error 21.

[code]  dim f as folderitem = specialfolder.desktop.child("LinuxMint.vmwarevm")
   Dim bs As BinaryStream = BinaryStream.Open(f, False) // Open as read-only
  system.DebugLog str(bs.Length)

“LinuxMint.vmwarevm” is a VM I copy to the desktop to test.

Michel, how?? That’s what I’m looking for.
When do that, I just see the length of the file growing in the debuglog every time I click the button.

In the buttons action:

Dim f As folderitem = specialfolder.desktop.child("1.xyz") Dim bs As BinaryStream = BinaryStream.Open(f, False) system.DebugLog str(bs.Length)

Then I copy 1.xyz to the desktop and start clicking the button.

6:42:44 PM My Application Launched 6:42:45 PM 713031680 6:43:25 PM 125829120 6:43:26 PM 125829120 159383552 6:43:27 PM 159383552 192937984 6:43:28 PM 226492416 6:43:29 PM 226492416 293601280 6:43:31 PM 360710144 6:43:33 PM 427819008 6:43:34 PM 427819008 494927872 6:43:36 PM 562036736 6:43:37 PM 629145600 6:43:38 PM 629145600 6:43:39 PM 696254464 6:43:40 PM 763363328 6:43:41 PM 763363328 6:43:42 PM 763363328 830472192 6:43:44 PM 897581056 6:43:45 PM 909464014 6:43:57 PM My Application Ended

so confused.

Try seeing if you can open it for READ AND WRITE
I think that should fail if the file is still uploading (it may not - I havent dived into the C++ code to see if we do open it exclusive when you do that)

No Norman, it doesn’t. :confused:
See above where I tried that as well. (BinaryStream.Open(FolderItem.Item(i), True))
I just tried it again but it normally opens. No error and I can read the length. (Increasing while it’s busy being copied)

Whatever I try, I can pretty much do everything without raising an error. LastErrorCode is always 0.

There are 2 interesting things I found out.

  • If I move the file to another Folder during the copy, the file is moved and the copy continues at the new location.
  • If I Lock the file during the copy, no error is raised and it seems that nothing happens. However, when the copy is done, the file disappears.

I’m still trying to figure this out somehow because using a timer and checking the filesize isn’t very reliable. As you can see in my previous reply, the filesize stays the same for multiple seconds because of the buffering. Now that’s on the same disk. No idea how bad it is with network drives or downloads.

What OS ?
OS X has FSEvents that might be of use as I believe they are raised at appropriate times (like when a file copy completes if thats what you watch for)
Not 100% of what Windows & Linux may have

I think MacOSLIb has FSEvents code in it

[quote=211455:@Marco Hof]Michel, how?? That’s what I’m looking for.
When do that, I just see the length of the file growing in the debuglog every time I click the button.

[/quote]

I can’t explain it, but it works that way here. It ay be possible that copying with the finder locks the file, whereas the program that creates the files on your end does not.

If you can read the length growing, then I would use a couple timers :

  • Multiple Timer1 checks the length of the file at regular intervals, for instance every tick (17 ms)
  • In Action, Timer1 First sets Timer2 to ModeOff, then if Length has changed, sets Timer2 to 30 ms, ModeSingle.
  • When the file is finished copying, Timer2 Action fires. In there, turn off both timers and do whatever is needed.

Just a thought…

If no error condition is occurring then perhaps it might be an idea to read for the expected EOF marker contained in the interested file. Accepted the stream itself EOF boolean might very well not give a reliable answer since it may have the situation of buffer filling up and this leaves the stream position equal to the so far copied file length. However the content of the stream would not have an internal EOF marker until the file has indeed completed being written.

I came up with an interesting method to handle this exact issue with osx a few years ago. Unfortunately it wont work for you. In my scenario I had control of the sender app and I appended the file size to the file name then the receiving app could check the file size until it matched.

Kevin, EOF is raised when there’s nothing more to read. So if I read faster that the copy is happening, I’m at the EOF but there’s still more to come.

Thanks everyone for the help. I don’t think this can be solved using Xojo alone. I’ll settle for a timer / filesize solution for now.
It’s not perfect because of buffering but I pretty much tried everything. (Except Norman’s FSEvents suggestion but I’m afraid that’s above my experience level)