Setting InProgressSuffix

When large files are transferred to the incoming directory, FileWatcher may detect the file before it is completely written to disk. In this situation, FileWatcher may read the partial file, and the processing may fail. To prevent this, you can use InProgessSuffix to stop processing of the file.

Procedure

  1. Set the InProgressSuffix in the configuration file to ignore files that end in this value.
  2. When a file is being created, add the suffix to the file name.
  3. After the file write is complete, rename the file by removing the suffix.

Result

This procedure is not necessary for small files:

<CatalogPoller version="1.0"> 
<IncomingDir>incoming</IncomingDir> 
<DoneDir>done</DoneDir> 
<RejectedDir>rejected</RejectedDir>
<InProgressSuffix>.inprogress</InProgressSuffix>

For example:

datafile123.dat.inprogress - while it is being written
datafile123.dat - rename to this when done.
<InProgressSuffix>.inprogress</InProgressSuffix>

In this case, the file is initially named as datafile123.dat.inprogress.

FileWatcher detects the file, but ignores it, because the file ends with the “.inprogress” suffix. After the file is written, it is renamed to datafile123.dat and FileWatcher will pick it up during its next polling interval.