Example FileWatcher

FileWatcher scans a file and sends it for processing. After the file is processed, the file is moved to the done folder. If more than one FileWatcher threads attempt to process the file, concurrency issues may occur.

To prevent concurrency issues:

  • Before FileWatcher picks up a file for processing, a lock is placed on the parent directory. This way, only one FileWatcher can pick up files from one directory. The FileWatcher locking synchronizes access to a directory for multiple FileWatcher threads.
  • After the file is processed, the file lock is released.
  • When the FileWatcher thread starts, it checks for any abandoned lock files for the available datasets. For example, lock files may have been left abandoned, as in the following cases:
    • The executing thread creates a lock file for a file.
    • The application terminates before the thread has completed processing.