Parsing of a Large Number of Records

When reading a large number of records from a file, the Parse Copybook Data activity might consume significant machine resources. To minimize memory usage, read the input in parts, parsing and processing a small set of records before moving on to the next set.

Note that the term record means the data for the entire copybook. If n records are read, that means the activity has read the entire copybook n times. If a copybook structure contains repeating items, all of them are read during the reading of a single record.

Rules

With the Input Type field set to File, you can specify the value for the noOfRecords input element under the Input tab. The following rules apply:

  • The Parse Copybook Data activity repeats reading of the copybook data up to the specified number of times. If more data is available in the file when you next invoke the Parse Copybook Data activity, that activity continues reading the data. Note that this behavior only applies to activities that read the same file — more specifically, the file whose input element fileName is of the same value.

    You can read different files or byte data with other Parse Copybook Data activities. Doing so does not affect the current position in this file.

    Important: The fileName input element must remain unchanged for subsequent reading to take place. For example, Parse Copybook Data reads \data\myfile and myfile separately even if both paths point to the same location.

  • The output element noOfRecords always contains the number of records read by the activity.
  • If you set Input Type to File, the Parse Copybook Data activity contains the output element EOF, whose value is true if all data has been read from the file. That value can serve as an indicator for you to stop invoking the Parse Copybook Data activity.
  • If you set a noOfRecords value but the file contains more data, the file remains open until all data has been read by subsequent invocations of the Parse Copybook Data activity or until the Plug-in throws an exception. The file is also closed at the end of the process execution.
  • If you do not set Input Type to File or specify a noOfRecords value, reading continues until the end of the input data is reached. At that point, if Input Type is set to File, the Parse Copybook Data activity closes the file.
  • If you do not specify a value or specify -1 as the value for the noOfRecords field, the activity reads all the records in the input file.
  • If you do not select the Multiple Records check box, the Parse Copybook Data activity ignores the value of noOfRecords, reads one record, and, subsequently, closes the file.
    Warning: Different operating systems might behave differently with regards to what operations are permissible on an open file. For example, on Windows a file cannot be deleted until it is closed, but Sun Solaris permits it.

    It is advisable not to design a process that depends on these aspects of a file system behavior.