Parsing a Large Number of Records
The input for this activity is placed in a process variable and takes up memory as it is being processed. If you are reading a large number of records from a file, the process may consume significant machine resources. To avoid using too much memory, you may want to read the input in parts, parsing and processing a small set of records before moving on to the next set.
To process a large number of records, follow this procedure:
Procedure
-
Create a Parse Data activity.
-
Specify the fields on the Configuration tab. Check the check box for the Manually Specify Start Record field.
-
Select the Parse Data activity and click the group icon on the tool bar to create a group containing the Parse Data activity. For more information about working with groups, see TIBCO ActiveMatrix BusinessWorks™ Process Design.
-
Specify "Repeat Until True Loop" as the Group action, and specify an index name (for example, "
i
"). -
The loop should exit when the
EOF
output item for the Parse Data activity is set to true. For example, the condition for the loop could be set to the following:string($ParseData/Output/done) = string(true())
-
Set the
noOfRecords
input item for the Parse Data activity to the number of records you want to process for each execution of the loop. -
If you do not check the Manually Specify Start Record field on the Configuration tab of the Parse Data activity, the loop processes the specified
noOfRecords
with each iteration until there are no more input records to parse.
You can optionally check the Manually Specify Start Record field to specify the startRecord
on the Input tab. If you do this, you must create an XPath expression to properly specify the starting record to read with each iteration of the loop. For example, the count of records in the input starts at zero, so the startRecord
input item could be set to the current value of the loop index minus one. For example, $i - 1
.
The procedure above is a general guideline for creating a loop group for parsing a large set of input records in parts. You may want to modify the procedure to include additional processing of the records, or you may want to change the XPath expressions to suit your business process.