Managing Metadata for Fixed-Format and Delimited Files
When the server accesses a data source, it needs to know how to interpret the data stored there. For each data source the server accesses, you create a synonym that describes the structure of the data source and the server mapping of the Fixed-Format or Delimited Files data types.
Creating Synonyms
Synonyms define unique names (or aliases) for each Fixed-Format or Delimited File that is accessible from the server. Synonyms are useful because they hide the underlying data source location and identity from client applications. They also provide support for extended metadata features of the server, such as virtual fields and additional security mechanisms.
Using synonyms allows an object to be moved or renamed while allowing client applications to continue functioning without modification. The only modification required is a redefinition of the synonym on the server. The result of creating a synonym is a Master File that represents the server metadata.
Creating Synonyms for Fixed-Format Files
A Fixed-Format file is a sequential file in which each field occupies a predefined position in the record, with unoccupied positions filled by blanks to maintain the fixed structure.
Create a Synonym for a Fixed-Format File
To create a synonym, you must have previously configured the adapter. You can create a synonym from the WebFOCUS Reporting Server browser interface or the ibi Data Migrator desktop interface.
- Procedure
- From the WebFOCUS Reporting Server browser interface Application page, click Get Data.
- On the
Configured Adapters section of the page, in Simple Mode, right-click an adapter
and click
Show
Connections. Right-click a connection.
Depending on the type of adapter you choose, one of the following options appears on the context menu.
- Show DBMS objects. This option opens the page for selecting synonym objects and properties.
- Create metadata objects. This option opens the page for selecting synonym objects and properties.
- Show files. This option opens a file picker. After you choose a file of the correct type, the page for selecting synonym objects and properties opens.
- Show local files. This option opens a file picker. After you choose a file of the correct type, the page for selecting synonym objects and properties opens.
- Show topics. This option opens the page for selecting synonym objects and properties for topics within the environment.
- Enter values for the parameters required by the adapter
as described in the synonym creation parameters reference.
- For Windows, UNIX, and IBM i, see Synonym Creation Parameters for Fixed-Format Files On Windows, UNIX, and IBM i IFS File Systems.
- For z/OS, see Synonym Creation Parameters for Fixed-Format Files On z/OS.
- After entering the parameter values, click Create Synonym.
The Status pane indicates that the synonym was created successfully.
The synonym is created and added under the specified application directory.
Synonym Creation Parameters for Fixed-Format Files On Windows, UNIX, and IBM i IFS File Systems
The following list describes the parameters for which you need to supply values, and the related tasks you need to complete to create a synonym for the adapter. These options may appear on multiple panes. To advance from pane to pane, click the buttons provided, ending with the Create Synonym button, which generates the synonym based on your entries.
Path to the directory containing multiple COBOL files.
Select Customize COBOL FD conversion options to customize how the COBOL FD is translated. If you do not select this checkbox, default translation settings are applied.
For more information, see Customization Options for COBOL File Descriptions.
Choose one of two options:
- Single File defines a single data file. This option is not applicable for remote connections.
- Collection of files defines a collection of data files to be described by one synonym.
Indicates the data retrieval type.
- One-time retrieval retrieves data once to create synonym for sequential file.
- File Listener provides continued retrieval.
File Listener attributes
Indicates how often the listener checks for the presence of a file in number of seconds. The default is 10 seconds.
Indicates the timeout interval for the listener in number of seconds. The default is 10 seconds.
Indicates the number of records processed within one request. This option is available for single file processing.
Indicates the number of files processed within one request. This option is available for processing collection of files. The default is 99,999,999.
Indicates the mechanism used by files agents to pick up files from their directories.
- IMMEDIATE processes the file as soon as it is detected.
- TRIGGER processes the file as soon as a trigger file is detected. This option requires an additional setting for the trigger file extension.
Indicates the extension of the trigger file. The trigger file extension is added to the full name of the file being listened for.
Indicates how the file is handled after it has been processed by a file agent.
- DELETE deletes the file.
- ARCHIVE copies the file with the timestamp to a specified directory defined in File Listener Archiving Directory option.
- KEEP keeps the files.
Defines the location where the data files will be archived when DISCARD strategy of ARCHIVE is used. This can be an application directory, a mapped application name, or a directory location on your file system.
When using a directory on a file system for local files, the name of the physical directory can be used.
When using a directory on a file system for remote data files, a directory relative to the initial directory can be used. For example, if your remote initial directory is /home/user1/apps, using sales will set the data files directory location to /home/user1/apps/sales.
Collection Definition attributes
Defines the location of data files for polling. This can be an application directory, a mapped application name, or a directory location on your file system.
When using a directory on a file system for local files, the name of the physical directory can be used.
When using a directory on a file system for remote data files, a directory relative to the initial directory can be used. For example, if your remote initial directory is /home/user1/apps, using sales sets the data files directory location to /home/user1/apps/sales.
File Selection Mask attributes
These attributes allow you to limit the list of data files.
Type a full name or a partial name with a wildcard symbol %. A full name returns just that entry. A name with a wildcard symbol may return many entries.
Type an extension with or without the wildcard symbol %.
Synonym field names processing options
Select the Validate checkbox if you wish to convert all special characters to underscores and perform a name check to prevent the use of reserved names. (This is accomplished by adding numbers to the names.) This parameter ensures that names adhere to specifications. See Validation for Special Characters and Reserved Words for more information.
When the Validate option is unchecked, only the following characters are converted to underscores: '-'; ' '; ' \'; '/'; ','; '$'. No checking is performed for names.
Select the Make unique checkbox if you wish to set the scope for field and group names to the entire synonym. This ensures that no duplicate names are used, even in different segments of the synonym. When this option is unchecked, the scope is the segment.
Indicates the name of the synonym.
Select an application directory. The default value is baseapp.
If you have tables with identical table names, assign a prefix or a suffix to distinguish them. For example, if you have identically named human resources and payroll tables, assign the prefix HR to distinguish the synonyms for the human resources tables. Note that the resulting synonym name cannot exceed 64 characters.
If all tables and views have unique names, leave the prefix and suffix fields blank.
To specify that this synonym should overwrite any earlier synonym with the same fully qualified name, select the Overwrite existing synonyms checkbox.
Extended data format attributes
This option provides data portability between servers by enabling processing of data loaded in different encoding systems.
Click the Extended data format attributes checkbox to display the Codepage option.
In the input box provided, enter the code page of the stored data. Your entry is added to the Master File of the generated synonym.
Synonym Creation Parameters for Fixed-Format Files On z/OS
The following list describes the parameters for which you need to supply values, and the related tasks you need to complete to create a synonym for the adapter. These options may appear on multiple panes. To advance from pane to pane, click the buttons provided, ending with the Create Synonym button, which generates the synonym based on your entries.
Choose one of the following options from the dropdown menu:
- Fully qualified PDS name to indicate
a partitioned data set on MVS.
In the input boxes provided, type a PDS name preceded by // and a Member name containing the location of the COBOL FD source. If you wish, you can filter the member name using a wildcard character (%).
- Absolute HFS directory pathname to indicate
a hierarchical file structure on USS.
In the input boxes provided, type a Directory name to specify the HFS location that contains the COBOL FD and a File name and File extension. If you wish, you can filter the file and extension using a wildcard character (%).
Synonym field name processing options
Select the Validate checkbox if you wish to convert all special characters to underscores and perform a name check to prevent the use of reserved names. (This is accomplished by adding numbers to the names.) This parameter ensures that names adhere to specifications. See Validation for Special Characters and Reserved Words for more information.
When the Validate option is unchecked, only the following characters are converted to underscores: '-'; ' '; ' \'; '/'; ','; '$'. No checking is performed for names.
Select the Make unique checkbox if you wish to set the scope for field and group names to the entire synonym. This ensures that no duplicate names are used, even in different segments of the synonym. When this option is unchecked, the scope is the segment.
Select Customize COBOL FD conversion options to customize how the COBOL FD is translated. If you do not select the checkbox, default translation settings are applied.
For more information, see Customization Options for COBOL File Descriptions.
Select an application directory. The default value is baseapp.
If you have tables with identical table names, assign a prefix or a suffix to distinguish them. For example, if you have identically named human resources and payroll tables, assign the prefix HR to distinguish the synonyms for the human resources tables. Note that the resulting synonym name cannot exceed 64 characters.
If all tables and views have unique names, leave the prefix and suffix fields blank.
Extended data format attributes
This option provides data portability between servers by enabling processing of data loaded in different encoding systems.
Click the Extended data format attributes checkbox to display the Codepage option.
In the input box provided, enter the code page of the stored data. Your entry is added to the Master File of the generated synonym.
To specify that this synonym should overwrite any earlier synonym with the same fully qualified name, select the Overwrite existing synonyms checkbox.
This column displays the name that is assigned to each synonym. To assign a different name, replace the displayed value.
Select files for which you wish to create synonyms:
- To select all files in the list, click the Select All button.
- To select specific files, select the corresponding checkboxes.
After choosing files, choose the corresponding data file name from the dropdown list.
Managing Synonyms
Once you have created a synonym, you can right-click the synonym name in the navigation pane of either the WebFOCUS Reporting Server browser interface or the ibi Data Migrator desktop interface to access the available options.
For a list of options, see Synonym Management Options.
Creating Synonyms for Delimited Files
A delimited file is a sequential file in which fields are stored in a predefined sequence, separated (that is, delimited) by a special character or characters. (Although fields must maintain their predefined order in the file, and delimiters are required, blanks are not required to represent unoccupied positions.
The types of field delimiters are:
- Comma.
- Tab.
- | (pipe).
- Token, which is any combination of characters defined in the data source Synonym. (Note that token-delimited files cannot be used in Joins.)
You can create different types of synonyms using the Adapter for Delimited Files:
- A synonym for a file that resides on a server or on a local machine.
- A synonym for a collection of files with a one-time retrieval.
- A synonym for a collection of files for a File Listener.
- Multiple Synonyms.
Create a Synonym for a Delimited File
To create a synonym, you must have previously configured the adapter. You can create a synonym from the WebFOCUS Reporting Server browser interface or the ibi Data Migrator desktop interface.
- Procedure
- From the WebFOCUS Reporting Server browser interface menu bar, click Applications,
then New, and then Synonym
Or
From the ibi Data Migrator desktop interface, right-click an application directory, then select New, and then Synonym.
The Select Data File for Delimited Flat File Synonym pane opens.
- Enter values for the parameters required by the adapter as described in the synonym creation parameters reference.
- After entering the parameter values, click Create Synonym.
The Status pane indicates that the synonym was created successfully.
The synonym is created and added under the specified application directory.
Synonym Creation Parameters for Delimited Files
The following list describes the parameters for which you will need to supply values, and the related tasks you need to complete to create a synonym for the adapter. These options may appear on multiple panes. To advance from pane to pane, click the Next button, ending with the Create Synonym button, which generates the synonym based on your entries.
Select this checkbox for Multiple Synonyms.
Is the path to multiple data files.
Defines the location of the data file.
From the dropdown menu select:
- Server side.
- Local machine.
Defines an absolute path to the data file on the server or a directory relative to the initial one. It includes the name of the data file.
Enter the path and file name or click the ellipsis (...) to select the file.
Click the ellipsis (...) to select the data file to be uploaded.
Select:
- Single File to define a single data file. This option is not applicable for remote connections.
- Collection of files to define a collection of data files to be described by one synonym.
Indicates the data retrieval type:
- One-time retrieval retrieves data once, to create a synonym for a sequential file.
- File Listener provides continual retrieval.
Record Specification
Defines a character or series of characters that indicate the separation between file records or data rows (for example, groups of delimited fields).
This attribute is reflected in the Access File as RDELIMITER.
Defines whether the string in the delimiter field is a printable string or a non-printable decimal character. Please note that a non-printable decimal character appears (translated) in the Access File as a hexadecimal value. If the delimiter is entered as hexadecimal, set this attribute to No.
Field Specification
Defines a character or series of characters that indicate the separation between file fields.
To use multiple characters, enter them together. For example, enter two slashes (//).
To use a tab character as a delimiter, enter the reserved word TAB.
Alternatively, you can enter hexadecimal values directly. For example, Linefeed and Carriage Return would be 0x0A and 0x0D, respectively.
If several characters form a separator, enter then as one string. For example, Linefeed and Carriage Return would be 0x0A0D.
Defines whether the string in the delimiter field is a printable string or a non-printable decimal character. Please note that the non-printable decimal character appears (translated) in the Access File as a hexadecimal value.
If the delimiter is entered as a hexadecimal value, set this attribute to No.
If this attribute is set to Yes, then the values in the Delimiter attribute should be entered as decimals. For example, for non-printable separators such as Linefeed, use the decimal value of 10. If several non-printable characters form a separator, use a space between the decimal values. For example, to use Linefeed and Carriage Return together, enter 10 or 13.
Select this checkbox to request that the names in the header line be used as column names in the synonym.
Specifies the number of rows above the header row that should be ignored when creating the synonym and reading the data.
If character columns are enclosed in quotation marks or another symbol, enter an enclosure symbol for the character column format used in the Delimited Files for which the synonyms are being generated. For example, for a file in which character columns are enclosed in double quotation marks, specify " in the Enclosure field.
Enter a value to indicate how many records from the DFIX data file to scan to determine the length of the columns.
If this parameter is omitted, all records from the file are processed to make this determination.
Indicates whether the original data layout, including empty records and linefeeds, will be preserved.
- No. The original characters are replaced by a printable string. This is the default.
- Yes. The original characters, including non-printable characters and linefeeds, will be preserved.
This attribute is reflected in the Access File as PRESERVEFRMT.
File Listener
Indicates how often the listener checks for the presence of a file in number of seconds. The default is 10 seconds.
Indicates the timeout interval for the listener in number of seconds. The default is 10 seconds.
Indicates the number of files processed within one request. The default is 99,999,999.
Indicates the mechanism used by files agents to pick up files from their directories.
- IMMEDIATE processes the file as soon as it is detected.
- TRIGGER processes the file as soon as a trigger file is detected. This option requires an additional setting for the trigger file extension.
Indicates the extension of the trigger file. The trigger file extension is added to the full name of the file being listened for.
Indicates how the file is handled after it has been processed by a file agent.
- DELETE deletes the file.
- ARCHIVE copies the file with the timestamp to a specified directory defined in File Listener Archiving Directory option.
- KEEP keeps the files.
Archiving directory on the server where the data file will be archived. You can use an absolute path or a directory relative to the initial one.
Collection Definition
Defines the location type of the data files. It can be a physical directory for local files or a directory relative to the initial directory for remote data files. For example, if your remote initial directory is /home/user1/apps, using sales in collection definition sets the data files location to /home/user1/apps/sales.
File Selection Mask
Type a full name or a partial name with a wildcard symbol %. A full name returns just that entry. A name with a wildcard symbol may return many entries.
Type an extension with or without the wildcard symbol %.
Synonym field names processing options
Select an application directory. The default value is baseapp.
If you have tables with identical table names, assign a prefix or a suffix to distinguish them. For example, if you have identically named human resources and payroll tables, assign the prefix HR to distinguish the synonyms for the human resources tables. Note that the resulting synonym name cannot exceed 64 characters.
If all tables and views have unique names, leave the prefix and suffix fields blank.
To specify that this synonym should overwrite any earlier synonym with the same fully qualified name, select the Overwrite existing synonyms checkbox.
This column displays the name that is assigned to each synonym. To assign a different name, replace the displayed value.
Select the checkboxes to left of Default Synonym Name to choose the files for which you want to create synonyms. The data files are highlighted when you choose the corresponding checkboxes.
Click Select All Files to select all the data files in the directory path.
Extended data format attributes
This option provides data portability between servers by enabling processing of data loaded in different encoding systems.
Select the Extended data format attributes checkbox to display the Codepage option.
In the input box provided, enter the code page of the stored data. Your entry is added to the Master File of the generated synonym.
Sample DFIX Data File and Generated Synonym
This example is for a single synonym for a file on the serve.
DFIX data file (Comma-delimited):
ENGLAND,JAGUAR,V12XKE AUTO,CONVERTIBLE,2,7427,8878.0 ENGLAND,JAGUAR,XJ12L AUTO,SEDAN,5,11194,13491,12000 ENGLAND,JENSEN,INTERCEPTOR III,SEDAN,4,14940,17850,0 ENGLAND,TRIUMPH,TR7,HARDTOP,2,4292,5100,0 JAPAN,DATSUN,B210 2 DOOR AUTO,SEDAN,4,2626,3139,4300 JAPAN,TOYOTA,COROLLA 4 DOOR DIX AUTO,SEDAN,4,2886,3339,35030 ITALY,ALFA ROMEO,2000 4 DOOR BERLINA,SEDAN,4,4915,5925,4800 ITALY,ALFA ROMEO,2000 GT VELOCE,COUPE,2,5660,6820,12400 ITALY,ALFA ROMEO,2000 SPI,2,25000,31500,0,13000 ITALY,MASERATI,DORA 2 DOOR,COUPE,2,25000,31500,0 GERMANY,AUDI,100 LS 2 DOOR AUTO,SEDAN,5,5063,5970,7800 GERMANY,BMW,2002 2 DOOR,SEDAN,5,5800,5940,8950 GERMANY,BMW,2002 2 DOOR AUTO,SEDAN,4,6000,6355,8900 GERMANY,BMW,3.0 SI 4 DOOR,SEDAN,5,10000,13752,14000 GERMANY,BMW,3.0 SI 4 DOOR AUTO,SEDAN,5,11000,14123,1894 GERMANY,BMW,530I 4 DOOR,SEDAN,5,8300,9097,14000 GERMANY,BMW,530I 4 DOOR AUTO,SEDAN,5,8400,9495,1560 FRANCE,PEUGEOT,504 4 DOOR,SEDAN,5,4631,5610,0
To generate a synonym for the DFIX comma-delimited data file, enter the following information on the Create Synonym panes of the WebFOCUS Reporting Server browser interface or the ibi Data Migrator desktop interface:
- In the Directory path field, enter the directory name containing the delimited flat file.
- Enter the file name (for example, dfix_car).
- Enter the file extension field (for example, .dat.)
- Click Select Files.
- On the second Create Synonym pane, select a file from the list of displayed data files and specify comma, as the separator, in the Delimiter field.
- Click Create Synonym. The synonym is
created and added under the specified application directory (baseapp
is the default).
A status window displays the message: All Synonyms Created Successfully
- From the message window, click Applications on the menu bar.
- Open the baseapp application folder in the navigation pane and click the synonym--in this example, dfix_car.
- Choose Edit as Text from the menu to view the generated Master File, then choose Edit Access File as Text to view the corresponding Access File.
Generated Master File:
FILENAME=CAR_DFIX, SUFFIX=DFIX ,
DATASET=directory/car_dfix.dat, $
SEGMENT=CAR_DFIX, SEGTYPE=S0, $
FIELDNAME=FIELD_1, USAGE=A7, ACTUAL=A7BV, $
FIELDNAME=FIELD_2, USAGE=A10, ACTUAL=A10BV, $
FIELDNAME=FIELD_3, USAGE=A23, ACTUAL=A23BV, $
FIELDNAME=FIELD_4, USAGE=A11, ACTUAL=A11BV, $
FIELDNAME=FIELD_5, USAGE=I1, ACTUAL=A1, $
FIELDNAME=FIELD_6, USAGE=I5, ACTUAL=A5V, $
FIELDNAME=FIELD_7, USAGE=I5, ACTUAL=A5V, $
FIELDNAME=FIELD_8, USAGE=I5, ACTUAL=A5V, $
Note that the FILE declaration in the generated Master File includes the attribute:
SUFFIX=DFIX
Generated Access File:
SEGNAME=DFIX_CAR, DELIMITER=',', HEADER=NO,$
The delimiter consists of a single printable character.
Sample DFIX Data File With Header and Enclosure
This sample file contains a header line and uses the enclosure character ".
"COUNTRY","CAR","MODEL","BODYTYPE","SEATS","DEALER_COST","RETAIL_COST","SALES" "ENGLAND","JAGUAR","V12XKE AUTO","CONVERTIBLE",2,7427,8878,0 "ENGLAND","JAGUAR","XJ12L AUTO","SEDAN",5,11194,13491,12000 "ENGLAND","JENSEN","INTERCEPTOR III","SEDAN",4,14940,17850,0 "ENGLAND","TRIUMPH","TR7","HARDTOP",2,4292,5100,0 "FRANCE","PEUGEOT","504 4 DOOR","SEDAN",5,4631,5610,0 "ITALY","ALFA ROMEO","2000 4 DOOR BERLINA","SEDAN",4,4915,5925,4800 "ITALY","ALFA ROMEO","2000 GT VELOCE","COUPE",2,5660,6820,12400 "ITALY","ALFA ROMEO","2000 SPIDER VELOCE","ROADSTER",2,5660,6820,13000 "ITALY","MASERATI","DORA 2 DOOR","COUPE",2,25000,31500,0 "JAPAN","DATSUN","B210 2 DOOR AUTO","SEDAN",4,2626,3139,43000 "JAPAN","TOYOTA","COROLLA 4 DOOR DIX AUTO","SEDAN",4,2886,3339,3503 "GERMANY","AUDI","100 LS 2 DOOR AUTO","SEDAN",5,5063,5970,7800 "GERMANY","BMW","2002 2 DOOR","SEDAN",5,5800,5940,8950 "GERMANY","BMW","2002 2 DOOR AUTO","SEDAN",4,6000,6355,8900 "GERMANY","BMW","3.0 SI 4 DOOR","SEDAN",5,10000,13752,14000 "GERMANY","BMW","3.0 SI 4 DOOR AUTO","SEDAN",5,11000,14123,18940 "GERMANY","BMW","530I 4 DOOR","SEDAN",5,8300,9097,14000 "GERMANY","BMW","530I 4 DOOR AUTO","SEDAN",5,8400,9495,15600
On the Synonym Creation panes, specify the Delimiter character as comma (,), the Enclosure character as double quotation marks ("), and check the Header box to use the names in the header line as field names (columns) in the Master File.
Generated Master File:
FILENAME=CAR_CSV, SUFFIX=DFIX ,
DATASET=directory/car_csv.csv, $
SEGMENT=CAR_CSV, SEGTYPE=S0, $
FIELDNAME=COUNTRY, ALIAS=COUNTRY, USAGE=A7, ACTUAL=A7BV, $
FIELDNAME=CAR, ALIAS=CAR, USAGE=A10, ACTUAL=A10BV, $
FIELDNAME=MODEL, ALIAS=MODEL, USAGE=A23, ACTUAL=A23BV, $
FIELDNAME=BODYTYPE, ALIAS=BODYTYPE, USAGE=A11, ACTUAL=A11BV, $
FIELDNAME=SEATS, ALIAS=SEATS, USAGE=I1, ACTUAL=A1, $
FIELDNAME=DEALER_COST, ALIAS=DEALER_COST, USAGE=I5, ACTUAL=A5V, $
FIELDNAME=RETAIL_COST, ALIAS=RETAIL_COST, USAGE=I5, ACTUAL=A5V, $
FIELDNAME=SALES, ALIAS=SALES, USAGE=I5, ACTUAL=A5V, $
The Delimiter, Enclosure, and Header entries are reflected in the Access File.
Generated Access File:
SEGNAME=CAR_CSV, DELIMITER=',', ENCLOSURE=", HEADER=YES, $
Usage and Actual Formats for Delimiters
Prior to Version 7 Release 6.1, server delimiter characters were identified by a field description in the Master File instead of by the DELIMITER parameter of the Access File.
Therefore, older synonyms may use this syntax with a FIELDNAME of DELIMITER, an ALIAS of the delimiter itself, and with USAGE and ACTUAL values as described in the following chart.
For example:
FIELDNAME=DELIMITER, ALIAS=',', USAGE=A1, ACTUAL=A1, $
USAGE and ACTUAL formats are determined as follows:
|
Type of delimiter |
USAGE |
ACTUAL |
|---|---|---|
|
Printable characters |
An where n is the number of characters |
An where n is the number of characters |
|
Non-printable character such as Tab |
I4 |
I1 |
|
Group (combination of printable and non-printable characters, or multiple non-printable characters) |
Sum of the individual USAGE lengths |
Sum of the individual ACTUAL lengths |
Updating Delimited Files
You can update Delimited Flat files using the MODIFY facility. The following delimiter characters are respected:
A tab character. This is the default.
A character string, for example ~.
A hex code. For example, 0x44 (a comma), or 0x0D0A (a return and a linefeed). The hex code uses ASCII for Windows or UNIX systems and EBCDIC for IBM Mainframes.
Delimited files have been added as a target type for ibi Data Migrator flows.