Loading a Database (lkt_dbload)

This command is the only DevKit command that creates a new database.
dvkerr_t lkt_dbload( lpar_t host, lpar_t name, lpar_t dbpars, lpar_t reclist, 
                     lpar_t *dbstats );
dvkerr_t lkt_dbloadT(lpar_t host, lpar_t name, lpar_t tran, lpar_t dbpars,
                     lpar_t reclist, lpar_t *dbstats );

In the normal case a database is created and loaded with an initial set of records (hence the command name, lkt_dbload). Records might then be added, deleted, or replaced in real-time using commands described later on in Updating Loaded Databases. Applications that do not require continual data updates might find it simpler to replace an entire database with a new version. We'll describe the best way to do this when we discuss the lkt_dbmove command (Renaming a Database (lkt_dbmove)).

Starting with release 4.4.1 loading is optimized based on the number of records to be loaded.

Passing the LPAR_INT_DBNUMRECORDS value in the dbpars list informs the TIBCO Patterns servers of the approximate number of records to be loaded so that it can choose the best loading strategy. The loading takes place even if this value is not given or is incorrect, the only consequence is that loading performance might not be optimal.

A batch of over 40,000 records is needed to achieve the best load performance. With batches of this size or larger the TIBCO Patterns servers employs multiple processors to speed up the loading.

 

For smaller numbers of records, a low overhead single stream load is performed. The best strategy for loading depends on your hardware and the number and nature of the tables to be loaded. If optimal load times are critical consult your TIBCO representative for the best means of optimizing loads for your particular case.

Note that if a multi-threaded fast load is performed the memory cap is checked only once, before the load is started. Thus the memory cap can be greatly exceeded. Therefore, if you are using memory caps it is not recommended that you use fast loading.

Input

host (optional)

This is used to identify the server. For more information, see Communicating with TIBCO Patterns Servers.

name (required)

This is a null-terminated character string that is unique to the table. Every database must have a non-zero length.

dbpars (optional)

is a list of lpars specifying parameters that in some way govern or affect the database over its entire lifetime. Every one of these parameters has a default value, so it is not strictly necessary to pass any dbpars list to lkt_dbload. If the defaults are satisfactory, simply pass a value of LPAR_NULL instead.

Here is the list of database parameters and their default values:

•

LPAR_BOOL_DBGIPFILTER is a Boolean parameter specifying whether or not to enable the GIP prefilter. See The GIP Prefilter - Usage Details for a full discussion.

Default value: true (if GIP prefilter is the default prefilter. It is False if it is not.)

•

LPAR_BOOL_DBSORTFILTER is a Boolean parameter specifying whether or not to enable the SORT prefilter. See The SORT Prefilter - Usage Details for a full discussion.

Default value: true (if SORT prefilter is the default prefilter. It is False if it is not.)

•

LPAR_BOOL_DBPSIFILTER is a Boolean parameter specifying whether or not to enable the PSI prefilter. See The PSI Prefilter - Usage Details for a full discussion.

Default value: true (if PSI prefilter is the default prefilter. It is False if it is not.)

•

LPAR_LST_ENCODINGS is only used if the SORT or PSI prefilter is enabled. See The SORT Prefilter - Usage Details and The PSI Prefilter - Usage Details for a full discussion of this parameter.

Default value: Every searchable field is indexed forward and backward.

•

LPAR_INT_PSI_DENSITY overrides the default encoding density for PSI-indexed tables. See The PSI Prefilter - Usage Details for a full discussion of this parameter.

•

LPAR_INT_DBNUMFIELDS specifies the number of fields that each record in the database has. If specified, every record that is ever loaded into this database must have exactly this number of fields (see Building the Input: Data Records for how this field structure is specified in terms of field lengths).

Default value: If the number of fields is not specified here it is taken from the number of fields in the LPAR_STRARR_FIELDNAMES parameter.

If that is not given it is taken from the number of fields in the LPAR_INTARR_FIELDTYPES parameter. If neither is given it is taken from the number of fields in the record source. It is an error if the number of fields is defined in none of these places.

•

LPAR_STRARR_FIELDNAMES is an array of strings containing the names of each field. The number of field names must be equal to the number of fields. Field names are required, however if records are being loaded from a CSV file the names might be defined in the CSV file. If the names are defined both here and in the CSV file the names must be identical.

Default value: If data is not being loaded from a CSV file with LPAR_BOOL_FNAMESFIRST set then this is required. Otherwise they default to the names as defined in the CSV file.

•

LPAR_INTARR_FIELDTYPES is an array of integers containing the field type of each field. The length of the field type array must be equal to the number of fields. For a description of field types and a list of valid field types see The TIBCO Patterns Table.

Default value: All fields are of type LKT_FLD_TYPE_SRCHTEXT by default.

•

LPAR_BLK_DBINFO is simply a block of information that you can attach to a database. It can contain any text data you wish. Its contents are returned any time database statistics are returned (see the following Output).

Default value: Databases have no database info unless you supply it.

•

LPAR_STRARR_CHARMAPS is an array of names of character maps. The length of the array must be the same as the number of fields in the database. Even though character maps apply only to fields of type LKT_FLD_TYPE_SRCHTEXT or LKT_FLD_TYPE_ATTRIBUTES a valid character map name must be supplied for every field. For a description of character maps see Character Mapping.

Default value: The default character map DVK_CMAP_STDNAME is assigned to every field.

•

LPAR_LST_PIDXDEF defines a partition index for this table. Any number of these values might be given. For a description of this option and partition indexes see Partition Indexes.

Default value: No indexes are created.

•

LPAR_INT_DBNUMRECORDS gives an estimate of the number of records in the record list. It is not necessary that this value be exact or even close.

This is used by the TIBCO Patterns servers to determine the most efficient means of loading the records.

Default value: 0 (It assumes a small batch of records). The exception is if loading using an LPAR_LST_REMOTEFILE file specification the assumption is that it is a large batch of records.

•

LPAR_BOOL_DOMAXWORK applies when loading records. If this flag is true invalid records are quietly ignored and loading of other records continues. If this flag is not given or is false an invalid record causes the entire table create and load operation to fail.

Some conditions that are considered to be an "invalid record" include: invalid UTF-8 data, duplicate record key, wrong number of fields, a completely empty record, an empty key value (if keys are not being generated automatically). It is not considered invalid for a record to have invalid or empty data values in non-text fields. These are simply stored as having the special empty or invalid value.

Default value: false.

•

LPAR_STR_PARENT_TBL must name an existing parent table. This table is created as a child table of the named parent table. If not given this is not a child table. A child table cannot also be a parent table, therefore it is not valid to give this argument and the LPAR_BOOL_PARENT_TBL argument with a value of true.

•

LPAR_BOOL_PARENT_TBL indicates whether this table is a parent table. If given and set to true, this table is created as a table that can act as a parent in a join relationship. If false, or not given this table is a standard or child table and cannot act as a parent table in a joint relationship. A parent table cannot also be a child table.

•

LPAR_BOOL_DBGIPGPU is a Boolean parameter specifying whether GPU acceleration is to be employed when searching this table. If GPU is not enabled on this server a value of true causes a FEATURESET error. This argument is ignored if this table is not GIP indexed. Set this value to true to enable GPU acceleration for this table.

•

LPAR_INTARR_GPU_DEVICE_IDS is used to specify which GPU devices is to be used when employing the GPU accelerator. A PARAMVAL error is returned if a given ID is not a valid GPU device ID. A GPU_DEVICE_NOT_ALLOWED error is returned if a given ID is a valid GPU device, but the device can’t be used. If GPU acceleration is not enabled and this is given, a FEATURESET error is returned. If this argument is not given, the default GPU devices are used.

If you are using the SORT or PSI prefilter it might be necessary to set additional database parameters (see the sections The SORT Prefilter - Usage Details andThe PSI Prefilter - Usage Details).

reclist (optional)

must be a list of records or a file specification. If a record list is given every record must contain a unique key.

File specifications take the form of an LPAR_LST_LOCALFILE or an LPAR_LST_REMOTEFILE. This list type lpar must contain either an LPAR_STR_CSVFILE or LPAR_STR_FWFFILE as well as any options appropriate for the lkt_dbread command. If no host is specified (so that the command runs in the local process), there is no distinction made between the LPAR_LST_LOCALFILE and LPAR_LST_REMOTEFILE forms. If a host is specified, an LPAR_LST_LOCALFILE indicates a file that is read by the client, the contents of which are sent to the server. With an LPAR_LST_REMOTEFILE, only the filename and dbread options are sent to the server. The file must exist within the server's loadable-data directory. Its contents are read directly by the server.

It is permitted to create a database with zero records by passing lkt_dbload either LPAR_NULL or a zero-item list as the value of reclist.

tran (optional)

identifies the user transaction (LPAR_LONG_TRAN_ID) under which the table load operation is to be performed.

Output

dbstats (optional)

is a list of lpars giving some statistics on the newly-created database. These include the effective values of the following database parameters described previously:

•

LPAR_STR_DBDESCRIPTOR

•

LPAR_BLK_DBINFO

•

LPAR_INT_DBNUMFIELDS

•

LPAR_STRARR_FIELDNAMES

•

LPAR_INTARR_FIELDTYPES

•

LPAR_STRARR_CHARMAPS

•

LPAR_BOOL_DBGIPFILTER

•

LPAR_BOOL_DBSORTFILTER

•

LPAR_BOOL_DBPSIFILTER

•

LPAP_BOOL_GIPGPU

•

LPAR_INTARR_GPU_DEVICE_IDS (returned only if the default set is not used)

•

LPAR_STR_PARENT_TBL (returned only for child tables)

as well as the following:

•

LPAR_STR_CHECKPOINT_STATUS describes the data-persistence status for this database. For checkpoint/restore it is one of: "Last Checkpoint: yyyy/mm/dd-hh:mm:ss", "Never Checkpointed", "Checkpoints Disabled", "No Checkpoint Info", "Gateway generated checkpoint id: yyyy/mm/dd-hh:mm:ss". For durable-data, it is "Automatic".

•

LPAR_LONGINTARR_FIELDBYTECOUNTS the total number of bytes of data stored in the table broken out by field.

•

LPAR_LONGINTARR_FIELDCHARCOUNTS the total number of characters of data stored in the table broken out by field after character mapping is applied.

•

LPAR_INT_DBNUMRECORDS is the number of records in the database.

•

LPAR_INT_DBKEYTREEBYTES is the number kilobytes used to index the record keys.

•

LPAR_INT_DBRECFIELDKBYTES is the number of kilobytes used to store the record data.

•

LPAR_INT_DBHEADERKBYTES is the number of kilobytes of memory used in storing various header data. It is the overhead.

•

LPAR_INT_DBIDXTOTALKBYTES is the number of kilobytes used to store index data. This includes either GIP, PSI, or SORT prefilter index memory needs and the memory used by any predicate partitioned indexes on the table.

•

LPAR_INT_DBTOTALKBYTES is the total memory occupied by the database in kilobytes.

•

LPAR_INT_PROCTOTALKBYTES is the total memory occupied by the process. This is returned only for servers built for the Linux platform.

•

LPAR_LONGINT_TRAN_ID is the transaction id of the transaction that owns or claims the object at that instance.

•

LPAR_INT_TRAN_OBJ_STATE is the state of an object when a transaction owns or claims it; if there is no such transaction, it returns LKT_TRAN_OBJ_EXISTS.

•

LPAR_INTARR_PSI_DENSITIES (PSI-indexed tables only) lists the densities of the PSI encodings.

•

LPAR_STR_TABLE_TYPE is one of: “Standard”, “Child”, “Parent” or “Unknown”. This indicates the type of table. “Unknown” is returned only if there was some kind of error in creating the table.

•

LPAR_STRARR_CHILD_TBLS is a list of the child tables for this table. This is returned only for a parent table. This list contains zero entries if the parent table has no child tables.

If you do not care to have any of these statistics, just pass lkt_dbload a pointer value of NULL for dbstats.

Error codes and items returned by lkt_dbload

ARRAYLEN	array lpar with wrong number of values
CHARCONV	record that contains illegally encoded character
DBEXISTS	name of already-existing database
DBPARAM	inappropriate item in a parameter list
DUPFIELDNAM	ES field names array with duplicate names
EXPECTDBDESC	item that should have been a table name
EXPECTLIST	item that should have been a list type lpar
FEATURESET	lpar applicable to a premium feature not available
FILEFORMAT	LPAR_STR_FILEFORMATERROR description of error
GPU_DEVICE_NOT_ALLOWED	Bad GPU device id list
INTERNAL	(none) or the record causing the error or LPAR_STR_ERRORDETAILS giving more details
IOERROR	LPAR_STR_SYSERROR description of error reading records
JOINSET	Name of table
NOCHARMAP	dbpars list containing invalid character map name
NODBDESC	(none)
NOMEM	record at which database memory cap was exceeded.
NORECKEY	record that lacks a record key
NOSRCHTXT	record that lacks searchable text
NOSYSINIT	(none)
NUMFIELDS	record that contains the wrong number of fields
PARAMCONFLICT	lpar that contains a conflicting value
PARAMTYPE	Invalid name
PARAMMISSING	list in which a required item is missing
PARAMVAL	lpar that contains an illegal value
RECEXISTS	a record containing a duplicated record key
TBLNOTCHILD	(none)
TRAN_UNKNOWN	lpar that contains the unknown transaction id
TRAN_IN_USE	lpar that contains the transaction id
TRANCONFLICT	list that contains LPAR_LONGINT_TRAN_ID and LPAR_STR_ERRORDETAILS
UNKFIELD	Encoding field names list with bad field name