Creating, Starting, and Dropping Data Tables

Live Datamart release 2.1.0 added support for adding a new Data Table to a project while the Server is running, for dropping a running Data Table, and for starting a new table or restarting a dropped table. This page describes the state transitions you can expect for Data Tables transitioning between these states.

Note

When using lv-client or the Java API to add or remove a Data Table from the Server, it might create new files and directories in the project's workspace. For example, you might see a new folder lv-dynamicLVConfs; consider this folder as internal use only, and do not modify the folder or any of its contents. If you need to make changes to a newly created or recently dropped Data Table, use supported client commands and API operations.

Overview

The add and remove feature addresses the use case where remote clients need to create simple tables that do not have field rules, Data Sources, Transforms, and so on. There is a Java client API Interface TableAdminManager in com.streambase.liveview.client that includes createTable(), startTables(), and dropTables() methods. The lv-client command provides createtable, droptable,and starttable subcommand, as described on the lv-client reference page.

This primary goal of these features is to allow remote clients using the Java API to dynamically create LiveView tables and to publish data to these tables. You can consume the published data using LiveQL, LiveView Desktop, or your own custom clients. Only the Java Client API supports these features.

A secondary goal is to allow LiveView developers to add or drop tables from a LiveView Server running in StreamBase Studio. The LiveView Project Viewer provides ways to create lvconf files that can then be added to a LiveView Server instance running in the context of Studio. You cannot add or remove tables from a remote system from Studio.

Managing Add and Drop Table Transitions

The Java Client API provides the methods listed above to add, drop, and start tables. However, client applications must also confirm various state transitions of the tables they are manipulating, both to know when state transitions have completed and to monitor for error conditions.

There are two ways to track table transitions:

  • Monitor the LiveView system tables LVTables, LVTableColumns, and LiveViewStatistics.

  • Use the Java client API TableListListener.

The following sections describe these two approaches.

Monitor System Table Behavior

This section describes the changes to look for in the LiveView system tables when the create, drop, and start table API calls are made.

createTable()

Use the createTable() method to define a table's name and schema. When run, each createTable() call creates a new row in the LVTables table. The row's group is DYNAMIC, its StatusENUM is DEFINED and the create time is set to the current time on the server.

An lv-client command like the following:

lv-client createtable --name Foo --schema "\"(x int, s string)\"" --keys x

produces the following LVTables results:

Name,Group,ShortDescription,Description,TableSpace,Capabilities,QueryLanguages,
  Indices,StatusENUM,StatusMessage,IsSystemTable,CreateTime
Foo,DYNAMIC,null,null,null,"Snapshot, Continuous, Alert Rules, Delete, 
  Top-N Queries, Time-windowed Queries, Time-delayed Queries, Aggregate Queries, 
  Publish, Can be externally hosted, Calculated Columns",LIVEVIEW,x,DEFINED,
  "Defined, but not loaded",false,2015-09-30 11:33:53.060-0400

New rows are added to the LVTableColumns table for each field in the newly added table:

Table,Name,Type,ShortName,Description
Foo,s,string,s,
Foo,x,int,x,

There are no changes to the LiveViewStatistics table from a createTable() call.

startTables()

Use the startTable() method to compile and load a newly created table. From Studio, you can use the LiveView Project Viewer to start lvconf files you have added.

For a call to startTable(), an LVTables row for the specified table is updated for Java API users and added for Studio users. The row's StatusENUM to LOADING, which tells you that the table is being compiled. The row updates to ENABLED when the table is ready.

Name,Group,ShortDescription,Description,TableSpace,Capabilities,QueryLanguages,
  Indices,StatusENUM,StatusMessage,IsSystemTable,CreateTime
Foo,DYNAMIC,null,null,null,"Snapshot, Continuous, Alert Rules, Delete, 
  Top-N Queries, Time-windowed Queries, Time-delayed Queries, Aggregate Queries, 
  Publish, Can be externally hosted, Calculated Columns",LIVEVIEW,x,LOADING,
  Loading.,false,2015-09-30 11:33:53.060-0400
Foo,DYNAMIC,null,null,null,"Snapshot, Continuous, Alert Rules, Delete, 
  Top-N Queries, Time-windowed Queries, Time-delayed Queries, Aggregate Queries, 
  Publish, Can be externally hosted, Calculated Columns",LIVEVIEW,x,ENABLED,,
  false,2015-09-30 11:33:53.060-0400

dropTables()

When using the dropTable() method, the LVTables row for a table that was formerly ENABLED updates its StatusENUM field to DISABLED. When the table is fully dropped, its LVTables row is deleted. The LVTableColumns rows for the dropped table are also deleted.

Name,Group,ShortDescription,Description,TableSpace,Capabilities,QueryLanguages,
  Indices,StatusENUM,StatusMessage,IsSystemTable,CreateTime
Foo,DYNAMIC,null,null,null,"Snapshot, Continuous, Alert Rules, Delete, 
  Top-N Queries, Time-windowed Queries, Time-delayed Queries, Aggregate Queries, 
  Publish, Can be externally hosted, Calculated Columns",LIVEVIEW,x,DISABLED,
  Table is being removed.,false,2015-09-30 11:33:53.060-0400

Also, when you drop a table that was created but not started, its LVTables row and associated LVTableColumns rows are removed.

Error Conditions

Error conditions can occur when creating or starting tables. There are some createTable() error conditions that throw an exception from the call with no corresponding system table entries created or changed. However both createTable() and startTable() can return successfully, and it is not until sometime later in the create and start process that errors are detected. If an error occurs with a table after it has had an LVTables row added, that row is updated with a StatusENUM of ERROR, and the StatusMessage provides some helpful text on the nature of the problem. For example, creating a table with an index field that is not present in the table schema, using a command like the following:

lv-client createtable --name Bar --schema "\"(x int, s string)\"" --keys y
lv-client starttable Bar

produces LVTables entries like these:

Name,Group,ShortDescription,Description,TableSpace,Capabilities,QueryLanguages,
  Indices,StatusENUM,StatusMessage,IsSystemTable,CreateTime
Bar,DYNAMIC,null,null,null,"Snapshot, Continuous, Alert Rules, Delete, 
  Top-N Queries, Time-windowed Queries, Time-delayed Queries, Aggregate Queries, 
  Publish, Can be externally hosted, Calculated Columns",LIVEVIEW,x,DEFINED,
  "Defined, but not loaded",false,2015-09-30 11:48:30.625-0400
Bar,DYNAMIC,null,null,null,"Snapshot, Continuous, Alert Rules, Delete, 
  Top-N Queries, Time-windowed Queries, Time-delayed Queries, Aggregate Queries, 
  Publish, Can be externally hosted, Calculated Columns",LIVEVIEW,y,ERROR,
  Invalid field reference 'y',false,2015-09-30 11:48:30.625-0400

The ERROR LVTables row remains until the client issues a dropTable() call.

Legal State Transitions Shown in LVTables

The following table shows the legal states that can be reached for each current state of the StatusENUM field of the LVTables system table:

Current State Next States API Call and Description
DEFINED LOADING startTable() — Table is being compiled.
  ERROR startTable() — Table was not successfully compiled.
  NA dropTable — Table was dropped and is gone.
DISABLED NA NA — Table is gone.
ENABLED DISABLED dropTable — Table is disabled and being removed.
ERROR LOADING startTable — Table was started again and is trying to be compiled.
  NA dropTable — Table was dropped and is gone.
LOADING ENABLED NA — Table is ready for use.
  ERROR NA — Table was not loaded successfully.
NA DEFINED createTable — Table was created successfully.
  ERROR createTable — Table was not created successfully.

Steps to Add a Table While Monitoring LVTables

Follow these steps:

  1. Create a table using from the command line or a Client API call.

  2. Register a live query against the LVTables table. You can restrict the the results to the table of interest by using a predicate in your query such as where Name=="YourTableName".

  3. Confirm that the row associated with your newly added table has a StatusENUM of DEFINED. If it has ERROR, you must drop the table, fix the problem, then try again.

  4. Start the table.

  5. Confirm the row associated with the newly added table has a StatusENUM of LOADING, followed by an update to ENABLED.

    Once a LiveView table has transitioned to the ENABLED state, it does not change its state again unless there is a subsequent dropTable() call. However, if the state changes to ERROR, you must drop the table, fix the problem, then try again.

Using TableListListener

The TableListListener interface is an API that provides client callbacks when a table's state changes. LiveView tables issue transition events as they are dynamically created, started, and dropped. See the Javadoc for this API for details on using it.