Advanced Tasks in LiveView

Increasing Linux Limit on Concurrent Open Files

By default, user accounts on Linux have a setting equivalent to ulimit -n 1024, which caps the maximum number of concurrently open file descriptors to 1,024 for all processes launched concurrently by that user. However, LiveView Server opens a new file descriptor for each new services-layer connection. The upper boundary set by LiveView is 3000, which is higher than the default per-user LInux value.

You may want to increase the ulimit setting for the user accounts running LiveView Server, to allow for up to 3000 connections allowed by LiveView, plus an additional number to account for network socket connections held open concurrently by StreamBase EventFlow applications in the same server, plus all other concurrent processes. A good starting value when running LiveView Server on Linux is to specify ulimit -n 4096 for the user account that will run the server. You may need to adjust this number higher.

Changing Character Encoding

By default, TIBCO LiveView and TIBCO StreamBase use the platform's default character encoding when reading files or communicating over certain network protocols. If you require another character encoding, you can change the default settings as follows. This example uses UTF-8 as the new character encoding.

Setting Character Encoding in StreamBase Studio

To set the file encoding for StreamBase Studio, edit the system properties for file.encoding and streambase.tuple-charset. These properties are set locally in a project's sbd.sbconf file.

Edit the sbd.sbconf file for your project as shown:

<java-vm>
  <sysproperty name="file.encoding" value="UTF-8"/>
  <sysproperty name="streambase.tuple-charset" value="UTF-8"/>
</java-vm>

This tells all StreamBase processes to emit tuples in UTF-8 encoding.

Setting Character Encoding Read From System Files

To set the default Java encoding for reading character data from files, set the environment variable STREAMBASE_STUDIO_VMARGS. In Windows, you can do this at the StreamBase Command Prompt with the set command (this one-line command is shown on two lines for clarity):

set STREAMBASE_STUDIO_VMARGS=-Dfile.encoding=UTF-8 -Dstreambase.tuple-charset=UTF-8 
     -Xms256m -Xmx1500M

This environment variable changes the default Java encoding when reading character data from files (file.encoding) and instructing the API to transfer character data as UTF-8 (streambase.tuple-charset). Start StreamBase Studio with the environment variables you just changed by using the command-line tool from the same StreamBase Command Prompt:

sbstudio

Once the STREAMBASE_STUDIO_VMARGS variable is set in StreamBase Studio, the character encoding for LiveView server will also be set to the new encoding.

Setting Character Encoding for LiveView Desktop

To tell TIBCO LiveView Desktop to read the new character encoding, start LiveView Desktop with the -vmargs argument. To do this at the command line, use this command:

lv-desktop -vmargs -Dstreambase.tuple-charset=UTF-8

This tells LiveView Desktop that incoming tuples are encoded as UTF-8.

Note

If your application uses a Feed Simulation that reads files and you start the simulation from outside StreamBase Studio, you need to give the same options to the simulation. For example, to use UTF-8, use the following:

sbfeedsim -J-Dfile.encoding=UTF-8 my-feedsim.sbfs

Connecting to LiveView with sbc

Readers familiar with the StreamBase environment may be comfortable connecting to StreamBase Server from the command prompt with the sbc command. Those readers can still use sbc to connect to LiveView Server as a basic way to enqueue data into a running LiveView table.

To test the example in this section, load and run the LiveView Minimal sample as described in Starting LiveView Server from StreamBase Studio.

The following information will clarify some of the important features of LiveView Server tables:

Each LiveView table is defined in a single LiveView configuration file. The basename of the lvconf file and the table name must be the same.
When LiveView Server loads the configuration file and generates one LiveView table per lvconf file, it places each table in its own container. The container's name is always the same as the table. Thus, the Minimal sample defines one table named Minimal that LiveView Server places in a container also named Minimal.
Each table container has exactly one input stream, always named DataIn.
Since we know the name of the container for each table, we know the StreamBase path to the DataIn stream for each table. For the Minimal sample, this path is Minimal.DataIn.

Using this information, we can obtain a description of the schema for the Minimal.DataIn stream using the sbc describe command:

sbc describe Minimal.DataIn

This returns the following output:

<stream input="true" name="Minimal.DataIn" schema="DataSchema" 
        uuid="E31A63B852D8DD9D15B0BD6CD1BA88B2">
  <schema name="DataSchema" uuid="4AB4515006E401E64FFED374ABD55947">
    <field description="" name="PublisherID" type="string"/>
    <field description="" name="PublisherSN" type="long"/>
    <field description="" name="CQSReferredCount" type="long"/>
    <field description="" name="CQSDataUpdatePredicate" type="string"/>
    <field description="" name="CQSDelete" type="bool"/>
    <field description="" name="pkey" type="string"/>
    <field description="" name="val" type="string"/>
  </schema>
</stream>

LiveView Desktop shows us only two fields for the Minimal table, but sbc describe shows us seven fields. The first five fields are required by the inner workings of LiveView, and are handled internally by LiveView Desktop. To enqueue data manually to a running LiveView table with sbc enqueue, we must specify values for all seven fields. We can use any values that match the specified data types for the two Publisher* fields, but you must set CQSReferredCount and CQSDataUpdatePredicate to null, and must set CQSDelete to false.

The two fields of interest hold a generic key-value string pair. Start LiveView Desktop and connect to the running Minimal sample as described above, and double-click to open the Minimal table.

Then open a terminal window on OS X or Linux, or open a StreamBase Command Prompt on Windows, and type the following commands in sequence, watching the results in the LiveView Desktop window. This example uses "sbc" as an arbitrary string for the PublisherID field, and arbitrarily increments the second field by one for each input line. As required, the example sends null and false to the three CQS fields. The last two fields, pkey and val, hold the values that show up in LiveView Desktop:

sbc enqueue Minimal.DataIn
sbc, 1, null, null, false, City, London
sbc, 2, null, null, false, Nation, France
sbc, 3, null, null, false, City, Paris
sbc, 4, null, null, false, Region, Buckinghamshire
sbc, 5, null, null, false, City, "High Wycombe"
sbc, 6, null, null, false, Nation, England
...

Continue entering geographical information or switch to another set of related key-value pairs, as you prefer. Rows with the same pkey primary key field replace previous rows with the same pkey text. For the entries above, the Minimal table in LiveView Desktop shows the following results.

Setting Read-Only Alert Rules

Alert rules can be set to read-only status. This is useful if you want alert rules to be preserved for multiple clients. Follow these steps:

Right-click in the project's root directory and select New → StreamBase Server Configuration File. This opens the New StreamBase Server Configuration File dialog.
Select the root directory for your project.
Enter sbd.sbconf in the Server Configuration file name field.
Click Finish. This creates an XML file with root element <streambase-configuration> and opens the file in StreamBase Studio.

Copy the following text into the file:

<java-vm>
  <sysproperty name="liveview.alert.disable.changes" value="true"/>
</java-vm>

Save this file and run the project. When LiveView Server next runs, directives in the sbd.sbconf file that you just created is automatically merged into the primary configuration file for the server. Thus, once LiveView Server starts, any non-local host client sessions that connect to the LiveView Server will receive a "Not authorized" message if the client tries to edit configured alert rules. Clients running on the same machine as the server process (localhost) are allowed to change alert rules.

Disable Alert Rules

Alert rules can be disabled with a system property so configured alert rules will not run with LiveView starts. This is useful if you have created an alert rule that might damage your system. The system properties liveview.alert.register.startup and liveview.alert.enabled control whether a project's alerts are registered and whether the alert service starts when the server starts. By default, these properties are set to true. If you set these properties to false, you can start LiveView Server without your configured alerts, and still use StreamBase Studio to edit or disable the alerts. To set either of these properties to false, follow these steps:

Right-click in the project's root directory and select New → StreamBase Server Configuration File. This opens the New StreamBase Server Configuration File dialog.
Select the root directory for your project.
Enter sbd.sbconf in the Server Configuration file name field.
Click Finish. This creates an XML file with root element <streambase-configuration> and opens the file in StreamBase Studio.
To prevent configured alerts from being registered at startup, copy the following text into the file under the java-vm tag:
```
  <sysproperty name="liveview.alert.register.startup" value="false"/>
```
Setting this property to false means that configured alerts will not be registered. This means that you can start the server and edit or disable the problem alerts.
To prevent the alert service from starting, copy the following text into the file under the java-vm tag:
```
  <sysproperty name="liveview.alert.enabled" value="false"/>
```

Using Parallelism and Concurrency

LiveView Server has the ability to process table queries in parallel, but does not do so by default. You can customize the parallelization and concurrency for each table to improve query performance. However, it is possible to over-specify these features and thereby overload the hardware that runs your LiveView Server instance. Thus, you must use the configuration elements described here with caution and careful testing.

In the lvconf file that defines each LiveView table, there are two attributes of the table's root element, snapshot-parallelism and snapshot-concurrency, that control the parallelism and concurrency of query processing. You can also specify these settings in a Table Space type table, then specify that Table Space in a <table-space-ref> element in the lvconf for some of the data tables in your project.

Snapshot parallelism determines the number of data regions used in parallel to publish to and scan from tables. Each data region contains approximately 1/N of the total rows in the table, where N is the snapshot-parallelism value. Use snapshot parallelism to improve load performance and query performance where a query needs to scan many rows of a table.

In general, a higher snapshot-parallelism value means that individual table scan snapshot queries can run faster — if your server has available CPU cores to support the value you specify.

Snapshot concurrency specifies the number of extra threads used to service the snapshot portion of queries. By default, snapshot concurrency is not enabled, which means that LiveView Server's single data-region thread also services all snapshot queries. Setting the snapshot-concurrency attribute to X means there will be X extra independent threads for snapshot processing. Additional snapshot query threads are most beneficial when you have ad hoc queries that cannot use indexes and your table size is several hundred thousand rows or larger.

For example, setting snapshot-concurrency=1 for a data table results in two threads: one thread available to run snapshot queries, while the default data-region thread remains available to handle data being published and to handle all continuous query processing.

With no snapshot-concurrency setting, or by setting it equal to the default of 0, only one task operates at a time. That task could be an update or a snapshot scan. With snapshot-concurrency set to 2, for example, up to three tasks can run simultaneously. These three tasks would be one publish and continuous query process, and two snapshot table scans.

Your LiveView deployment should have more CPU cores than the aggregate total number of (snapshot-parallelism times snapshot-concurrency). The ideal configuration provides one core for every (snapshot-parallelism setting times snapshot-concurrency setting times the number of LiveView data tables that will be actively queried). For snapshot-parallelism=2 and snapshot-concurrency=1, the ideal LiveView Server would provide 2 * 2 = 4 cores for each active LiveView data table.

Using Data Conflation

LiveView allows you to control the rate of data outflow from a source table to an aggregation table. This is called data conflation. The publish-interval-millis attribute of the <data-table> tag, if set, adds latency and reduces volume by limiting output from the base table to the aggregate table to the specified publication rate. When set, the update delivers only the current row value (at most one row per pkey) instead of "as fast as the data arrives." Updates into the base table still happen at the as fast as data arrives.

For tables with an <aggregation> data-source only (not other data-source types) you can add conflate-data=true which only changes how often the aggregate-processor emits results into the table. This period is synchronized with publish-interval-millis. With the default behavior of conflate-data=false, the aggregate-processor emits frequently from the source table. With conflate-data=true the last aggregate result per pkey is delivered to the table to be processed by the <insert-rule> and <update-rule> expressions.

Calculating Customer Data Under Management

If you license a TIBCO^® Suite that includes Live Datamart as a Suite Component, then your license for TIBCO Live Datamart includes a limit on the amount of customer data under management. This is the sum total of your data made available to client applications served by TIBCO Live Datamart. The exact limit in gigabytes of customer data under management authorized by your Suite license is stated on your TIBCO Order Form.

You can track the amount of your customer data served by TIBCO Live Datamart with a simple query sent to the server. You can issue the following query from any client application, including TIBCO LiveView™ Web, TIBCO LiveView™ Desktop, the lv-client command line tool, or from a client application you write using the TIBCO LiveView™ Client APIs in Java or .NET (on Windows only):

select sum(MBMemoryUsed) as UserDataUnderManagement from LiveViewStatistics

This command returns a single integer value in megabytes.

To determine which of your TIBCO LiveView tables is using the most data, view the LiveViewStatistics system table in a client application. The MBMemoryUsed column shows the approximate data for each table in megabytes.

Java Garbage Collection Settings

Live Datamart Server releases before 2.1.0 explicitly specified the ConcurrentMarkSweep (CMS) collector as its Java garbage collector.

Live Datamart Server in releases 2.1.0, 2.1.1, and 2.1.2 do not specify an explicit Java garbage collector, which means the garbage collector setting falls back to the JVM's default. Live Datamart versions since 2.1.0 use JDK 8 by default, whose default garbage collector is the throughput collector.

To use the G1 garbage collector instead of the throughput collector with releases 2.1.0, 2.1.1, and 2.1.2, add the following lines to the sbd.sbconf file at the root of your Studio project:

<java-vm>
  <param name="jvm-args" value="
    -XX:+UseG1GC
    -XX:MaxGCPauseMillis=500 "/>
</java-vm>

Starting with Live Datamart Server 2.1.3, the G1 garbage collector is explicitly specified, using internal settings equivalent to the following:

<java-vm>
  <param name="jvm-args" value="
    -XX:+UseG1GC
    -XX:MaxGCPauseMillis=500 
    -XX:ConcGCThreads=1 "/>
</java-vm>

If you specify a different JDK to run Live Datamart, that JDK's default garbage collector is used. For example, if you use the Azul Zing JDK, it has its own garbage collection implementation.

If you have a site policy that specifies using a different Java garbage collector than the defaults specified above, contact TIBCO Technical Support for assistance.