You can configure external library storage in the cloud using
Amazon Web Services S3 (AWS), Azure Blob Storage, or Google Cloud Storage. You
can also configure external library storage in a file system.
Before you begin
- You must have
administrative credentials for Spotfire Server.
- Depending on which
solution to use, you also need one of the following:
- To use AWS, you must
have an Amazon S3 account and a bucket name. Every server database (or database
cluster) should have its own bucket.
- To use Azure, you
must have an Azure Blob Storage account.
- To use Google, you
must have a Google Cloud Storage account.
- File system storage
is primarily intended for test systems. In addition to the information in
Library content storage outside of the Spotfire
database, you should be aware that the file system operations have no
transactions, which might lead to corruption of files.
Items stored in external systems are identified by their data IDs. If
different servers use the same bucket or container, importing files to Cluster
B—when the files already exist in Cluster A—will overwrite the files in Cluster
A. To share the same bucket/container between clusters you can use
--key-prefix, with different values for different
clusters.
To reduce Spotfire environment downtime while migrating library
storage to an external system, you can use the
migrate-library CLI command, as described
below. With this method, you can perform parts of the migration in advance and
plan the required downtime to a time of your choice.
Note: The steps below must be performed in the right order.
Note: The final cleanup step can cause a temporary but considerable
strain on the database.
Procedure
-
Prepare the configuration for the external library storage using
the appropriate CLI command.
-
Export the server configuration file with
export-config, as described in
Executing commands on the command line.
-
Configure your external storage using one of the following
commands:
-
Import the configuration file with
import-config.
-
Copy the current library content from the database to the external
storage system using the
migrate-library command.
This command can be run while the Spotfire environment is
running, to avoid downtime. By default, the command runs in a loop until no
more changes are detected, to keep up with any changes to the library. Before
switching, it should be run one final time. Note that this command can be
run on any computer that has access to the
database.
Note: Before the final execution of the command you should take all
Spotfire servers offline, to avoid any inconsistencies in the final migration.
Example
1:
# Uncomment the first row below if the default temp folder is unable to hold large library files.
# $ export CONFIG_JAVA_OPTS=-Djava.io.tmpdir=</path/to/larger/disk>
$ config.sh migrate-library --external-storage=<configured storage>
-
When you are sure everything is migrated and the servers are
offline, enable the external storage as follows:
-
Export the server configuration file with
export-config, as described in
Executing commands on the command line.
-
Use the
config-library-external-data-storage command
to switch to use your previously configured external storage. This command also
does a basic consistency check of the migrated content.
-
Import the configuration file with
import-config.
-
Verify that there are no inconsistencies between the external
storage and the Spotfire database.
Example:
$ config.sh check-external-library
-
Start the Spotfire servers.
-
Clean up the database by running the
cleanup-migrated-library CLI command, to
remove all library content from the database.
Note that this step might take a long time. It will also require
a larger than usual database transaction log space, or other temporary
resources, depending on the number of library items you have and how large the
total content is.
Results
The migration to
external library storage is complete.
You can migrate back to the database using the
migrate-external-library command.
Comment: It is also possible to do a migration by first backing up the
database, then using the export-library-content command, remove all content
from the database, configure config-library-external-data-storage, and use one
of the configuration commands from step 1b above, and finally import the
library again. However, remember that this might take a lot of time and disrupt
the work for your users.
1 By default, the migrate-library command will run as
many background threads as there are cores on the computer. This is important
to take into account for the temp folder mentioned in the example, as well as
when taxing the database (one thread ≈ one database connection).