Installing and Configuring ibi WebFOCUS DSML Services

This topic describes the installation and configuration of ibi™ WebFOCUS® DSML Services.

ibi WebFOCUS DSML Services Products

The following WebFOCUS products are included in DSML Services:

  • Instant Insights. You can easily run advanced analyses and generate visualizations and narratives on your data sets, without manually preparing and analyzing your data, or having prior knowledge of data science or statistics. With one click, the Instant Insights capabilities in Designer recognize trends in your data, generate customizable visualizations as charts with natural language headers, and categorize the charts into tabs. You can add these charts to the Designer canvas and build a page, or add them to existing pages or dashboards. Then, you can save and share your insights with others. For more information on Instant Insights, see the ibi™ WebFOCUS® User Guide.
  • Machine Learning Functions. When creating a Data Flow, you can easily run predictive analytics on your data sets using Machine Learning functions, without prior knowledge of advanced statistics. Build, train, and run multiple iterations of predictive models in parallel, evaluate and compare models actively, and select which model you want to save. Then, you can re-run your model against new data sets. For more information on Machine Learning Functions, see the ibi™ WebFOCUS® User Guide.
  • Metadata Classification. Examines your data and assigns classifications to the columns, which can then be used to match columns from separate data sources. You can classify data that you upload, and use it to match fields in a Union in a Data Flow. When uploading a data file, you can view the recommended metadata classification values for each character-valued column. You can choose to keep the recommended values, or to change them. Including classification values in your data improves the accuracy of mapping column tables correctly to each other. This is useful if you are integrating data from multiple sources, or if integrating data into a system with a predefined hierarchy. For more information on Metadata Classification, see the ibi™ WebFOCUS® User Guide and the ibi™ WebFOCUS® Reporting Server Administration manual.
  • Natural Language Query (NLQ). Lets you ask questions about your data using everyday language. This provides you with valuable insights and allows you to make informed business decisions. NLQ translates natural language into SQL code that can be run against a database. The natural language phrases are matched with relevant database schema columns within your data, and your query results display as a table.

    Examples of natural language queries include:

    • What is my total revenue?
    • Give me sales by product name.
    • Show me patients by doctor.
    • Show me offices by city and country.
    • How many JVC models did I sell in 2020?

ibi WebFOCUS DSML Services Installation Requirements

Important:

  • DSML Services can only be run on an Intel x86_64 Linux Ubuntu system, Release 20.04 or higher.
  • The Unbuntu libraries must be installed. You can run the following command to install the libraries: 
    sudo apt-get install libxrender1 libxtst6 libxi6 unzip
  • The DSML Services installation is only available for the English language.

Hardware Requirements

The following are hardware requirements for DSML Services:

  • Memory. Minimum of 16 GB physical memory (RAM) on the system, with most of it free for DSML usage.
  • CPU. Minimum 4-core processor.
  • Disk space. Minimum of 100GB free space on the disk where DSML will be installed.

Note: The installer detects the hardware requirements and does not allow the installation to continue if the requirements are below minimum requirements.

ibi WebFOCUS DSML Services Installation Components and Steps

ibi™ offers two ways to install DSML:

  • Using binary components.
  • Using scripts that ibi™ provides to create the Docker container. For more information, see the ibi™ WebFOCUS® Container Edition Installation and Deployment Guide.

Install DSML Using Binary Components

    Procedure
  1. From the eDelivery site at https://edelivery.tibco.com/storefront/index.ep, download the installer for DSML Services.
  2. If the LANG environmental variable is not already set to en_US.UTF-8, issue the following command: 
    export LANG=en_US.UTF-8
  3. Start the installer. For example: 
    IBI_dsml_release_number_linux548_x64.bin

    The installation lays down the following four binary components of DSML, under the install_root/ibi/dsml/bin/ directory:

    • instant-insights
    • metadata
    • ml-functions
    • dsml-nlq
  4. DSML Services run under four HTTP listeners, included in the four binary components.

    You can customize the install_root/ibi/dsml/conf/run_dsml_services.sh shell script to change the four ports and worker counts (except for NLQ), as necessary. These four HTTP listener end points are hidden to the end user and the WebFOCUS Reporting Server software.

    Note: The NLQ listener is dependent on the following:

    • The NLP library is running on port 9000.
    • The training model directory is available under the install_root/ibi/dsml/bin/directory.
  5. You need to configure the NGINX Reverse Proxy to hide the four HTTP listeners and to expose only one HTTP listener. If you change the port numbers in the shell script, you need to make the corresponding changes to the port numbers in the install_root/ibi/dsml/conf/dsml.conf file. You can make any other necessary changes for configuring NGINX in the dsml.conf file, or accept the default values.
    • To install and configure NGINX on the system, you need the sudo or root privilege. If NGINX is not installed, then NGINX needs to be installed as sudo or root. Typically, NGINX is found under the /etc/ngnix directory.

      Provided there is Internet access on the system, and you have the sudo or root privilege, you can run the following command to install NGINX:

      sudo apt install nginx

      For more information on installing NGINX, see:

      https://www.nginx.com/resources/wiki/start/topics/tutorials/install/

    • If you do not have the root or sudo privilege, or internet access, then assistance is required from the IT department to get NGINX installed on the system. Once NGINX is available on the system, do the following as root or with the sudo privilege:

      1. If you already have processes running for the four components: instant-insights, metadata, ml-functions, and dsml-nlq, find them using the ps command, and stop them.
      2. If you want to customize ports, edit the nginx.conf file located in the nginx/conf directory.
      3. Issue the following command to remove the /etc/nginx/site-enabled/default file, if it exists:
        sudo rm -f /etc/nginx/sites-enabled/default
      4. Issue the following command to copy the dsml.conf file to the /etc/nginx/site-enabled/default file:
        sudo cp -f install_root/ibi/dsml/conf/dsml.conf /etc/nginx/sites-enabled
      5. Issue the following command to restart NGINX:
        sudo systemctl restart nginx
      6. Issue the following command to change to the full path of the conf directory:
        cd install_root/ibi/dsml/conf
      7. Issue the following command to run DSML Services:
        ./run_dsml_services.sh

    Note:

    • Running the ./run_dsml_services.sh script attempts to download the necessary libraries required by dsml-nlq (refer to the get_model.sh (training model) and get_nlp.sh (nlp Java library) scripts for details on how to download the library components manually). Provided you have access to the internet, the script runs one of the libraries (nlp) using Java provided by the installation. If the provisioning of the libraries fail, then this script will not run dsml-nlq, and you will have to manually provision the libraries using the run_nlp.sh and get_model.sh scripts and then bring up dsml-nlq separately, using the start_nlq.sh script.
    • If you do not have an internet connection or have restricted access, then the provisioning of the libraries will fail. In such cases, you will need to provision the libraries on another machine with internet access and then copy the libraries over to the following locations:
      • nlp library contents must go under the dsml_home/lib/corenlp/src directory.
      • training model library contents must go under the dsml_home/bin directory.

      Now, you can run the run_nlp.sh script to bring nlp Java program required by dsml-nlq and then you can run the start_nlq.sh script.

ibi WebFOCUS DSML Services Directory Structure

The following table describes the DSML directories created by the installation. These directories are created in the install_root directory, where install_root can be any writable path for the user, with at least 100 GB of free disk space. The default directory is $HOME/ibi/dsml/.

Directory Description

install_root/ibi/dsml/Uninstall/

Contains the files used by the DSML uninstall program.

install_root/ibi/dsml/bin/

Contains the four binary components:

  • instant-insights
  • metadata
  • ml-functions
  • dsml-nlq

In addition, the bin directory will contain training model libraries (distilbert-base-nli-stsb-mean-tokens).

install_root/ibi/dsml/conf/

Contains the DSML script files:

  • dsml.xml
  • run_dsml_services.sh

install_root/ibi/dsml/doc/

Contains the readme file, readme.txt.

install_root/ibi/dsml/jdk/

Contains the Java included in DSML that is used by the installer.

install_root/ibi/dsml/lib/

Contains the corenlp/src NLP library.

install_root/ibi/dsml/logs/

Contains the DSML log files.

Connecting to the ibi WebFOCUS DSML Microservice

Assuming NGINX is running under port 80, there should be one HTTP URL that will go into the WebFOCUS Reporting Server configuration for it to connect to the DSML microservice.

You can test whether your four DSML components are accessible by issuing the following commands:

http://hostname/machinelearning/v1/system

Machine learning sample response:

{"data":{"buildDate":"2023-02-03T15:00:15Z","gitCommit":"25e84c85821",


"version":"1.1.0"}}

http://hostname/metadata/v1/system

Metadata sample response:

{"data":{"buildDate":"2023-01-24T15:07:27Z","gitCommit":"6a1b9f631e4",


"version":"1.0.0"}}

http://hostname/autoanalytics/v1/system

Autoanalytics sample response:

 {"data":{"buildDate":"2023-01-23T05:26:00Z","gitCommit":"c9707ae9b8e",
"version":"1.6.2"}}

http://hostname/nlq/v1/system

NLQ sample response:

{"data":{"buildDate":"2023-02-03T04:30:54Z","gitCommit":"830088e97af",


"version":"1.1.0"}}

Note: The py_serv URL is the URL for all services. Add the following pyserv_url command to the server configuration file (edaserv.cfg). The format is:

pyserv_url=http://hostname

Note: The installation setup is finished and you should now be able to use DSML Services.