Setting up the Failover (New HA Pair)

Both appliances need to be installed with compatible versions of the LogLogic software.

Prerequisites

The failover membership cannot be established if there is a mismatch in software versions between the two nodes. Both the software need to have the same:
  • HA protocol version
  • Log storage format version
  • LogLogic LSP installed version
  • Database schema version

Procedure

  1. Set the NTP server on both appliances that will be in the HA pair.
    1. Log in to the GUI web server using the private IP address of the appliance.
    2. Change the administration password, if required.
    3. Configure the appliance time through the Administration > System Settings > Time tab.
      It is good practice to use the network time protocol (NTP) for failover configurations. Setting the NTP server on one appliance in the HA pair automatically sets it on both appliances and reboots them both accordingly.

      It is essential to set the NTP server time correctly during appliance installation.

  2. From the CLI, through a serial console, log in to the appliance that is the active node in the HA pair.
    Note: Use a serial console instead of an SSH connection. Network configuration changes can disconnect an SSH connection during this procedure. In all circumstances, a serial connection is maintained.
  3. Configure the appliance with a private IP address:
    set ip <IP address> <netmask> <gateway> [ifdev] [defaultgw]

    For example, using 10.1.1.71 as the private IP address and 10.1.1.1 as the gateway:

    > set ip 10.1.1.71 255.255.255.0 10.1.1.1 bond0
    New interface settings:
    ip 10.1.1.71 255.255.255.0 10.1.1.1 
    CHANGES HAVE NOT BEEN SAVED!
    > save
    >
    Note: If you configure more than one interface in the set ip command, make sure the subnets do not overlap.
  4. Configure failover:
    1. Ensure that Advanced Features are disabled on both master and slave appliances.
    2. Provide the public IP address of the failover and the private IP address of the second appliance by using the set failover configure command.
    Note: Make sure that the subnet in the set failover configure command is the same as the one configured for the same interface in the set ip command.

    Also, if you specify an interface it must be an existing interface configured with a different IP address in the set ip command.

    For example, using 10.1.1.177 as the public IP address and 10.1.1.72 as the private IP address of the other appliance:

    In case of IPv6, use :: as the value of <broadcast>.

    > set failover configure
     
    CAUTION:
     
    You will be prompted to designate one appliance to be the destination of the
    initial data migration ("destination appliance").
     
    All log data from the source appliance will be copied to the destination
    appliance. In cases where both appliances contain data for the same time
    periods the data from the destination appliance will be overwritten. Data
    from time periods not present on the source appliance will be deleted from
    the destination appliance.
     
    Should the resulting volume of data exceed the capacity of the destination appliance,
    the oldest data will be deleted or archived, as needed. If the source appliance is not
    configured to use an external storage system, then the destination appliance will be
    configured not to use one either.
     
    To minimize such storage space issues, it is best to use appliances of the same model
    and that have enough space for the entire data set.
     
    Enter the public Ip for the HA partner pair
    in the form <ip> <netmask/prefix> <broadcast> <ifdev>, or 0 to cancel:
    >> 192.168.1.248 255.255.255.0 192.168.1.255 eth0
    This appliance shall be the destination of the initial data migration [Y/N]:
    >> n
    Enter the Ip address of the peer appliance in the form <ip>:
    >> 192.168.1.249
     
    CHANGES HAVE NOT BEEN SAVED!
    > 
  5. Save the changes to apply the new configuration:
    > save
     
    Writing changes to disk...
    Generating new SSL certificate...
    Generating RSA private key, 2048 bit long modulus
    ........................+++
    ........+++
    e is 65537 (0x10001)
    Generating RSA private key, 2048 bit long modulus
    ..........................+++
    .....................+++
    e is 65537 (0x10001)
    Using configuration from /loglogic/conf/certs/llssl_.cnf
    Check that the request matches the signature
    Signature ok
    The Subject's Distinguished Name is as follows
    countryName           :PRINTABLE:'US'
    stateOrProvinceName   :PRINTABLE:'California'
    localityName          :PRINTABLE:'Palo Alto'
    organizationName      :PRINTABLE:'Cloud Software Group, Inc.'
    organizationalUnitName:PRINTABLE:'MiniHTTPS'
    commonName            :PRINTABLE:'192.168.1.248'
    emailAddress          :IA5STRING:'support@tibco.com'
    Certificate is to be certified until May 12 01:23:11 2028 GMT (3650 days)
     
    Write out database with 1 new entries
    Data Base Updated
    Creating mysql replication user repl_ll_601 
    Granting ALL PRIVILEGES on all nodes to repl_ll_601.
    Granting ALL PRIVILEGES on node 192.168.1.245 to root
    Granting ALL PRIVILEGES on node 192.168.1.249 to root
     
    STOPPING MASTER TASK...............................................(ok)
     
    [writing new cluster configuration]
     
    STARTING MASTER TASK...(ok)
    done.
  6. Repeat this procedure on the standby appliance.
    Warning: On the standby appliance, in Step 4 indicate Y for the appliance to be the destination of automatic migration.

    If both appliances are configured as destinations, or if neither appliance is, the HA pair does not form. Both appliances report that they are out of cluster.

  7. Confirm that the NTP server settings, and the actual observed time, are identical on both appliances in the HA pair.
    It is essential to set the NTP server time correctly during appliance installation.
  8. (Optional) Use the following commands related to Advanced Features in HA mode, as required. You must be logged in as the root user. For more information on the system command, see the system command section.
    • To enable Advanced Features: >system logu enable
    • To disable Advanced Features: >system logu disable
    • To show the Advanced Features status: >system logu status
    • To enable Monthly Index: >system monthly_index enable
    • To disable Monthly Index: >system monthly_index disable
    • To show the Monthly Index status: >system monthly_index status
    Important:
    • Monthly Index can be enabled only if Advanced Features are enabled.
    • In HA configuration, Advanced Features and Monthly Index can be enabled only through the CLI, and the configuration can be made only on the master node. Before HA configuration, disable Advanced Features on both master and slave appliances.
    • The zookeeper_sync engine performs the sync from the master to the slave node.
    • Enabling advanced features causes the mtask engines to reboot.

Result

The failover is now set up and both appliances are synchronizing their data.

You can log into the active node using the public IP address of the failover to finish appliance configuration. During normal operations, it is good practice to use the public IP address of the failover for configuration changes and the private IP address of the standby to run reports as this leaves the active node fully available for collecting and processing logs. The standby is always the same and only changes in case of failover. The Administration > System Settings > Network tab always shows the private IP address of an appliance.