Healthcheck for DASH High Availability Setup

Body

There are many time or scenario that you need to check the status of your DASH HA clusters after you had perform fix pack upgrade or stop the server for maintenance.

In the normal situation, these results will be shown if the clusters are working properly.

Run the JazzSM/profile/ui/bin/consolecli.sh ListHANodes --username smadmin --password smadmin

Scenario 1

If the ListHANodes are showing Java null error, please do ensure that your Db2 server is reachable and is started properly via the db2start command

Scenario 2

If the LIstHANodes are returning ACTIVE and InSync but pages or roles are not sync properly between servers, then please check the Systemout.log for any SSL Handshake errors.

Please refer to this technote to fix the SSL certificated issue.

http://www-01.ibm.com/support/docview.wss?uid=swg21675805

If your ListHANodes return error message,

Scenario 3

For Unreachable message, make sure that your JazzSM server is started properly via the startServer.sh server1 command and that you can login to the DASH URL

link via https://<yourhostname>:16311/ibm/console

Scenario 4

Is the LIstHANodes are returning INACTIVE and NodeSync are showing Unreachable and the DASH URL can be accessed, please check if your server name have FQDN.

It cannot be only server1.test and should be server1.test.ibm.com in the host file name and JazzSM_HOME/profile/config/cells/JazzSMNode01Cell/Nodes/JazzSMNode01/serverindex.xml file

Scenario 5

If the NodeSync is showing INACTIVE,

You can perform these command to refresh the node with the latest content from database.

The ForceHARefresh exports data from the database and imports it locally. The database module version for Dashboard Application Services Hub must be lower than local node for export and import to succeed. Use this command to upgrade a cluster when there are no nodes that are in synch with the database module versions. Use the ForceHARefresh command, verify the result, and then use the ForceHAUpdate command to update the database.

  ./consolecli.sh ForceHARefresh --username smadmin --password netcool

To force a database update after running the ForceHARefresh command, as an administrator, run the ForceHAUpdate command. The ForceHAUpdate command pushes the local configuration to the database and updates the modules table to match the local node's module versions. Notifications are sent to other nodes to synchronize. Notified nodes with module versions that match those of the originating node are synchronized. Notified nodes with module versions that do not match, go into maintenance mode until an administrator updates their modules accordingly.

  ./consolecli.sh ForceHAUpdate --username smadmin --password netcool  

Scenario 6

If you encounter these problem where you can't create a page on the cluster, you would need to run this command

for example

  JazzSM/profile/ui/bin/consolecli.sh Unlock --username smadmin --password netcool

Additionally, you can also check if these tables are created and updated properly in the DB2. Please use the db2inst1 user that is used to create this Db2 database.

Enter Db2 on the Unix command line and then connect to the database used for DASH. In the following we assume that database name is DASH:

  db2inst1:> db2    db2 => connect to DASHDB    Run below command to list tables in the database. You should see below tables.    db2 => list tables    GLOBAL_CONFIG  MODULES  NODES  NODES_STORES  STORES

Below SQL can be used to list the number of nodes in the cluster.
 

  db2 => SELECT * FROM NODES

It should return the numbers of nodes cluster with the right IP hostname that you had configured the clusters servers.