IBM Support

QRadar: Software update checklist for administrators

Question & Answer


Question

What steps can administrators review before they attempt to update their QRadar deployment?

Answer


Important notes before you update to QRadar

The sections listed on this page contain specific alerts administrators can review before they upgrade. For a more comprehensive list of fixes and issues, see QRadar Software 101.
 

QRadar 7.5.x alerts

  •   UP8 Upgrade check: Administrators must confirm if they have appliances that use LUKS encryption. During the 7.5.0 Update Package 8 upgrade, the installer will check systems and block the upgrade if the installer detects LUKS drive encryption. For more information, see QRadar: Hosts with LUKS encryption cannot be upgraded to 7.5.0 Update Pack 8.
  •   UP8 Important Leapp disk space: Leapp pretests fail to ensure if the /storetmp directory has sufficient disk space to store the upgrade cache directory. You must ensure that all appliances have at minimum 10GB of space available in the /storetmp​​​​​​​ directory before you upgrade to 7.5.0 Update Package 8. For more information, see the QRadar 7.5.0 Update Package 8 release notes.
  •   UP8 Important (new post-installation requirement for HA): Administrators with High Availability (HA) appliances in their deployment must complete a post-installation step that is new in QRadar 7.5.0 Update Package 8. After the update completes, you must complete the procedure outlined in DT365145.
  •  UP8 Important: QRadar 7.5.0 Update Package 8 users with WinCollect 7 must update to the latest version. If you upgrade to QRadar 7.5.0 Update Package 8 and have WinCollect 7.x agents deployed in managed mode, you must install the WinCollect 7.3.1-43 SFS file as outlined in the WinCollect 7.3.1 P3 release notes.
  •   UP7 Important: Administrators need to confirm their managed hosts are encrypted before you upgrade to QRadar 7.5.0 Update Package 7 to prevent a known issue with deploys documented as IJ49176/DT247083. QRadar Support is recommending administrators enable encryption on managed host before you begin an upgrade to 7.5.0 Update Package 7.
  •  Critical: QRadar 7.5.0 Update Pack 2 or later appliances upgrade from QRadar 7.2.8 installations can experience an issue where applications cannot start after an upgrade due to a filetype issue. All administrators should confirm their Console appliance reports ftype=1 before you upgrade to ensure you do not experience APAR IJ41796: Docker services do not start when 7.2.8 or earlier appliances are updated to 7.5.0 UP2 IF2 or later.
  •  Critical: Administrators who have hostnames with underscore characters can experience an exception when you attempt to open the Admin tab. RFC 822 does not permit underscore characters for hostnames and administrators might be required to update their hostnames to avoid the exception. Per IBM Documentation, 'Hostnames that do not confirm to standards are no longer supported in QRadar 7.3.X, 7.4.x, or 7.5.x and later. A valid hostname is a string up to 24 characters that include only [a-z][A-Z][0-9], minus sign (-), and period (.)'.

QRadar 7.4.x alerts

  •  Critical: Administrators with domains enabled must review the following flash notice QRadar Support recommends users upgrade to QRadar 7.4.3 Fix Pack 2 where this domain issue is resolved.
  •  Critical: Administrators on QRadar 7.4.1 and earlier versions are advised to NOT run qchange_netsetup as advised in the following flash notice: https://www.ibm.com/support/pages/node/6427875
  •  Critical: Administrators who upgrade, rebuild, or reinstall QRadar appliances to a version earlier QRadar 7.4.2 Fix Pack 1 must apply the fix for IJ30161 to their Console to resolve a "Waiting for license" issue.
  •  Important: Administrators with Event Collectors upgrading from QRadar 7.3.2 Fix Pack 3 or later to a QRadar 7.4.2 version must run the GlusterFS migration utility. For more information, see: QRadar: Steps to migrate from GlusterFS to Distibuted Replication Block Device on Event Collectors with the migration script.
  •   Important: Administrators can remove any heap dump files located in /store/jheap on QRadar appliances before you begin an upgrade to QRadar version 7.4.1 or later as the upgrade can hang as described in APAR IJ31074.
  •  Critical: Administrators who have hostnames with underscore characters can experience an exception when you attempt to open the Admin tab. RFC 822 does not permit underscore characters for hostnames and administrators might be required to update their hostnames to avoid the exception. Per IBM Documentation, 'Hostnames that do not confirm to standards are no longer supported in QRadar 7.4.x and later. A valid hostname is a string up to 24 characters that include only [a-z][A-Z][0-9], minus sign (-), and period (.)'.

QRadar 7.3.x legacy alerts for historic use

  •  Important: A flash notice was issued for a custom property concurrency in QRadar 7.3.3 and 7.3.2 versions that can cause ariel writer thread issues. For more information, see: QRadar Custom property concurrency can cause search and ariel data loss (APAR IJ21718). Fix packs and interim fixes are available in QRadar 7.3.2 Patch 5 Interim Fix 01 and QRadar 7.3.3 Patch 1.
  •  Important: A flash notice was issued for QRadar 7.3.2 Patch 3. Users who have Event Collectors and Routing Rules must update to QRadar 7.3.2 Patch 4 to avoid this issue. For more information, see: QRadar 7.3.2 Patch 3: Event Collector Appliances and Routing Rules (APAR IJ18032).
  •  Important: Administrators with Event Collectors upgrading from QRadar 7.3.2 Fix Pack 3 or later to QRadar 7.4.2 versions must run the GlusterFS migration utility. For more information, see: QRadar: Steps to migrate from GlusterFS to Distibuted Replication Block Device on Event Collectors with the migration script.
  •  A known issue has been added to the QRadar 7.3.2 Patch 3 release notes for APAR IJ17937 related to log in access for LDAP/AD environments. A post-installation workaround for administrators is available. 
  •  WinCollect administrators need to review APAR IJ17394 for authorized service token issue after a QRadar 7.3.2 Patch 2 upgrade. Administrators who do not use the default 'WinCollect' authorized service token for their agents or removed the WinCollect default user can experience token issues as described in the APAR. This issue is now resolved in QRadar 7.3.2 Patch 2 IF02 as listed on the WinCollect 101 page.
  •  Users with custom SNMP traps enabled need to be aware of APAR IJ17204 before you start an upgrade to QRadar 7.3.2.
  •  App Nodes must be removed from the deployment before you apply 7.3.2. There is a script designed to back up your App Node data and remove the App Node from the deployment. See Migrating from an App Node to an App Host for details on this.
  •  Ensure you move any important data in /storetmp to a new directory in /store before you update to QRadar 7.3.2. For more information on where users can safely store files long term, see: QRadar 7.3.2: Files in /storetmp are removed daily by disk maintenance.
  •  QRadar 7.3.2 includes stricter rules for Ariel queries to address APAR IJ13437. You must run the aqlValidator script to determine whether any Ariel queries must be updated before you update to 7.3.2. If you have auto updates enabled, type the following command to run aqlValidator: /opt/qradar/support/apar/aqlValidator. If you do not have this script on your system, see QRadar: How to Manually Install the QRadar Weekly Auto Update Bundle to manually apply the Auto Updates. You will then be able to run the script. For more information about the aqlValidator script, see APAR IJ13446.
  •   After you complete your software update to QRadar 7.3.2, verify that all apps are working as expected. If your app connects to outside servers through a proxy and the app not functioning as expected after you update to QRadar 7.3.2, see: QRadar 7.3.2: How to tune proxy configurations for app containers.

Update Packs, Fix Packs, and upgrade files

There are two different checklists in this article, each having a slightly different process:

  • SFS files are used upgrade QRadar software versions.
    • QRadar releases Update Packages for QRadar 7.5.0 and later as SFS files as part of our continuous delivery model, which can include fixes and new features. QRadar Update Packages can upgrade QRadar from 7.4.x versions directly to QRadar 7.5.x. 
    • QRadar 7.4.x and earlier versions use 'Fix Pack' SFS files to apply major software changes to the QRadar appliance, such as operating system updated for Red Hat Enterprise Linux and updating the QRadar software version. For example, updating software from QRadar 7.3.3 to QRadar 7.4.2 can be completed in a single SFS installation. The Fix Pack SFS checklist provides details on what administrators can check before they begin a software update.
  • ISO files are used for new installations of QRadar software.

    - For information on software releases and were to find SFS or ISO release notes, see the QRadar Software List.
    - For more information on versions, see Technote 7008656: V.R.M.F Maintenance Stream Delivery Vehicle terminology explanation.



     

 

Fix packs

 

The most common software update scenario for users is a QRadar Fix Pack update. Fix Packs are published to IBM Fix Central in the form of an SFS file and is used to update QRadar software to the same software stream (V.R.M). For example, updating software from QRadar 7.3.3 to QRadar 7.4.2. This checklist outlines information that administrators should consider or review before installing an SFS update that might help administrators. If you are planning a software update, you should read the Important Notices tab for issues that the support team has noted as these are issues that will impact most users and we want to raise awareness. For resolved issues, administrators can review the APARs 101 page.

 

Quick Links

 

Checklist for teams and administrators

  1.  Notify users of scheduled maintenance.
  2.  Verify that running scans and reports are complete.
  3.  Request that users close all open QRadar sessions to prevent error logs messages.
  4.  Advise users to close any active 'screen' sessions that are open to QRadar appliances.
  5.  Download your update to your local workstation. A link is provided in every QRadar Release Note.
  6.  If you are unsure of the IP addresses or hostnames for all appliances in the deployment, run the /opt/qradar/support/deployment_info.sh utility to get a CSV file that contains a list of IP addresses for each appliance in the deployment.

    Hardware and data reviews

    1.  Administrators should always consider updating to the latest QRadar version available.
    2.  Verify the checksum of the QRadar software downloaded from IBM Fix Central. A checksum file is provided for all downloads.
    3.  HA clusters must have primary appliances in the online 'Active' state and the secondary appliance should have a status of 'Standby' before you update. If any of the HA appliances are Unknown, administrators must contact support before you start your update.
    4.  Verify that IMM configured and functional on all appliances. For more information, see: Integrated Management Module II: User's Guide.
    5.  Verify the firmware on the appliance is at the latest version. For more information, see the QRadar Firmware List.
    6.  Due to a Red Hat bug in dbus that causes a leak of scope units in systemd or logind, a connection timeout can occur causing the patch to fail. To check for this condition in your deployment, run the following from the Console: /opt/qradar/support/all_servers.sh -C -k "find /run/systemd/system/ -name "session-*.scope" | wc -l" If you find any systems that have around 1,000 or more results returned, you should reboot that server before you upgrade. This clears the session scope files and prevents the timeout condition.
    7.  Confirm all appliances are at the same software version: /opt/qradar/support/all_servers.sh -C -k /opt/qradar/bin/myver -v > myver_output.txt
    8.  Verify that all previous updates are unmounted: /opt/qradar/support/all_servers.sh -C -k "umount /media/updates"
    9.  Verify disk space for the deployment: /opt/qradar/support/all_servers.sh -C -k df -h /root /var/log | tee diskchecks.txt
    10.  Verify all external storage which is not /store/ariel or /store is not mounted.
    11.  Verify if there are extra repository files in /etc/yum.repos.d/: ll /etc/yum.repos.d/
           If there are extra files, move them out: mv /etc/yum.repos.d/*.repo /root
    12.  Verify that there are no scripts auto mounting external backup storage.
    13.  Back up all static routes. The routing table can be displayed by using the command ip route.
    14.  Verify that there are no heap dump files in /store/jheap that might cause issues with the patch. For more information see, How to keep a heap dump file from hanging a system patch or upgrade.
    15.  Verify that the following directories are mounted and available (for HA pairs, see Step 14):
      • /store - Stores event and flow data on each appliance.
      • /storetmp - Stores configuration information on each appliance in QRadar 7.3.0 and later.
      • /store/tmp - Stores configuration information on each appliance in QRadar 7.2.8 and earlier.
      • /transient - Stored saved searches and index information.
    16.  In an HA pair, the following partitions should be reviewed before you start a software update.
      • /store is only be mounted on the ACTIVE appliance, and NOT the STANDBY.
      • /transient should be mounted on both appliances (ACTIVE & STANDBY) as this directory is used for replication.
    17.  Administrators may choose to run a pretest of the system(s) in test mode using /media/updates/installer -t on important appliances in their deployment, before installing the upgrade. Be sure to review the warning messages before you start a software upgrade.                                                                                                             Warning: When administrators use the test mode flag (-t) with Fix Packs (SFS files), the web server and hostcontext services are stopped while the tests are running. After the test is complete, they are started back automatically. Event collection can continue on appliances at V7.3.2 and later, but searches, rules, scan imports, and other product functionality are impacted until the services are started.
    18.  Administrators with Event Collectors upgrading from QRadar 7.3.2 Fix Pack 3 or later to a QRadar 7.4.2 version must run the GlusterFS migration utility. For more information, see: QRadar: Steps to migrate from GlusterFS to Distibuted Replication Block Device on Event Collectors with the migration script.
       

    QRadar software review

    1.  Review system notifications for errors and warnings for the following messages before you attempt to update. These error and warning system notifications should be resolved before you attempt to update.
      •  Performance or event pipeline degradation notifications
      •  Memory notifications
      •  TX sentry messages or process stopped notifications
      •  HA active or HA standby failure system notifications
      •  Disk failure system notifications
      •  Disk Sentry noticed one or more storage partitions are unavailable notifications
      •  Time synchronization system notifications
      •  Unable to execute a backup request notification
      •  Data replication experiencing difficulty notifications
      •  RAID controller misconfiguration notifications
    2.  Manually complete a deploy to verify it completes successfully in the user interface: Admin > Deploy Changes.
    3.  Verify the latest configuration backup completed successfully and download the file to a safe location.
    4.  Are any applications in the error state or not displaying properly? Review for display issues, such as blank tabs, application error messages, or errors in the user interface. If you experience any of these issues, you must resolve them before your upgrade begins.
    5.  Recommend stopping apps before running the installer.

    Post-installation

    1.  Clear your browser cache and alert all users to clear their browser cache. Optionally, you can use Private Browsing mode to ensure that pages are not cached.
    2.  Complete an auto update from the QRadar admin interface. Click the Admin tab > Auto Updates > Get New Updates.
    3.  Unmount the /media/updates directory on all hosts, type: /opt/qradar/support/all_servers.sh -C -k "umount /media/updates"
    4.  Delete the sfs file from all appliances.
    5.  Review any static routes or customized routing. Software updates can remove static routes and require administrators to re-create the routes after the software update completes.
    6.  Review any iptables rules that are configured to see whether interface names that have changed due to the Red Hat Enterprise operating system updates.
    7.  Start the apps that you stop before applying the upgrade/patch activity.

    If you experience a software upgrade issue, what to do

    1. Never reboot or stop an update in progress unless advised by QRadar Support. Updates in progress will display "Patch in progress -- DO NOT REBOOT" messages.
    2. If the update displays an error message, you can note the error message for your case with QRadar Support.
    3. If the command line is available, SSH to the appliance and type: /opt/qradar/support/get_logs.sh -s
    4. The output of the get_logs.sh utility is required to review your case. After the get_logs.sh utility completes, download the .tgz file to your local workstation. The get_logs.sh utility indicates location and file name of the tgz file to provide to support.
    5. Open a case with QRadar Support and describe the error.
    6. If your appliances are unavailable or not functional, you can indicate that you have a 'System down' issue.
    7. A QRadar Support representative contacts you using your preferred method of communication.

     

    Upgrading

     

    Upgrades are major release updates and typically delivered as an ISO file. For example, users who are on QRadar 7.2.8 must use the upgrade ISO file to update to the QRadar 7.3.1 software stream. An upgrade generally increments the R (revision) in the V.R.M.F format used by IBM for software numbering. This tab outlines checks administrators can review before you start a major software update as it is common that the installer to update QRadar software and the Red Hat Enterprise operating system (OS).

     

    Quick Links

     

    Checklist for teams and administrators

    1.  Notify users of scheduled maintenance.
    2.  Verify that running scans and reports are complete.
    3.  Request that users close all open QRadar sessions to prevent error logs messages.
    4.  Advise users to close any active 'screen' sessions that are open to QRadar appliances.
    5.  Download your update to your local workstation. A link is provided in every QRadar Release Note.
    6.  If you are unsure of the IP addresses or hostnames for all appliances in the deployment, run the /opt/qradar/support/deployment_info.sh utility to get a CSV file that contains a list of IP addresses for each appliance in the deployment.
    7.  Verify that teams move their personal files to a safe location, this can include:
      • Scripts
      • Personal utilities
      • Important files or exports
      • Jar files or test fixes that were provided by QRadar support
      • Verify no important files exist in temporary directories (/tmp or /store/tmp)

      Hardware and data reviews

      1.  Administrators should always consider updating to the latest QRadar version available.
      2.  Verify the checksum of the QRadar software downloaded from IBM Fix Central. A checksum file is provided for all downloads.
      3.  HA clusters must have primary appliances in the online 'Active' state and the secondary appliance should have a status of 'Standby' before you update. If any of the HA appliances are 'Unknown' administrators must contact support before you start your update.
      4.  Verify that IMM is configured and functional on all appliances. For more information, see: Integrated Management Module II: User's Guide.
      5.  Verify the firmware on the appliance is at the latest version. For more information, see the QRadar Firmware List.
      6.  Administrators can also take an additional backup step by completing a CMT export of their dashboards, rules, etc. For more information, see: Exporting QRadar Content with CMT.
      7.  Confirm all appliances are at the same software version: /opt/qradar/support/all_servers.sh -C -k /opt/qradar/bin/myver -v > myver_output.txt
      8.  If you have offboard storage configured (/store mounted off the appliance), see the QRadar Offboard Storage Guide.
      9.  Verify that all previous updates or patches are unmounted: /opt/qradar/support/all_servers.sh -C -k "umount -v /media/*"
      10.  Verify disk space for the deployment: /opt/qradar/support/all_servers.sh -C -k df -h /root /var/log | tee diskchecks.txt
      11.  Verify if there are extra repository files in /etc/yum.repos.d/: ll /etc/yum.repos.d/
             If there are extra files, move them out: mv /etc/yum.repos.d/*.repo /root
      12.  Administrators should complete a pretest using /media/cdrom/setup -t on important appliances in their deployment. This process typically takes between 2-5 minutes and does not restart services.
      13.  Verify that the following directories are mounted and available (for HA pairs, see Step 13):
        • /store - Stores event and flow data on each appliance.
        • /storetmp - Stores configuration information on each appliance in QRadar 7.3.0 and later.
        • /store/tmp - Stores configuration information on each appliance in QRadar 7.2.8 and earlier.
        • /transient - Stored saved searches and index information.
      14.  In an HA pair, the following partitions should be reviewed before you start a software update (Example):
        • /store should only be mounted on the ACTIVE appliance, and NOT the STANDBY.
        • /transient should be mounted on both appliances (ACTIVE & STANDBY) as this directory is used for replication.

      QRadar software review

      1.  Review system notifications for errors and warnings for the following messages before you attempt to update. These error and warning system notifications should be resolved before you attempt to update.
        •  Performance or event pipeline degradation notifications
        •  Memory notifications
        •  TX sentry messages or process stopped notifications
        •  HA active or HA standby failure system notifications
        •  Disk failure system notifications
        •  Disk Sentry noticed one or more storage partitions are unavailable notifications
        •  Time synchronization system notifications
        •  Unable to execute a backup request notification
        •  Data replication experiencing difficulty notifications
        •  RAID controller misconfiguration notifications
      2.  Manually complete a deploy to verify it completes successfully in the user interface: Admin > Deploy Changes.
      3.  Verify the latest configuration backup completed successfully and download the file to a safe location.
      4.  Verify the installed version of WinCollect. QRadar Support recommends you update to WinCollect v7.2.5 or later.
      5.  Are any applications in the error state or not displaying properly? This can be blank tabs, error messages or errors in the user interface. If yes, these issues should be resolved before the update is started.

      Post-installation

      1.  Clear your browser cache and alert all users to clear their browser cache. Optionally, you can use Private Browsing mode to ensure that pages are not cached when using QRadar.
      2.  Start a QRadar auto update from the QRadar admin interface (Auto Updates > Get Manual Updates). This can take several hours to fully install. Review system notifications to confirm that the auto update was successful.
      3.  Unmount the /media/cdrom directory on all hosts, type: /opt/qradar/support/all_servers.sh -C -k "umount /media/cdrom"
      4.  Delete the ISO from all appliances.
      5.  Review any static routes or customized routing. An upgrade can remove static routes that might need to be re-created by the administrator after the upgrade completes.
      6.  Review any iptables rules that are configured to see if the interface names that have changed in QRadar 7.3.1 due to the Red Hat Enterprise 7 operating system updates.
      7.  Verify that all apps are working as expected. If your app connects to outside servers, you have a QRadar proxy configured, and the app not functioning as expected after you update to QRadar 7.3.2, see: QRadar 7.3.2: How to tune proxy configurations for app containers.

      You've experienced an update issue, what should you do?

      1. Never reboot or stop an update in progress unless advised by QRadar Support. Updates in progress will display "Patch in progress -- DO NOT REBOOT" messages.
      2. If the update displays an error message, you can note the error message for your case with QRadar Support.
      3. If the command line is available, SSH to the appliance and type: /opt/qradar/support/get_logs.sh -s
      4. The output of the get_logs.sh utility will be required to review your case. After the get_logs.sh utility completes, download the .tgz file to your local workstation. The get_logs.sh utility will indicate location and file name of the tgz file to provide to support.
      5. Open a case with QRadar Support and describe the error.
      6. If your appliances are unavailable or not functional, you can indicate that you have a 'System down' issue.
      7. A QRadar Support representative contacts you through your preferred method of communication.

      To assess the general health of your deployment, it is helpful to have standard checks to follow in order to make sure core functionality has been restored after updating your software, restarting the QRadar Console or managed hosts.

      1. Is the user interface for QRadar accessible?

      Can you log in to the Console user interface from a browser?  
      TIP: It is best to clear your browser cache or attempt to log in using the browser's private or incognito mode after a software update.

      What to review
      1.  Attempt to log in to the QRadar command-line using SSH as the root user.
        If a software update is still in progress, the command-line prints, "Patch still in progress - DO NOT REBOOT!". Users might need to wait for the patch update to complete on the Console and services to be restarted before the user interface is available.
      2.  If you cannot access the user interface or command line using SSH, then connect using IMM or Hypervisor session log in to the console to see if it is running. If you can connect and the console is running contact your network administrator. 
      3.  If you cannot access the user interface, but can access the Console system via SSH, type systemctl status tomcat.
        If status is active or activating, the service is still initializing. In some cases, you may need to wait for 10-15 minutes after starting Tomcat for the user interface to become accessible.

      2. Are all Managed Hosts showing the expected Status?

      If any of the Managed Hosts are not in Active status, check the following:
      1.  From the Console command-line interface (CLI), try establishing an SSH connection to each non-Active Managed Host (MH).

        If the connection fails or times out:
      2.  Does a Full Deploy task complete successfully for all Manage Hosts?
        • After the Deploy task has completed and returned status for all hosts, for any Managed Hosts that report Timed Out or otherwise fail to deploy:
          • Check for the file /opt/qradar/conf/hostcontext.NODOWNLOAD on the MH. For more information on how to resolve this issue, see Technote 10744001 - Deploy Changes Does Not complete or Times out.
          • Check that all partitions have sufficient free disk space:
            • Connect to the MH via SSH as 'root' and type: df -h. If the percentage of disk space used ('Use%') for each partition is above 85% free up space on the relevant partitions and try again. For additional information on how to resolve space disk issue see our Support 101 page on Troubleshooting disk space issues.

      3. Do all Dashboards populate?

      Dashboards largely rely on ariel queries against accumulated data (also referred to as Global Views (GV) or Aggregated Data Views (ADV).
      • If some Dashboards are failing to populate, but others are working, we're likely seeing a problem with the individual Global Views. To troubleshoot corrupted Global Views, please note which Dashboards are affected.
      • If all Dashboards are failing to populate, check the statuses of the accumulator service as the accumulator should be an active service on all managed hosts.
        • On all hosts the accumulator service should be Active. From the Console command line, type:
            /opt/qradar/support/all_servers.sh -C 'systemctl is-active accumulator'
          For any hosts where accumulator is not Active, try manually restarting the service with the command `systemctl restart accumulator`.
        • The Console and all EP, FP, EP/FP, and Data Node hosts should also have a running ariel service. On the Console ariel runs as ariel_proxy_server. All other relevant hosts will have ariel as ariel_query_server.
          • On the Console run:
            systemctl is-active ariel_proxy_server
          • To check ariel on the remaining Managed Hosts. Run this command from the Console CLI:
            /opt/qradar/support/all_servers.sh -a '16%,17%,18%,21%' 'systemctl is-active ariel_query_server'
          • If the ariel* service is stopped, try starting the relevant service manually with `systemctl restart <service>` on the affected systems.
            • If ariel fails on any system, you may not be able to retrieve event or flow data from that Managed Host, accumulated or otherwise.

      4. Are Offenses generating and updating?

      1.  In the 'Last Event/Flow Seen' for the most recently updated Offenses, is the timestamp recent? 
      2.  Confirm whether ecs-ep is running on all 31xx, 18xx, 17xx, and 16xx hosts:
             - For 7.2.8: /opt/qradar/support/all_servers.sh -C -a '31%,18%,17%,16%,software' "service ecs-ep status"
             - For 7.3.x: /opt/qradar/support/all_servers.sh -C -a '31%,18%,17%,16%,software' "systemctl is-active ecs-ep"

        A. If service status is failed
        If the service state is 'failed', try restarting the service:
          - For 7.2.8: /opt/qradar/support/all_servers.sh -C "service ecs-ep restart"
          - For 7.3.x: /opt/qradar/support/all_servers.sh "systemctl restart ecs-ep"
        If ecs-ep fails to start on any systems, collect the logs for the Console system and any affected managed host where the ecs-ep is not starting as expected and open a case with QRadar Support.

        B. If service status is running on the managed hosts
        Rule out the possibility that the system is working correctly and no events or flows that are triggering rules to generate offenses. Administrators can confirm offense creation by creating a rule:
        1. In the QRadar UI, click the Offenses tab, then select Rules.
        2. Once the Rules display loads select Actions > New Event Rule.
        3. Identify a Source IP (or IP range) in your environment that is consistently generating events
        4. Configure the new event rule with this criteria, using the address or range identified above as the value for:
          • Apply Offenses Test Rule on events which are detected by the local system and when the source IP is one of the following IP addresses and click Next.
        5. Set the Responses options:
          • Select the Ensure the detected event is part of an offense check box.
          • Respond no more than 1 per 1 minute per source IP.
          • Click Finish.
          • On the Offenses tab, ensure the rule is enabled.
          • On the Log Activity tab, add a filter for the Source IP or range you configured in the rule, and watch for incoming events.
        6. When events show up for this IP or range, you should see that a new Offense is created and the events for the source IP or range are associated with that Offense.
        7. Disable the rule after you confirmed offenses are generated by the Console as the test is complete.

      5. Can you search on and view events or flows?

      1.  Do you see new event and flow data in QRadar?
             1A. Click the Log Activity tab and select View > Real Time (streaming).
             1B. Click the Network Activity tab and select View > Real Time (streaming).
      2.  Are your Console, Event Processors, and Event and Flow Processors receiving events?
        In Log Activity tab, select Quick Searches > Event Processor Distribution - Last 6 Hours. Verify that your Console (31xx), Event Processors (16xx), and Event and Flow Processors (18xx) appliances are represented on the search results.
      3.  Can I run normal searches?

      6. Is your Assets tab populated as expected?

      1.  Is the Assets page loading and populating as expected?
      2.  Is the Last Modified Time for the most recently modified asset relatively recent?

      7. Can all users log in to the Console user interface?

      1.  Can the local 'admin' account login to the user interface?
      2.  Can local non-admin Users log in to the user interface?
      3.  Can non-local users (LDAP, etc; where applicable) login to the user interface?

      8. Are my apps all loaded and running?

      9. Additional CLI checks

      1.  The checks described above should cover most core functionality. You can run the following command to ensure all QRadar processes are started:
          /opt/qradar/upgrade/util/setup/upgrades/wait_for_start.sh
        NOTE: Wait for at least 3 iterations after an upgrade to confirm all processes display a status of running. If any services display as stopped after a few iterations, note the services and press Ctrl +c to stop the utility.
        image-20200217175930-1
      2.  If using high availability (HA), administrators can use the ha cstate command to confirm the status and synchronization of HA pairs: Troubleshooting QRadar HA deployments.
      3.   Review System Notifications in the QRadar Dashboard for any Predictive Disk Failure: QRadar: Troubleshooting Disk Failure or Predictive Disk Failure Notifications
      4.   To confirm the Console appliance polls for X-Force Threat Intelligence feeds, type the following command:  tail -f /var/log/dca/sca_server.log

        The Console attempts to poll for data every 3 minutes and must not return errors in the sca_server.log file. When successful, the logs display the new database version for IP reputation (IPR) and URL classifications as they are installed.
        2020-02-17 17:50:22.354 I UPDATE: Result #1 id=20, module=IPR Classification, contentUpdated=true, engineUpdated=false, details=1 0 19614
        2020-02-17 17:50:22.354 I UPDATE: Detail #0: component=ipr_database old version=6.01117773 new version=6.01117774 available=true downloaded=true installed=false return=download or install scheduled 0 19614   
         

      My upgrade failed a post-install check, how do I proceed?

      1. Record all troubleshooting steps from any procedure that failed.
      2. If possible collect logs from the Console and the affected systems. Refer to the link below on how to collect logs.
          Getting logs from a QRadar deployment
      3. Open a case with IBM QRadar Support. Include logs and all information from troubleshooting step in the case.
      4. If your appliances are unavailable or not functional, you can indicate that you have a 'System down' issue.
      5. A QRadar Support representative will contact you using your preferred method of communication

      [{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

      Document Information

      Modified date:
      25 March 2024

      UID

      ibm10738599