IBM Support

QRadar: How to migrate applications from app framework v1 to v2

News


Abstract

A new version of the QRadar® application framework is available for development teams. Application developers can review and complete the steps outlined in this article to migrate your apps to application framework version 2.

Content

Overview

In response to the end of life announcement for CentOS 6 and Python 2, a new version of the QRadar® App Framework is developed. App Framework v2 uses a new app base image derived from Red Hat Universal Base Image and code executes in Python 3 (v3.6.8 or later). More specifically, the v2 app base image is the RHEL8 ubi-minimal image plus a number of add-ons including Python 3, OpenSSL, systemd, and SQLite.

If you want to run on both v1 and v2 of the framework, you need to make available two different versions of your app. This guide shows how to migrate an app from v1 to v2 of the App Framework, and describes some of the new features in v2.

Notice: For a video discussion on application framework v1 to v2 migration, see: https://www.securitylearningacademy.com/enrol/index.php?id=5656.
 

Migrating your application

To migrate an app from v1 to v2 of the App Framework, complete the following steps:

1. Update manifest.json

image
By default, applications can continue to install with the CentOS 6 base image, unless you specify otherwise in your app’s manifest.json file. A new manifest field named image identifies the base image (qradar-app-base) and version (2.0.0 or later) that you want to use, as demonstrated in this example:
{
  "name": "Hello World",
  "description": "Simple example app",
  "version": "1.0",
  "image": "qradar-app-base:2.0.0",
  "uuid":"558d7935-f00b-42da-a278-c82abdb12b34",

  "areas": [{
    "id": "QHelloWorld",
    "text": "Hello World",
    "description": "Hello World area",
    "url": "index",
    "required_capabilities": ["SEM"]
  }]
}
Important: Use the same uuid in your manifest as your CentOS version of the app.
 
dependencies
If your manifest includes a dependencies entry for specifying the location of RPMs and Python packages inside your app, you must remove it, as it is no longer supported. For more details on dependency location, see step 5 Update app layout.


environment_variables
In App Framework v1, the environment variable PATHSTART could be used to alter the container PATH setting so that Python 2.7 was used instead of Python 2.6. In v2, as all processing is done with Python 3, PATHSTART is no longer required for this purpose and, if not needed for any other PATH customization, can be removed.

 

2. Check qpylib usage

The new app base image includes qpylib preinstalled as a Python package. If your app has a qpylib directory containing an older version of qpylib source code, remove the directory and use the version in the base image instead. The source code for qpylib is now maintained in an open source repository on GitHub: https://github.com/IBM/qpylib.

Implementation changes in qpylib:

  • Due to the introduction of new features in App Framework SDK v2, most of the sdk/live abstraction is removed.
  • Most functions are the equivalent of their pre-GitHub counterparts, although there are some minor changes to the parameters of these functions: get_root_path, get_store_path, REST.
  • The encryption module encdec.py updates.
    • Encryption algorithms are strengthened.
    • Support for backwards-compatible decryption of old secrets was added.
    • Error-handling is changed, and all errors are reported as EncryptionError.
  • New module ariel.py adds support for performing Ariel searches.
     

3. Convert Python 2 code to Python 3

The v2 app base image supports Python 3 only, so any Python 2 code in your app must be converted to Python 3. There is a porting tool available called 2to3, which helps with this process. For more details, see https://docs.python.org/3/howto/pyporting.html.

 

4. Check qjslib usage

qjslib now supports usage as a module or as a browser script. Like qpylib, the source code for qjslib is now maintained in an open source repository on GitHub: https://github.com/IBM/qjslib. Instructions on how to use qjslib in your app can be found there.

 

5. Update app layout

The v1 src_deps directory is replaced with a directory named container. Its contents are:

  • container/pip: replaces src_deps/pips and includes Python package dependencies
  • container/rpm: replaces src_deps/rpms and includes RPM dependencies
  • container/build: replaces src_deps/init for scripts that can be executed at image build time
  • container/run: replaces src_deps/init for scripts that can only be executed at container start time
  • container/clean: holds cleanup.sh script to be run when the app container is shut down
  • container/service: holds scripts that start app background processes
  • container/conf: holds app configuration files
  • container/conf/supervisord.d: the contents of this directory are copied to /etc/supervisord.d/
    and included as a part of the supervisord configuration.

To enable fast container startup, best practice dictates that as far as possible, all dependencies must be installed and all startup scripts must be run when the app image is built. Do not use container/run unless you have to.

Also, the location of pip and rpm dependencies can no longer be overridden via the app manifest. You must use container/pip and container/rpm.
 

6. Check for location-dependent code

In the app container, all app-related files and directories are now located under /opt/app-root rather than /.

The subdirectories of /opt/app-root include:

  • app: the source code for your app
  • container: as described in section 5
  • store: the directory for persisting app data, particularly log files
  • bin and startup.d: these directories hold container plumbing scripts

The manifest.json file now lives at the top level of the deployed app (in /opt/app-root) rather than being relocated inside the app directory as it was in v1.

If you have any source code that directly accesses app files or directories using a hardcoded path location, you may need to update it. Python-based apps are abstracted from these changes if they use qpylib utility functions to access items within an app.
 

7. Update app dependencies

The v2 app base image has a different operating system and a different version of Python from v1, so any RPM dependencies must be el8 versions compatible with Red Hat 8, and any Python packages must be compatible with Python 3.6.8 or later.

RPM example: nginx-1.11.13-1.el6.ngx.x86_64.rpm from CentOS 6 might be replaced by nginx-1.17.8-1.el8.ngx.x86_64.rpm.

Note that the package manager in the v2 base image is microdnf. The yum package manager is not installed.

Also, bear in mind that the v2 base image contains updated versions of software such as Flask and supervisor. Your app will need to be compatible with the new versions.
 

8. Check permissions and privileges

For security reasons, processes within the app container no longer execute as root but instead use a non-privileged user account. supervisord runs Flask-based apps as this user, and any other processes should run as this user if possible. The same non-privileged user owns all app-related files and directories inside the container.

If you need to run something as root, use the as_root script provided in /opt/app-root/bin.

 

New features in QRadar App Framework v2

App Framework SDK v2

SDK v2 supports the new app base image and Python 3. These are some of the SDK’s new features:

  • Using Docker, you can now test your app locally by building an app image and running an app container, effectively using the same environment as when an app is deployed to a QRadar server.
  • Root privileges are no longer required to run the SDK v2 install script. The SDK is installed in a sub-directory of the user’s home directory, and has its own Python 3 virtual environment. Any existing Python installation on the user’s machine is left untouched, as is any installation of SDK v1.
  • SDK v2 actions are invoked using the command qapp rather than the v1 entry point qradar_app_creator.
  • The combination of different install locations and different entry points means that SDK v1 and v2 can coexist. This allows developers to maintain an older CentOS version of an app while working on a newer Red Hat UBI version.
  • The updated SDK example app demonstrates best practice for writing a Flask-based app for App Framework v2.
     

Container cleanup script
If your app has a container/clean/cleanup.sh script, that script will be executed when the app container is stopped. The script is triggered when either a SIGINT or SIGTERM signal is received. Having a cleanup script allows graceful shutdown of things like database connections. Without a cleanup script, the container process is forcefully killed after 10 seconds.

Supervisord configuration
If you place .conf files in your app’s container/conf/supervisord.d directory, those files will be copied to /etc/supervisord.d in the container, and will be picked up thanks to the inclusion of this entry in /etc/supervisord.conf:

[include]
files = /etc/supervisord.d/*.conf

Note: only .conf files will be copied; subdirectories will be ignored.

Named service access
For apps that don’t use Flask and instead use, for example, nginx, if you add nginx as a named service on port 5000 then it will be accessible via the standard app base url /console/plugins/<app id>/app_proxy as well as the named service url /console/plugins/<app id>/app_proxy:<service name>.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]

Document Information

Modified date:
14 May 2021

UID

ibm16356287