RPG Cafe, Fall 2022: CHARCOUNT-NATURAL mode

Content

Fall 2022: CHARCOUNT NATURAL mode to handle data correctly that has different-sized characters, such as UTF-8

Details

Options to process strings by natural characters instead of bytes or double bytes

• UTF-8 data can have characters with 1, 2, 3, or 4 bytes. 2-byte characters are common.
• UTF-16 data can have characters with 2 or 4 bytes.
• Mixed EBCDIC data can have characters with 1 or 2 bytes, and it contains "shift" characters to indicate the change between SBCS and DBCS data.
• Mixed ASCII data can have characters with 1 or 2 bytes.

With the default standard-character-size mode (STDCHARSIZE), data is handled by bytes or double-bytes, without regard for the length of individual characters in the data.

With natural mode:

• Most string built-in functions such as %SUBST, %SCAN, %XLATE, %SPLIT operate by characters rather than by bytes or double bytes. However, %LEN always works with the number of bytes or double bytes and %STR always works with the number of bytes. New built-in function %CHARCOUNT returns the number of characters in a string.
• If data is truncated during assignment or parameter-passing, the truncation removes complete characters.
• If data is truncated during I/O operations, the truncation removes complete characters.

There are several options to enable or disable CHARCOUNT NATURAL mode.

• Specify Control keyword CHARCOUNT to set the initial CHARCOUNT mode.
• Specify Control keyword CHARCOUNTTYPES to list the data types that you want to be handled in CHARCOUNT NATURAL mode.
• Specify directive /CHARCOUNT NATURAL to set the mode to CHARCOUNT(*NATURAL) for statements following the directive.
• Specify directive /CHARCOUNT STDCHARSIZE to set the mode to CHARCOUNT(*STDCHARSIZE) for statements following the directive.
• Specify *NATURAL as the last parameter of a built-in function to enable CHARCOUNT NATURAL mode for the built-in function even if the data type of the operands is not relevant according to the CHARCOUNTTYPES keyword. Specify *STDCHARSIZE as the last parameter of a built-in function to disable CHARCOUNT NATURAL mode for the built-in function.
• Specify File keyword CHARCOUNT(*NATURAL) to set CHARCOUNT NATURAL mode when data is moved from RPG variables to the output buffer and key buffer for the file. Specify File keyword CHARCOUNT(*STDCHARSIZE) to disable CHARCOUNT NATURAL mode when data is moved from RPG variables to the output buffer and key buffer for the file. The CHARCOUNT mode for a file defaults to the current CHARCOUNT mode for the definition statement for the file.

This example shows how you can operate in standard-character-size mode in general, and operate in natural mode only when you know the UTF-8 data might have multi-byte characters.

           dcl-s string varchar(20) ccsid(*utf8);
           dcl-s n int(10);
           dcl-s string2 varchar(20);

           string = 'ábcdë';

           n = %len(string);
           // n = 7 (á and ë are 2-byte characters)
           n = %charcount(string);
           // n = 5

           string2 = %subst(string : 1 : 3);
           // string2 = "áb"
           string2 = %subst(string : 1 : 3 : *natural);
           // string2 = "ábc"

This example shows how you can operate in natural mode by default for UTF-8 data. The example shows how you can switch back to standard-character-size mode, but in general, you would switch to standard-character-size when you know you are working with data that has only 1-byte characters.

           ctl-opt charcounttypes(*utf8);
           ctl-opt charcount(*natural);

           dcl-s string varchar(20) ccsid(*utf8);
           dcl-s n int(10);
           dcl-s string2 varchar(20);

           string = 'ábcdë';

           n = %len(string);
           // n = 7 (á and ë are 2-byte characters)
           n = %charcount(string);
           // n = 5

           string2 = %subst(string : 1 : 3);
           // string2 = "ábc"
           string2 = %subst(string : 1 : 3 : *stdcharsize);
           // string2 = "áb"

           /charcount stdcharsize

           string2 = %subst(string : 1 : 3);
           // string2 = "áb"
           string2 = %subst(string : 1 : 3 : *natural);
           // string2 = "ábc"

.                                                            .

.                                                            .