Unicode Enabling of ABAP

Contents
Unicode Overview
What is Unicode Need for Unicode Unicode Character formats

Unicode @ SAP
SAP Pre-Unicode solutions Why SAP adopted Unicode

Impact of Unicode on ABAP

Overview Concepts and Conventions Restrictions in Unicode New ABAP Features Tools for Unicode enabling

Unicode Overview

What is Unicode?

What Is Unicode?
Fundamentally, computers store letters and other characters by assigning a number for each one. Unicode provides a unique number (Code point) for every character, no matter what the platform, no matter what the program, no matter what the language. Notation U+nnnn (where nnnn are hexadecimal digits)

Character encoding schema for (nearly) all characters used world wide

What is Unicode? (contd.)

Unicode = universally encoded character set to store information from any language The Unicode standard primarily encodes scripts rather than languages Scripts comprise several languages that historically share the same set of symbols In many cases a script may serve to write dozens of languages (e.g. the Latin script) In other cases one script complies to one language (e.g. Hangul) Additionally it also includes punctuation marks, diacritics, mathematical symbols, technical symbols, musical symbols, arrows, etc. In all, the Unicode Standard comprises >95.000 characters, ideograph sets, symbols (version 4.0)

What is Unicode? (contd.)

Unicode Standard: The Unicode Standard is a character coding system designed to support the worldwide interchange, processing and display of written text of the diverse languages and technical disciplines of the modern world. In addition, it supports classical and historical texts of many written languages. The Unicode Consortium: The Consortium cooperates with W3C ISO and has liaison status "C" with ISO/IEC/ JTC1/SC2/WG2, which is responsible for in refining the specification and expanding the character set of ISO/IEC 10646

What is Unicode? (contd.)

Unicode: The last character set? It is an open character set, which means that it keeps growing and adding less frequently used characters.

The standard assigns numbers from 0 to 0x10FFFF, which is more than a million possible numbers for characters.
5% of this space is used, 5% is in preparation, about 13% is reserved for private use, and about 2% is reserved not for use

The remaining 75% is open for future use but not by any means expected to be filled up and finally there is a character set with plenty of space!

Unicode Overview

Need for Unicode

Need For Unicode

Hundreds of encodings have been developed, each for small groups of languages and special purposes. There is no single, authoritative source of precise definitions of many of the encodings and their names.

No single encoding could contain enough characters: for example, the European Union alone requires several different encodings to cover all its languages.
Even for a single language like English no single encoding was adequate for all the letters, punctuation, and technical symbols in common use. Incompatibilities between different code pages

Need For Unicode (contd.)

These encoding systems conflict with one another. That is, two encodings can use the same number for two different characters, or use different numbers for the same character.

Any given computer (especially servers) needs to support many different encodings; yet whenever data is passed between different encodings or platforms, that data always runs the risk of corruption.
Programs are written to either handle one single encoding at a time and switch between them, or to convert between external and internal encodings

Need For Unicode (contd.)

Languages & Code Pages

Unicode, One Code Page For All Scripts

Unicode Overview

Unicode Character Formats

Unicode Character Formats

Representation of Unicode characters:

UTF-16: Unicode Transformation Format, 16 bit Encoding

- Fixed Length, 1character = 2 bytes - Platform dependent BYTE order - 2 BYTE alignment restriction UTF-8: Unicode Transformation Format, 8 bit Encoding - Variable length, 1character = 1 to 4 BYTES - Platform Independent - No alignment restriction - 7 BIT US ASCII compatible

Unicode Character Formats (contd.)

UTF-32: Unicode Transformation Format, 32 bit Encoding

For single characters, 32-bit integer variables are most appropriate for the value range of Unicode.

For strings, however, storing 32 bits for each character takes up too much space, especially considering that the highest value 0x10FFFF, takes up only 21 bits. 11 bits are always unused in a 32-bit word storing a Unicode code point. Therefore, you will find that software generally uses 16-bit or 8-bit units as a compromise, with a variable number of code units per Unicode code point.

Unicode Character Formats (contd.)

Unicode @ SAP

SAP Pre-Unicode solutions

SAP Pre-Unicode Solutions

Single Code Page System System using one standard code page which can support a specific set of languages.
Blended Code Page System (Release 3.0D) Multi byte blended code pages, which contain characters out of several standard code pages. Blended code pages are not standard code pages, but SAP-customized code pages that were devised to support an increased number of possible language combinations in a single code page. a) Ambiguous Blended Code Page System: Two characters can share the same code point. b) Unambiguous Blended Code Page System: Each code point refers exactly to one character.

SAP Pre-Unicode Solutions (contd.)

MDMP System Configuration (Release 3.1I) Multiple Display/Multiple Processing. System using more than one system code page on the application server. Allows languages to be used together in one system although the characters of those languages are not in the same code page.

SAP Pre-Unicode Solutions (contd.)

Language Combinations Before Unicode

Each user can only access one code page at a time: a user who logs in as a Japanese user cannot enter German characters ,and all German characters in the database will not be correctly displayed.

SAP Pre-Unicode Solutions (contd.)

It is possible for a user to log on with German and then manipulate the character set and font settings so that he can enter what appear to be Japanese characters; these characters will not be correctly stored in the database and this data will be corrupt If a user wants to enter f.i. Japanese, he/she must log on in Japanese To insure that no data corruption occurs, the following restrictions must be followed: Global data must contain only 7-bit ASCII characters, which are in all code pages Users may use only the characters of their log-in language or 7-bit ASCII Batch processes must be assigned with the correct user ID and language EBCDIC code pages are not supported

View in Different Code Pages

Recommendations From SAP (Pre-Unicode)

In general, using a single standard code page for new installations and upgrades is the optimal decision If additional languages or language combinations are needed, SAP recommends Unambiguous Blended Code Pages for new installations and MDMP for existing installations. Unambiguous Blended Code Pages only support certain language combinations and therefore an MDMP setup may be the only possibility for new installations as well.

Unicode @ SAP

Why SAP adopted Unicode?

Why SAP adopted Unicode?

Globalization = Internationalization + Localization

The Unicode Standard has already been adopted by industry leaders as Apple, HP, IBM, JustSystem, Microsoft, Oracle, Sun, Sybase, Unisys and many others. Unicode is required by modern standards such as XML, Java, ECMAScript (JavaScript), LDAP, CORBA 3.0, WML, etc., It is the official way to implement ISO/IEC 10646.

Why SAP adopted Unicode? (contd.)

Allows text data from different languages to be stored in one repository Enable a single set of source code to be written to process data in virtually all languages Simplifies addition of new language support to an e-business application since character processing and storage remains unchanged Lowers cost of implementation Faster speed to market Better customer satisfaction

Why SAP adopted Unicode? (contd.)

View In UNICODE System

Unicode - SAP

Platforms supported by SAP for Unicode systems

The following indicates the current development status and availability different OS and Database combinations for Unicode-based mySAP technical components:

Unicode & ABAP

Overview

Overview

Non-Unicode Versions of SAP :- Versions prior to 4.7 Unicode Versions of SAP :- Version 4.7 & above

Each character mapped using 16 bits (= 2 bytes) which offers a maximum of 216 bit combinations
Affects any older ABAP program in which an explicit or implicit assumption is made about the internal length of a character

Overview (contd.)

Character Expansion Model - Separate Unicode & Non-Unicode versions of R/3 - No explicit Unicode Data type in ABAP - Single ABAP source for Unicode & Non-Unicode systems

OVERVIEW (contd.)

Program attributes Unicode checks active - Required to run on a Unicode System If attribute is set additional restrictions: - Apply at run time and compile time - Apply in Unicode as well as NonUnicode System. - Ensure that program will run on both US & NUS with identical behavior

Non-Unicode Unicode System System

Attribute Set (Unicode enabled) Attribute not Set (not Unicode enabled) Ok Ok

Ok

Not allowed

Overview (contd.)
Screen Shot of Program Attribute Screen

Unicode & ABAP

Concepts & Conventions

Concepts & Conventions

Data Types Data Layout of Structures Unicode Fragment Views

Data Types

The following data types can be interpreted as Character Type in a Unicode program
C: Character N: Numeric character D: Date T: Time STRING: String Character-type structures: Structures which either directly or in substructures contain only fields of types C, N, D or T.

Byte Type : X, Xstring (for bit operations)

Data Layout of Structures

For several data types (like F and I) the memory address should start with multiples of 4 or 8 and for Character-type it should start with multiples 2 or 4 depending on the Unicode representation.
Within structures, bytes are inserted before or after components with alignment requirements to achieve the necessary alignment. Examples: BEGIN OF struc1, a(1) TYPE X, b(1) TYPE X, c(6) TYPE C, END OF struc1. For struc1 there is no alignment gap for NUS or US.

Data layout of structures (Contd..)

Examples:
BEGIN OF struc2, a(1) TYPE X, BEGIN OF struc3, b(1) TYPE X, c(6) TYPE C, END OF struc3, d TYPE I, END OF struc2.

Unicode Fragment View

BEGIN OF struc, a(2) TYPE C, b(4) TYPE N, c TYPE D, d TYPE T, e TYPE F, f(2) TYPE X, g(4) TYPE X, h(8) TYPE C, i(8) TYPE C, END OF struc.

Unicode Fragment Views: F1, F2, F3, F4,F5, F6

Unicode & ABAP

Restrictions in Unicode

Overview

Access Using Offset & Length Addressing

Assignments Casting Data Objects Processing Strings in UNICODE Determining Length & Distance Specifying Key for Table Access Database Operations Structure Enhancement Other changes

Access Using Offset & Length Addressing

Accessing Single fields: - Offset or Length based access supported for single fields of Character Type, strings, X, Xstrings Accessing Structure: - This access type results in errors if both character type and noncharacter type fields are present in the area addressed by the offset and the length Passing parameters to subroutines: - Passing parameters using perform with offset and length specification beyond field boundaries is not allowed in Unicode programming

Access Using Offset & Length Addressing (contd.)

Accessing Field Symbols: - Offset- or length-based access with ASSIGN is only permitted within a predefined range - ASSIGN feld [+off(len)]TO <f>. : Range corresponds to the field boundaries in case of elementary fields or, in case of flat structures, to the purely character-type starting fragment - ASSIGN <g>[+off(len)] TO <f>. : Range of the target FS = Range of the Source FS

Assignments

Conversion between Flat Structures (MOVE): - The fragments of both structures up to the second-last fragment of the shorter structure are identical - The last fragment of the shorter structure must be of character or bytetype group - The corresponding fragment of the longer structure is a character or byte type group with a greater length

Assignments (contd.)

Conversion between internal tables - Tables can be converted if their row types are convertible. The restrictions

described for structures also apply for the conversion of tables. Implicit Conversions: - The same rules also apply to all ABAP statements that use implicit conversions according to the MOVE semantics. APPEND wa TO itab. INSERT wa INTO itab. MODIFY itab FROM wa. READ TABLE itab ...INTO wa. LOOP AT itab INTO wa.

Processing Strings
Statements for processing strings:
CLEAR ... WITH CONCATENATE CONDENSE CONVERT TEXT ... INTO SORTABLE CODE FIND OVERLAY REPLACE SEARCH SHIFT SPLIT TRANSLATE ... TO UPPER/LOWER CASE TRANSLATE ... USING FROM CODEPAGE and FROM NUMBER FORMAT

are not allowed with TRANSLATE The arguments must be single fields of type C, N, D, T, STRING or purely character-type structures CONCATENATE a x b INTO c statement is not possible if a, b, and c are charactertype but x has type X.

Processing Strings (contd.)

Comparison operators for processing byte and character strings:

- Operators require single fields of type C, N, D, T, STRING or purely character-type structures as arguments CO CN CA NA CS NS CP NP

BYTE-CO BYTE-CN BYTE-CA BYTE-NA BYTE-CS BYTE-NS BYTE-CP BYTE-NP

Processing Strings (contd.)

Functions for processing byte and character strings:

STRLEN only for character-type fields and returns the length in characters

XSTRLEN for finding length of byte strings

NUMOFCHAR - returns the number of characters in a character-type field

Processing Strings (contd.)

Output in fields and lists: If the source field is a flat structure in a WRITE statement, it must have character type only, in a Unicode system. This affects the following statements: WRITE f. WRITE f TO g[+off][(len)].. WRITE (name) TO g.. WRITE f TO itab[+off][(len)] INDEX idx.. WRITE (name) TO itab[+off][(len)] INDEX idx..

Establishing Length & Distance

NUS DESCRIBE DISTANCE BETWEEN.. US - Addition of IN BYTE MODE / IN CHARACTER MODE NUS DESCRIBE FIELD.. LENGTH US - Addition of IN BYTE MODE / IN CHARACTER MODE

ABAP File Interface

Opening Files: The OPEN DATASET dsn ... statement must include at least one of the following additions:
... IN TEXT MODE ENCODING ... (UTF-8) ... IN BINARY MODE ... (UTF-8 ) ... IN LEGACY TEXT MODE ...(Non Unicode Format) ... IN LEGACY BINARY MODE ... (Non Unicode Format)

ABAP File Interface (contd.)

Reading & Writing Files: READ DATASET dsn INTO f : For Reading File TRANSFER f TO dsn : For Writing into File If the file is opened in TEXT MODE, f must be a character-type field ( C, N, D, or T), a string, or a purely character-type structure If the file is opened in LEGACY TEXT MODE, or LEGACY BINARY MODE, conversion errors may occur

Other Changes relating to Unicode

Bit Statements: SET BIT i OF f [TO g]. GET BIT i OF f [INTO g]. f O x, f Z x, and f M x . The field f must be either X or XSTRING in all the above bit operations.

Other Changes relating to Unicode (contd.)

ADD n1 THEN n2 UNTIL nz [ ACCORDING TO sel ] GIVING m [RANGE

str ]. ADD n1 THEN n2 UNTIL nz TO m [ RANGE str ]. Operands n1, n2, and nz must be of the same type and length. The distance between nz and n1 must be an integral multiple of the distance between n2 and n1. The fields n1, n2, and nz must be in one structure. If this is not statically recognizable, you must use the RANGE str addition to explicitly specify a structure as a valid area. If it is discovered that the addressed fields are not within the structure specified using the RANGE addition, a syntax or runtime error occurs.

Other Changes relating to Unicode (contd.)

Loops with VARY & varying Additions: DO ... VARYING f FROM f1 NEXT f2 [ RANGE f3 ]. WHILE ... VARY f FROM f1 NEXT f2 [ RANGE f3 ].
The fields f, f1, and f2 must be type-compatible with one another RANGE for valid accesses must be implicitly or explicitly implemented

Other Changes relating to Unicode (contd.)

Generating Subroutine: GENERATE SUBROUTINE POOL itab NAME name The generated program inherits the contents of the Unicode flag of the generating program.
Saving Programs: INSERT REPORT prog FROM itab. This statement now includes a new addition, UNICODE ENABLING uc, with which the Unicode flag of the inserted report is given the value of uc

Other Changes relating to Unicode (contd.)

Types and GET/SET PARAMETER: GET PARAMETER ID pid FIELD f SET PARAMETER ID pid FIELD f The field f must be of character type. For non-character type we use IMPORT & EXPORT statements.

Specifying the Key for Tables Accesses

A syntax or runtime error occurs when you access the database with a generic key unless the key is purely character-type. This affects the following commands: READ TABLE dbtab ...SEARCH GKEQ ... READ TABLE dbtab ...SEARCH GKGE ... LOOP AT dbtab ... REFRESH itab FROM TABLE dbtab.

Please note that this statement is Obsolete and should no longer be used.

Database Operations

SELECT * FROM dbtab ... INTO wa / INTO TABLE itab ... FETCH NEXT CURSOR c ... INTO wa / INTO TABLE itab. INSERT INTO dbtab ... FROM wa / FROM TABLE itab. UPDATE dbtab ... FROM wa / FROM TABLE itab. MODIFY dbtab ... FROM wa / FROM TABLE itab. DELETE dbtab FROM wa / FROM TABLE itab.
The fragment views of the work area and the database table must be identical with regard to the length of the database table. If the work area is a single field, the field must be character-type and the database table purely character-type

Storing Data Clusters in Database Tables

Data clusters are not converted when they are migrated from a nonUnicode database to a Unicode system. For this reason, there may be ABAP cluster tables in a Unicode system that contain non-Unicode characters. These characters are automatically converted during each import. When the data is exported, any Unicode characters that may be contained in the data objects are stored in a platform-specific format in the Unicode system.

Structure Enhancements

Problems caused by structure enhancements: Enhancements change the fragment views and hence affect the checks for assignments and comparisons Enhancement classification in ABAP Dictionary:

Structure Enhancements (contd.)

Enhancement handling in program check:

Unicode & RFC

Unicode & ABAP

New ABAP Features

Assignments to Field Symbols

Range: ASSIGN feld1 TO <fs> RANGE feld2. Sets the range limits, making it possible to define addresses past field limits Increment: ASSIGN field INCREMENT n TO <fs> First the range for the access is defined from the length of field and the INCREMENT definition of the range for the access as defined by ASSIGN fld+n*sizeof[field] (sizeof[fld]) TO <f>. If sy-subrc > 0 then no increment is made to the field symbol

Assignments to Field Symbols (contd.)

Casting: ASSIGN field TO CASTING. ASSIGN field TO <fs> CASTING TYPE type. ASSIGN field TO <fs> CASTING TYPE (typename). ASSIGN field TO <fs> CASTING LIKE fld. ASSIGN field TO <fs> CASTING DECIMALS dec. Provides different views on a structure with casts on different types. Treats the contents of a field as a value of another type using a field symbol field must be at least as long as the type that was assigned to the field symbol, <fs> If the field symbol type is a deep structure the system checks for the offset and reference component match of the area covered by <fs>

Includes with Group Names

Creating Data Objects Dynamically

Data Objects: Creating and Accessing data objects on the heap

Creating Data Objects Dynamically (contd.)

Table Objects: Creation of Table Objects at run time.

CREATE DATA dref (TYPE [STANDARD|SORTED|HASHED] TABLE OF (LineType | (Name) | REF TO DATA | REF TO Obj)) | (LIKE [STANDARD | SORTED | HASHED] TABLE OF LineObj )[ WITH (UNIQUE|NON-UNIQUE) ( KEY (K1 ... Kn | (KEYTAB) | TABLE_LINE )| DEFAULT KEY ) ][ INITIAL SIZE M ]

The line type and table key can be entered statically or dynamically

Storing Data Clusters

Using fields of type XSTRING as data containers

Writing data to an XSTRING (EXPORT):

Storing Data Clusters (contd.)

Reading Data from an XSTRING (IMPORT):

Automatic Conversion of data during IMPORT.

Generic Types for Field Symbols & Parameters

New Classes For UNICODE

Character Utilities: Class CL_ABAP_CHAR_UTILITIES

New Classes For UNICODE (CONTD.)

Conversion Classes:

Unicode & ABAP

Tools for Unicode Enabling

ABAP Unicode Scan Tool UCCHECK

Transaction UCCHECK is used to examine a Unicode program set for syntax errors without having to set the program attribute "Unicode checks active" for every individual program.
From the list of Unicode syntax errors, one can go directly to the affected programs and remove the errors. It is also possible to automatically create transport requests and set the Unicode program attribute for a program set.

Transaction UCCHECK

ABAP Coverage Analyzer

Coverage Analyzer (transaction SCOV):

Persistently traces the execution of all program objects within one system. Traces all processing blocks i.e. forms, methods, modules and ABAP events. Collects information number of calls number of runtime errors number of program changes

Transaction SCOV

THANK YOU