Вы находитесь на странице: 1из 39

IBM Software Group

WebSphere MQ Data Conversion


Part II

Blanche Ishida and Karen Stanley


WebSphere MQ Support

WebSphere Support Technical Exchange


IBM Software Group

Agenda
 Terms and Definitions

 WebSphere MQ Message Header (MQMD)

 Customer Scenarios

 IBM Technotes

 Useful Links

 Summary

 Questions and Answers

WebSphere Support Technical Exchange 2


IBM Software Group

Recap: Terms and Definitions


 Character Set
 Is a defined set of characters used to represent textual
information (e.g. 0-9, a-z, A-Z, .,;:!?/-)
 A Character Set generally supports more than one
language

 Code Page
 (AKA Code set) A code page can be defined as a table
with a mapping of alphanumeric code and its binary
representation.
 Where each character in a character set is assigned a
numerical representation (often used interchangeably
with character set e.g. charset in HTML)

WebSphere Support Technical Exchange 3


IBM Software Group

Recap: Terms and Definitions


 A code point is the location of a character within
the code page

 CCSID Coded Character Set ID is a unique


number (0-65535) used by IBM to uniquely identify
a Coded Character Set and a Codepage.

WebSphere Support Technical Exchange 4


IBM Software Group

Recap: Terms and Definitions


 ASCII : American Standard Code for Information Interchange.
The code used as the basis for most codes apart from
EBCDIC.
 EBCDIC : Extended Binary Coded Decimal Information Code
(EBCDIC). It is typically used on z/OS, iSeries and VSE
 UNICODE : A code page which defines most of the
characters used by the languages now used throughout the
world. UCS-2, UTF-8, UTF-16 and UTF-32 for encodings.
 SBCS Single Byte Character Set. A code page with up to 256
characters which can be represented by a single 8 bit byte.
 DBCS Double Byte Character Set. A code page with up to
65536 characters.

WebSphere Support Technical Exchange 5


IBM Software Group

Recap: MQMD
 The Message Descriptor is not entirely character data, so
some of it will be converted as character data, some as
numeric, and some is passed unconverted.

 Note: MsgId, CorrelId, and Accounting Token are NOT


considered to be character data, and as such WILL NOT be
converted as character data. Many applications put
character data into these fields, but if the message is being
exchanged between ASCII and EBCDIC platforms, and the
application on the other side needs to be able to read these
fields, then it is the responsibility of the application (or a user
exit) to convert these fields. Also the data maybe in
hexadecimal representation for MsgID and CorrelID.

WebSphere Support Technical Exchange 6


IBM Software Group

Recap: WebSphere MQ Message Header


 Encoding Numeric encoding of message data
MQMD offset X18
 CodedCharSetId Character set identifier of message data
MQMD offset X1C
 Format Format name of message data
MQMD offset X20
 MsgId Message identifier
MQMD offset X30
 CorrelId Correlation identifier
MQMD offset X48

WebSphere Support Technical Exchange 7


IBM Software Group

Recap: MQMD : Encoding


 Encoding is generally taken to mean the method
that this platform uses to represent numeric data.
There are two general types:
 LittleEndian, used by Intel processors (e.g
Windows, Linux on Intel). e.g. the number 437
would be represented (in hex) as XBF01 (we
refer to this as byte swap)
 BigEndian. e.g. the number 437 would be
represented (in hex) as X01BF.

WebSphere Support Technical Exchange 8


IBM Software Group

Recap: CCSID
 The Queue Manager CCSID is important as it
defines the default CCSID for messages being
created using MQPUT and is the default CCSID
which will be used for conversion when messages
are retrieved using MQGET. If the channel option
CONVERT(YES) is set the conversion is attempted
for the receiving Queue Managers CCSID.

 To display the queue manager CCSID use the


DIS QMGR command

WebSphere Support Technical Exchange 9


IBM Software Group

Recap: MQMD : CodedCharSetId


 Some well known CCSIDs:

 CCSID 437 -- this is an ASCII code page, used mainly under


OS/2, DOS, and Microsoft Windows console (OEM) windows.
 CCSID 850 -- this is another common ASCII code page, used
mainly on under OS/2, DOS, and Microsoft Windows console
(OEM) windows used on PCs in Europe and some AIX
locales.
 CCSID 819 -- this is the ISO 8859-1 standard Western
European ASCII code page. Used by most UNIX locales.
 CCSID 500 -- This is an EBCDIC code page, used mainly on
z/OS. It is known as the International codepage.
 CCSID 037 -- This is another popular EBCDIC code page,
used on OS/400. It is the US English codepage.

WebSphere Support Technical Exchange 10


IBM Software Group

Recap: FORMAT
 The queue manager built-in formats all have
names beginning with MQFMT. They are listed and
described in the WebSphere MQ Application
Programming Reference under the Format field of
the Message descriptor (MQMD)

 Some built-in formats

MQFMT_NONE (default)
MQFMT_STRING (for character data conversion)

WebSphere Support Technical Exchange 11


IBM Software Group

Recap: What Gets Converted?


For Channel Communication -- When a channel between two
WebSphere MQ Queue Managers is started, the two Queue Managers
will negotiate which CCSID and encoding protocol they will use for
channel communications.

Reference IBM Technote 7005729 : Data Conversion under


WebSphere MQ

WebSphere Support Technical Exchange 12


IBM Software Group

Recap: Customer Scenario 1


 Windows/Sender  z/OS /Receiver
WMQ Level 6.0.?.? WMQ Level 6.0
OS Level ??? OS Level ???
QMGR CCSID 437 QMGR CCSID 500

Problem : Characters |, ] and ! are not being properly


handled when sent from Windows to z/OS.

Reference trace file : CCSID37.CCSID500.doc

WebSphere Support Technical Exchange 13


IBM Software Group

Recap: Customer Scenario 1 cont


 Solution : Change the queue manager on z/OS to
CCSID 37 or a create a channel exit program that will
handle the conversion.

Reference IBM Technote 1207860 : Conversion


character difference between CCSID 37 and 500

WebSphere Support Technical Exchange 14


IBM Software Group

Code page 37 - Code page 500 differences

WebSphere Support Technical Exchange 15


IBM Software Group

Common problem code points


Character CCSID 437 CCSID 819 CCSID 500 CCSID 37
! 21 21 4F 5A
AA AC BA 5F
9B A2 B0 4A
| 7C 7C BB 4F
[ 5B 5B 4A BA
] 5D 5D 5A BB
^ 5E 5E 5F B0

WebSphere Support Technical Exchange 16


IBM Software Group

Customer Scenario 2
 Windows WMQ Client  z/OS /SVRCONN
WMQ Level 6.0.x WMQ Level 6.0
OS Level ??? OS Level ???
Putting Application CCSID 437 QMGR CCSID 500
Channel defined with CONVERT=Y Must have Client attachment

Problem: Messages put to z/OS queue with a CCSID of 437 are not
converted as expected.

WebSphere Support Technical Exchange 17


IBM Software Group

Customer Scenario 2 cont


 Solution : Messages PUT to the z/OS queue manager
or any platform should have MQFMT_STRING format
specified in the message header (MQMD).

 Reference: WebSphere MQ Application Programming


Reference (SC34-6062)

Reference trace file :


CCSID437.CCSID500.NO.MQSTR.doc

WebSphere Support Technical Exchange 18


IBM Software Group

The message is entirely character data


If the message you are passing is entirely character data then it is
very easy to have WebSphere MQ do all of the work for you. In the
application that is doing the MQPUT of the message, you should
set MQMD.Format to MQFMT_STRING. MQFMT_STRING is a
constant that equates to the string value MQSTR . This value in
the Format field will let WebSphere MQ know that the message
consists entirely of character string data.

Reference IBM Technote 7005729 : Data Conversion


under WebSphere MQ

WebSphere Support Technical Exchange 19


IBM Software Group

Recap: Common Procedure to ensure conversion


on a MQGET
 In the application doing the MQGET of the message, set MQMD.CodedCharSetId to
0 to pick up the default CCSID of the QMGR. This will ensure that WebSphere MQ
knows the correct CCSID to convert to.
 In the application that is doing the MQGET of the message, make sure you specify
MQGMO_CONVERT as one of the MQGMO.Options. This field has many options
that should be either added or bitwise ORed together if you need to specify more
than one.
 It is recommended that you always do the data conversion on the MQGET of the
message, since the message may take many hops across different platforms on its
way to its destination, and doing data conversion on each hop is a waste of
resources.
 If you specify CONVERT(YES) on the SENDER channel that sends the data to its
final destination, WebSphere MQ will convert the data into the CCSID of the queue
manager on the target system.

WebSphere Support Technical Exchange 20


IBM Software Group

Customer Scenario 3
 CICS TS 3.2  z/OS
Using CICS Bridge WMQ Level 6.0
OS Level ???
QMGR CCSID 500

Problem: No data conversion, binary zeroes returned, bad


messages, Abends, unpredictable results.

Solution : Apply CICS TS apar PK58227 AND WMQ apar PK59397

Local Fix : As a temporary work around for this issue change the
Sender channel definition to have YES for the CONVERT parameter.

WebSphere Support Technical Exchange 21


IBM Software Group

Customer Scenario 4
 Linux/SDR  iSeries/ RCVR
WMQ Level 6.0.1.1 WMQ Level 6.?.?.?
OS Level ?? OS Level ???
QMGR CCSID 819 QMGR CCSID 932

Problem: When a message is sent from Linux (819) to iSeries (932)


not all data is converted as expected. A return code of 2119 is
generated MQRC_NOT_CONVERTED.

Solution : Data conversion from code page 819 to 932 is not


supported. Customer will need to use an alternate CCSID, for
example 1208 (unicode).

Reference trace file : MQRC_NOT_CONVERTED.doc

WebSphere Support Technical Exchange 22


IBM Software Group

Customer Scenario 5
 Windows Client  iSeries/ SVRCN
WMQ Level 6.0.2.4 WMQ Level 6.0.2.4
OS Level ?? OS Level 5.3
QMGR CCSID ??? QMGR CCSID 37
Application CCSID 819 Application CCSID 1208

Problem: Messages put to iSeries queue with a CCSID of 819 are not converted
as expected. Server application has get option of MQGMO_CONVERT specified for
the MQGET api. Application fails with a RC 2190
MQRC_CONVERTED_STRING_TOO_BIG. Messages are not retrieved from the
queue.

Solution : Conversion between 819 (SBCS) and 1208 (MBCS) will fail if
application buffer size is not large enough to handle data. Verify the size of the
receiving buffer length. Remember when converting from a SBCS to MBCS to
double the size of the buffer.

Reference trace file : MQRC_CONVERTED_STRING_TOO_BIG.doc

WebSphere Support Technical Exchange 23


IBM Software Group

Customer Scenario 6
 Windows Client  iSeries/ SVRCN
WMQ Level 6.0.2.4 WMQ Level 6.0.2.4
OS Level ?? OS Level 5.3
QMGR CCSID ??? QMGR CCSID 37
Application CCSID 819 Application CCSID 1208
Data Length 25 Bytes Buffer Length 24 Bytes
MQGET options: Convert, Accept
Truncated message

Problem: Messages put to iSeries queue with a CCSID of 819 are not
converted as expected. Server application has get options of
MQGMO_CONVERT and MQGMO_ ACCEPT_TRUNCATED_MSG
specified for the MQGET api, which fails with a RC 2079
MQRC_TRUNCATED_MSG_ACCEPTED. However, the messages are
retrieved successfully, but not all data is converted.

WebSphere Support Technical Exchange 24


IBM Software Group

Customer Scenario 6 cont.


 Solution : Conversion between 819 (SBCS) and 1208
(MBCS) will fail if application buffer size is not large enough
to handle data. In this scenario the customer has specified
MQGET option MQGMO_ACCEPT_TRUNCATED_MSG ,
which will allow the message to be retrieved from the queue.
However, the buffer length specified for the application
cannot hold the entire converted data. Verify the size of the
receiving buffer length. Remember when converting from a
SBCS to MBCS/DBCS to double the size of the buffer.

 Reference trace file :


MQRC_CONVERTED_MSG_TOO_BIG.doc

WebSphere Support Technical Exchange 25


IBM Software Group

What Unicode CCSID can be used?


 Reference IBM Technote 7005729: Data
Conversion under WebSphere MQ

1200
1208
13488

 NOTE: There are many unicode CCSIDs, however


you must select which one is best for your
environment.

WebSphere Support Technical Exchange 26


IBM Software Group

Problem Determination
 Complete problem description
 WMQ Environment description
 WMQ Level, for example WMQ 6.0.2.2
 OS level, for example iSeries 5.3.0; Windows XP SP2
 QMGR CCSID, for example 37, 437, 5026, 1047
 QMGR Sender Channel CONVERT setting
 Putting Application MQMD CCSID and Format setting
 Getting Application MQMD CCSID setting
 Capture the message on the XMITQ.
Verify if it is correct.
 Capture the message on the Destination Queue.
Verify if it is correct
 WMQ Trace on both sending and receiving servers
 IBM WebSphere MQ Data Conversion : Technote 7010477

WebSphere Support Technical Exchange 27


IBM Software Group

Useful Links
 WebSphere MQ Support
 www.ibm.com/software/integration/wmq/support
 System i Globalization
 http://www.ibm.com/servers/eserver/iseries/software/globali
zation/
 Unicode Home Page
 http://www.unicode.org/
 Character Sets and Code Pages, Encoding
 http://www.i18nguy.com/unicode/codepages.html
 Language Tools
 http://www.google.com/language_tools?hl=en
 http://babelfish.altavista.com/

WebSphere Support Technical Exchange 28


IBM Software Group

Useful Tools
 WebSphere MQ Supportpac
 IH03 (IH03: WebSphere Message Broker V6-Message display,
test & performance utilities)
IBM Technote 4000637
RFHUTIL and RFHUTILC allows enduser to place test message
to a queue with a particular CCSID

 WebSphere MQ Sample
 AMQSBCG (distributed (Windows and Unix))
 AMQSBCG4 (iSeries)
These programs are passed the name of a queue and a queue
manager. It then reads each message from the queue and
outputs the following to the stdout
Sample programs do not have MQGMO_CONVERT
specified
Formatted message descriptor fields
Message data (dumped in hex and, where possible,
character format)
 CSQ4BCL1 (z/OS)

WebSphere Support Technical Exchange 29


IBM Software Group

IH03: Message from i5/OS queue manager

WebSphere Support Technical Exchange 30


IBM Software Group

WebSphere Support Technical Exchange 31


IBM Software Group

WebSphere Support Technical Exchange 32


IBM Software Group

Message from z/OS queue manager

WebSphere Support Technical Exchange 33


IBM Software Group

iconv API
 On the i5/OS(qshell), AIX, Linux and z/OS (OMVS)
platforms, enduser can take advantage of the iconv
unix api/command to verify data conversion.

 Example from the i5/OS platform:


$
> iconv
iconv: 001-0098 Required argument is not specified.
iconv: 001-3051 usage: iconv -f fromCCSID -t toCCSID [file ...]
$

NOTE: On the i5/OS platform, use the DSPF command with F10=Display Hex to
view data in hexdecimal format.

WebSphere Support Technical Exchange 34


IBM Software Group

iconv API cont


> iconv -f 37 -t 37 ccsidfile
this is a test message. Now is the time for all good men to come to the aid of their country.
> iconv -f 37 -t 819 ccsidfile
$ / _/ +? _ ? /%% ?? _> ? ?_ ? / ?
?>`
> iconv -f 37 -t 437 ccsidfile
$ / _/ +? _ ? /%% ?? _> ? ?_ ? / ?
?>`
> iconv -f 37 -t 500 ccsidfile
this is a test message. Now is the time for all good men to come to the aid of their country.
> iconv -f 37 -t 1200 ccsidfile
$ / _/ +? _ ? /%% ?? _
> ? ?_ ? / ? ?>`
> iconv -f 37 -t 1208 ccsidfile
$ / _/ +? _ ? /%% ?? _> ? ?_ ? / ?
?>`

WebSphere Support Technical Exchange 35


IBM Software Group

Documentation
 The following manuals discuss MQ conversion topics. Just refer to
the index of these manuals for conversion or CCSID
 WMQ Commands Manual Reference
 WMQ Application Programming Reference
 WMQ Application Programming Guide
Sample programs documented in this manual
 WMQ Intercommunication
 WMQ Conversion document

 All manuals and documents can be found on the WebSphere MQ


Support webpage:
www.ibm.com/software/integration/wmq/support

WebSphere Support Technical Exchange 36


IBM Software Group

Summary
 Terms and Definitions

 IBM Technotes
 7010477 Webcast replay: WebSphere MQ Data
Conversion

 Useful Links
 WebSphere MQ Product Support page
www.ibm.com/software/integration/wmq/support
Search on Data Conversion to review IBM
Technotes regarding this topic

WebSphere Support Technical Exchange 37


IBM Software Group

Additional WebSphere Product Resources


 Discover the latest trends in WebSphere Technology and implementation,
participate in technically-focused briefings, webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/
 Learn about other upcoming webcasts, conferences and events:
www.ibm.com/software/websphere/events_1.html
 Join the Global WebSphere User Group Community: www.websphere.org
 Access key product show-me demos and tutorials by visiting IBM Education
Assistant: ibm.com/software/info/education/assistant
 Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf
.html
 Sign up to receive weekly technical My support emails:
www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange 38


IBM Software Group

Questions and Answers

WebSphere Support Technical Exchange 39