Вы находитесь на странице: 1из 45

IBM Software Group

WebSphere MQ Data Conversion

Blanche Ishida and Karen Stanley


WebSphere MQ Support

WebSphere Support Technical Exchange

IBM Software Group

Agenda
 Terms and Definitions
 WebSphere MQ Message Header (MQMD)
 Customer Scenarios
 IBM Technotes
 Useful Links
 Summary
 Questions and Answers

WebSphere Support Technical Exchange

IBM Software Group

Terms and Definitions


 Character Set
 Is a defined set of characters used to represent textual
information (e.g. 0-9, a-z, A-Z, .,;:!?/-)
 A Character Set generally supports more than one
language
 Code Page
 (AKA Code set) A code page can be defined as a table
with a mapping of alphanumeric code and its binary
representation.
 Where each character in a character set is assigned a
numerical representation (often used interchangeably
with character set e.g. charset in HTML)

WebSphere Support Technical Exchange

IBM Software Group

Terms and Definitions


 A code point is the location of a character within
the code page
 CCSID Coded Character Set ID is a unique
number (0-65535) used by IBM to uniquely identify
a Coded Character Set and a Codepage.

WebSphere Support Technical Exchange

IBM Software Group

Terms and Definitions


 ASCII : American Standard Code for Information Interchange.
The code used as the basis for most codes apart from
EBCDIC.
 EBCDIC : Extended Binary Coded Decimal Information Code
(EBCDIC). It is typically used on z/OS, iSeries and VSE
 UNICODE : A code page which defines most of the
characters used by the languages now used throughout the
world. UCS-2, UTF-8, UTF-16 and UTF-32 for encodings.
 SBCS Single Byte Character Set. A code page with up to 256
characters which can be represented by a single 8 bit byte.
 DBCS Double Byte Character Set. A code page with up to
65536 characters.

WebSphere Support Technical Exchange

IBM Software Group

MQMD
 The Message Descriptor is not entirely character data, so
some of it will be converted as character data, some as
numeric, and some is passed unconverted.
 Note: MsgId, CorrelId, and Accounting Token are NOT
considered to be character data, and as such WILL NOT be
converted as character data. Many applications put
character data into these fields, but if the message is being
exchanged between ASCII and EBCDIC platforms, and the
application on the other side needs to be able to read these
fields, then it is the responsibility of the application (or a user
exit) to convert these fields. Also the data maybe in
hexadecimal representation for MsgID and CorrelID.

WebSphere Support Technical Exchange

IBM Software Group

WebSphere MQ Message Header


 StrucId Structure identifier
 Version Structure version
number
 Report Options for report
messages
 MsgType Message type
 Expiry Message lifetime
 Feedback Feedback or reason
code
 Encoding Numeric encoding of
message data
 CodedCharSetId Character set
identifier of message data
 Format Format name of
message data

WebSphere Support Technical Exchange

Priority Message priority


Persistence Message persistence
MsgId Message identifier
CorrelId Correlation identifier
BackoutCount Backout counter
ReplyToQ Name of reply queue
ReplyToQMgr Name of reply queue
manager
 UserIdentifier User identifier








 AccountingToken Accounting token

IBM Software Group

WebSphere MQ Message Header cont


 ApplIdentityData Application
data relating to identity
 PutApplType Type of
application that put the
message
 PutApplName Name of
application that put the
message
 PutDate Date when message
was put in GMT
 PutTime Time when message
was put in GMT
 ApplOriginData Application data
relating to origin

WebSphere Support Technical Exchange

 Note: The remaining fields are


ignored if Version is less than
MQMD_VERSION_2.
 GroupId Group identifier
 MsgSeqNumber Sequence number
of logical message within group
 Offset Offset of data in physical
message from start of logical
message
 MsgFlags Message flags
 OriginalLength Length of original
message

IBM Software Group

MQMD
There are two parts to a MQ message.
There is a Message Descriptor (MQMD), followed by User data.
D4C44040 00000001 00000000 00000008 MD ............
00007530 00000000 00000311 000001F4
..............4
D4D8E2E3 D9404040 00000000 00000000 MQSTR ........
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 546 CodedCharSetId : 437
Format : 'MQSTR '
Priority : 0 Persistence : 0

WebSphere Support Technical Exchange

IBM Software Group

MQMD : Encoding
 Encoding is generally taken to mean the method
that this platform uses to represent numeric data.
There are two general types:
 LittleEndian, used by Intel processors (e.g
Windows, Linux on Intel). e.g. the number 437
would be represented (in hex) as XBF01 (we
refer to this as byte swap)
 BigEndian. e.g. the number 437 would be
represented (in hex) as X01BF.

WebSphere Support Technical Exchange

10

IBM Software Group

Encoding scheme by platform


















Platform Encoding
zSeries
AIX
iSeries
HP-UX
Windows
SINIX
Sun Solaris (on SPARC processors)
Sun Solaris (on INTEL processors)
Linux (Intel)
Linux (zSeries)
NonStop Kernel (NSK)
OVMS Alpha
Open VMS VAX
Tru64 Unix

WebSphere Support Technical Exchange

Scheme
BigEndian
BigEndian
BigEndian
BigEndian
LittleEndian
BigEndian
BigEndian
LittleEndian
LittleEndian
BigEndian
BigEndian
BigEndian
BigEndian
BigEndian

11

IBM Software Group

CCSID
 The Queue Manager CCSID is important as it
defines the default CCSID for messages being
created using MQPUT and is the default CCSID
which will be used for conversion when messages
are retrieved using MQGET. If the channel option
CONVERT(YES) is set the conversion is attempted
for the receiving Queue Managers CCSID.
 To display the queue manager CCSID use the
DIS QMGR command

WebSphere Support Technical Exchange

12

IBM Software Group

MQMD : CodedCharSetId
 Some well known CCSIDs:
 CCSID 437 -- this is an ASCII code page, used mainly under
OS/2, DOS, and Microsoft Windows console (OEM) windows.
 CCSID 850 -- this is another common ASCII code page, used
mainly on under OS/2, DOS, and Microsoft Windows console
(OEM) windows used on PCs in Europe and some AIX
locales.
 CCSID 819 -- this is the ISO 8859-1 standard Western
European ASCII code page. Used by most UNIX locales.
 CCSID 500 -- This is an EBCDIC code page, used mainly on
z/OS. It is known as the International codepage.
 CCSID 037 -- This is another popular EBCDIC code page,
used on OS/400. It is the US English codepage.

WebSphere Support Technical Exchange

13

IBM Software Group

MQMD : Format
 This indicates to the receiver of a message the format of the
application data in the message.
 The queue manager has a number of built-in formats
with names beginning MQ, for example MQFMT_STRING. If these
do not meet your needs, you can define your own formats (userdefined formats), but you must not use names beginning with MQ for
these. When you create and use your own formats, you must write a
data-conversion exit to support a program getting the message
using MQGMO_CONVERT.
The format name MQFMT_NONE is a special value that indicates
that the nature of the data in the message is undefined. As a
consequence, the queue manager does not attempt conversion
when the message is retrieved from the queue.

WebSphere Support Technical Exchange

14

IBM Software Group

FORMAT
 The queue manager built-in formats all have names beginning with
MQFMT. They are listed and described in the WebSphere MQ
Application Programming Reference under the Format field of the
Message descriptor (MQMD)
 Some built-in formats
MQFMT_NONE (default)
MQFMT_STRING (for character data conversion)
MQFMT_DEAD_LETTER_HEADER
MQFMT_IMS
MQFMT_IMS_VAR_STRING
MQFMT_PCF
MQFMT_RF_HEADER
MQFMT_RF_HEADER_2
MQFMT_TRIGGER
MQFMT_XMIT_Q_HEADER

WebSphere Support Technical Exchange

15

IBM Software Group

Customer Scenario 1
 iSeries/Sender
WMQ Level 5.3

 VSE/Receiver
WMQ Level 2.1.2

OS Level 530

OS Level ???

QMGR CCSID 37

QMGR CCSID 1047

Sender Convert *YES

Problem : Sender channel fails with message AMQ9541


Message . . . . :

CCSID supplied for data conversion not supported.

Cause . . . . . :
The program ended because, either the source CCSID '&1'
or the target CCSID '&2' is not valid, or is not currently supported.
Recovery . . . :
Correct the CCSID that is not valid, or ensure that the
requested CCSID can be supported.

WebSphere Support Technical Exchange

16

IBM Software Group

Customer Scenario 1 cont


 Solution : At the time this issue was submitted the
customer had the following options
Change CONVERT to *NO on the Sender channel
Change VSE QMGR CCSID to 924/37
 As of May/2007 CCSID 1047 has now been added,
along with CCSIDs 5346, 5347, 5349, 5350, 5351,
5352, and 5353 to the iSeries server. Reference apar
SE28865 (PTF SI277906 for OS/530 and SI27516 for
OS/540).

WebSphere Support Technical Exchange

17

IBM Software Group

Customer Scenario 2
 iSeries/Sender
WMQ Level 5.3.12

 zOS/Receiver
WMQ Level 5.3.1

OS Level 530

OS Level ???

QMGR CCSID 37

QMGR CCSID 5026

Sender Convert *NO

Problem : Enduser indicated that they have an iSeries


communicating with z/OS. The sender channel is going into
RETRYING status and message AMQ9503 (Channel
negotiation failed) is generated when the PNGMQMCHL is
issued.

WebSphere Support Technical Exchange

18

IBM Software Group

Customer Scenario 2 cont


 Solution : Found that the receiver/responder
determined it could not handle the Fap level and/or
encoding scheme being negotiated between the
channels.
 Work-Around
Implement an intermediate queue manager
iSeries (CCSID 37)  z/OS (CCSID 5035)
 z/OS (CCSID 5026)

WebSphere Support Technical Exchange

19

IBM Software Group

Customer Scenario 2 cont


 Work-Around
Implement server/requester channel pair
iSeries (CCSID 37)  z/OS (CCSID 5026)

This configuration will only work as long as the


initiator is the z/OS
WMQ z/OS L3 team did find that this was a
defect and created a usermod, however it was
not verified by the customer as they decided to
continue to function with the intermediate queue
manger in place.

WebSphere Support Technical Exchange

20

IBM Software Group

What Gets Converted,?


For Channel Communication -- When a channel between two
WebSphere MQ Queue Managers is started, the two Queue Managers
will negotiate which CCSID and encoding protocol they will use for
channel communications.

Reference IBM Technote 7005729 : Data Conversion under


WebSphere MQ

WebSphere Support Technical Exchange

21

IBM Software Group

Customer Scenario 3
 Windows/Sender
WMQ Level 5.3.10

 iSeries/Receiver
WMQ Level 5.3.11

OS Level ???

OS Level ???

QMGR CCSID 437

QMGR CCSID 500

Problem : Characters |, ] and ! are not being properly


handled when sent from Windows to iSeries.

WebSphere Support Technical Exchange

22

IBM Software Group

Customer Scenario 3 cont


 Solution : Change the queue manager on iSeries to
CCSID 37 or a create a channel exit program that will
then handle the conversion.
Reference IBM Technote 1207860 : Conversion
character difference between CCSID 37 and 500

WebSphere Support Technical Exchange

23

IBM Software Group

Code page 37 - Code page 500 differences

WebSphere Support Technical Exchange

24

IBM Software Group

Common problem code points


Character

CCSID 437

CCSID 37

CCSID 500

A5

5A

4F

F5

5F

BA

A4

4A

B0

F4

4F

BB

AB

BA

A4

BB

BB

A5

0B

B0

5F

WebSphere Support Technical Exchange

25

IBM Software Group

Customer Scenario 4
 Sun Solaris/WMQ Client
WMQ Level 5.3.12

 z/OS /SVRCONN and RCVR


WMQ Level 5.3.12

OS Level ???

OS Level ???

Putting Application CCSID 819

QMGR CCSID 500

 z/OS / Sender
WMQ Level
OS Level ???
QMGR CCSID 500
 Windows / WMQ Client
WMQ Level
OS Level
 Getting Application CCSID 1252

WebSphere Support Technical Exchange

Must have Client attachment

Problem: Messages put to z/OS


queue with a CCSID of 819 are
processed properly by the
Getting client (Windows)
application. However,
messages put by the z/OS
queue manager with CCSID of
500 are not converted by getting
client application.
26

IBM Software Group

Customer Scenario 4 cont


 Solution : Messages PUT from the z/OS queue
manager or any platform should have MQFMT_STRING
format specified in the message header. Also the
Getting Client application should make sure it has
CCSID 437 specified, along with MQGMO option
Convert.
 Reference: WebSphere MQ Application Programming
Reference (SC34-6062)

WebSphere Support Technical Exchange

27

IBM Software Group

The message is entirely character data


If the message you are passing is entirely character data then it is
very easy to have WebSphere MQ do all of the work for you. In the
application that is doing the MQPUT of the message, you should
set MQMD.Format to MQFMT_STRING. MQFMT_STRING is a
constant that equates to the string value MQSTR . This value in
the Format field will let WebSphere MQ know that the message
consists entirely of character string data.

Reference IBM Technote 7005729 : Data Conversion


under WebSphere MQ

WebSphere Support Technical Exchange

28

IBM Software Group

Common Procedure to ensure conversion on a


MQGET
 In the application doing the MQGET of the message, set MQMD.CodedCharSetId to
0 to pick up the default CCSID of the QMGR. This will ensure that WebSphere MQ
knows the correct CCSID to convert to.
 In the application that is doing the MQGET of the message, make sure you specify
MQGMO_CONVERT as one of the MQGMO.Options. This field has many options
that should be either added or bitwise ORed together if you need to specify more
than one.
 It is recommended that you always do the data conversion on the MQGET of the
message, since the message may take many hops across different platforms on its
way to its destination, and doing data conversion on each hop is a waste of
resources.
 If you specify CONVERT(YES) on the SENDER channel that sends the data to its
final destination, WebSphere MQ will convert the data into the CCSID of the queue
manager on the target system.

WebSphere Support Technical Exchange

29

IBM Software Group

Customer Scenario 5
 iSeries /SDR
WMQ Level 5.3.12
OS Level 5.2
QMGR CCSID 420

 Tandem /RCVR
WMQ Level 5.1
OS Level K1000
QMGR CCSID 1089
 Windows / RCVR
WMQ Level 6.0.0.0
OS Level 2003 SP1
QMGR CCSID 1256

Problem: When a message is put with Arabic character LamAlif it is not being converted properly between CCSID 420 and
CCISD 1089(Tandem) or 1256(Windows).

WebSphere Support Technical Exchange

30

IBM Software Group

Customer Scenario 5 cont


 Solution :This is not a character conversion limitation for
WebSphere MQ, as not all characters can be converted
from one CCSID to another. In this case the Lam-Alif
character does not exist in CCSID/Code Page 1089 nor
1256, so a substitution character would be used, for
example X1A.
Enduser would need to specify a user defined message
format and write their own conversion code.
 Reference: WebSphere MQ Application Programming
Reference (SC34-6062)

WebSphere Support Technical Exchange

31

IBM Software Group

Problem Determination
 Complete problem description
 WMQ Environment description
WMQ Level, for example WMQ 6.0.2.2
OS level, for example iSeries 5.3.0; Windows XP SP2
QMGR CCSID, for example 37, 437, 5026, 1047
QMGR Sender Channel CONVERT setting
Putting Application MQMD CCSID and Format setting
Getting Application MQMD CCSID setting
Capture the message on the XMITQ.
Verify if it is correct.
 Capture the message on the Destination Queue.
Verify if it is correct








 WMQ Trace on both sending and receiving servers


 IBM MustGather Technote : 1176955

WebSphere Support Technical Exchange

32

IBM Software Group

IBM Technotes
MustGather: Documentation required by
WebSphere MQ data conversion

1176955

Data Conversion under WebSphere MQ

7005729

Conversion character differences between


CCSID 037 and CCSID 500

1207860

Data conversion in MQ intermittently does


not behave as expected

1240018

EBCDIC NL character is not converting


correctly in ASCII 0x15 (EBCDIC NL) 
converts to 0xDA

1116311

AMQ6174 MQRC 2119 putting a message


to a remote queue

1109867

WebSphere Support Technical Exchange

33

IBM Software Group

IBM Technotes cont


2110 MQRC Format Error

1232346

2080 MQRC TRUNCATED MSG FAILED

1167205

Data conversion does not occur in a


clustering environment

1268752

What CCSID is used by default for


WebSphere MQ client messages

1215215

MQ data conversion and data conversion


exit programs

1167007

WebSphere Support Technical Exchange

34

IBM Software Group

Useful Links
 WebSphere MQ Support
 www.ibm.com/software/integration/wmq/support
 System i Globalization
 http://www.ibm.com/servers/eserver/iseries/software/globali
zation/
 Unicode Home Page
 http://www.unicode.org/
 Character Sets and Code Pages, Encoding
 http://www.i18nguy.com/unicode/codepages.html
 Language Tools
 http://www.google.com/language_tools?hl=en
 http://babelfish.altavista.com/

WebSphere Support Technical Exchange

35

IBM Software Group

Useful Tools
 WebSphere MQ Supportpac
 IH03 (IH03: WebSphere Message Broker V6-Message display,
test & performance utilities)
IBM Technote 4000637
RFHUTIL and RFHUTILC allows enduser to place test message
to a queue with a particular CCSID
 WebSphere MQ Sample
 AMQSBCG (distributed (Windows and Unix))
 AMQSBCG4 (iSeries)
These programs are passed the name of a queue and a queue
manager. It then reads each message from the queue and
outputs the following to the stdout
Formatted message descriptor fields
Message data (dumped in hex and, where possible,
character format)

WebSphere Support Technical Exchange

36

IBM Software Group

AMQSBCG/AMQSBCG4
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : '
'
Priority : 0 Persistence : 0
MsgId : X'C1D4D840C3D6D4D4D6D540404040404046D2D73320001402'
CorrelId : X'000000000000000000000000000000000000000000000000'
BackoutCount : 0
ReplyToQ
:'
'
ReplyToQMgr : 'COMMON
'
** Identity Context
UserIdentifier : 'STANLEY '
AccountingToken :
X'0F40404040404040404040404040404000000000000000000000000000
000008'
ApplIdentityData : '
'

WebSphere Support Technical Exchange

37

IBM Software Group

AMQSBCG/AMQSBCG4 cont
** Origin Context

PutApplType : '8'
PutApplName : '021195/STANLEY/QP0ZSPWT
PutDate : '20070903' PutTime : '18330499'
ApplOriginData : ' '

'

GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset
: '0'
MsgFlags
: '0'
OriginalLength : '-1'
**** Message
00000000:
00000010:
00000020:
00000030:

**** length - 52 bytes

4A61 7661 2076 6572 7369 6F6E 206F 6620 '//.?>.?.'


7468 6520 4D51 5055 5431 2061 7069 2020 '.(&../..'
203A 20A3 A331 302E 3030 2024 2420 A2A2 '...tt.........ss'
2023 2320
'....
'

WebSphere Support Technical Exchange

38

IBM Software Group

z/OS Browse Sample


 WebSphere MQ Support pac
MA17 (WebSphere MQ for z/OS Message
Handler Sample)
IBM Technote 4000071

WebSphere Support Technical Exchange

39

IBM Software Group

z/OS MA17 browse sample


Queue Manager : RTP4
Queue
: RTP4.QL
Forward to Q Mgr : RTP6
Forward to Queue :

:
:

:
:

Action : : (D)elete (F)orward


Message Content :
-----------------------------------------------------------------------------Message Descriptor
StrucId
: `MD `
Version
: 000000001
Report
: 000000000
MsgType
: 000000008
Expiry
: -00000001
Feedback
: 000000000
Encoding
: 000000785
CodedCharSetId : 000000037
Format
: `MQSTR `
Priority
: 000000000
Persistence
: 000000000
MsgId
: `D4D8E3D7E4E361F2F0F0F7F0F8F3F0F1F6F4F9F3F6F9F1F6`X
CorrelId
: `D4D8E3D7E4E361F2F0F0F7F0F8F3F0F1F6F4F9F3F6F9F1F6`X
BackoutCount : 000000002
ReplyToQ
:`
`
ReplyToQMgr
: `RTP6
`

WebSphere Support Technical Exchange

40

IBM Software Group

Documentation
 The following manuals discuss MQ conversion topics. Just refer to
the index of these manuals for conversion or CCSID
 WMQ Commands Manual Reference
 WMQ Application Programming Reference
 WMQ Application Programming Guide
Sample programs documented in this manual
 WMQ Intercommunication
 WMQ Conversion document
 All manuals and documents can be found on the WebSphere MQ
Support webpage:

www.ibm.com/software/integration/wmq/support

WebSphere Support Technical Exchange

41

IBM Software Group

CCSIDs specified in this presentation


CCSID

Description

37

USA, Canada, Netherlands, Portugal, Brazil, Australia, New


Zealand

420

Arabic EBCDIC

437

USA PC-Data

500

International EBCDIC

819

ISO 8859-1 ASCII

1047

Latin 1 (Open Systems)

1089

Arabic ISO 8859-6

1252

MS-WIN Latin-1

1256

MS-WIN Arabic

WebSphere Support Technical Exchange

42

IBM Software Group

Summary
 Terms and Definitions
 Conversion vs. Translation
 CCSID (Coded Character Set Identifier)
 Encoding
 IBM Technotes
 Useful Links
 WebSphere MQ Product Support page
www.ibm.com/software/integration/wmq/support
Search on Data Conversion to review IBM
Technotes regarding this topic

WebSphere Support Technical Exchange

43

IBM Software Group

Additional WebSphere Product Resources




Discover the latest trends in WebSphere Technology and implementation,


participate in technically-focused briefings, webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/

Learn about other upcoming webcasts, conferences and events:


www.ibm.com/software/websphere/events_1.html

Join the Global WebSphere User Group Community: www.websphere.org

Access key product show-me demos and tutorials by visiting IBM Education
Assistant: ibm.com/software/info/education/assistant

Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf
.html

Sign up to receive weekly technical My support emails:


www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange

44

IBM Software Group

Questions and Answers

WebSphere Support Technical Exchange

45

Вам также может понравиться