Friday, 24 January 2014

Steps to Configure SSL on Websphere MQ Channels

Hi Dears .. First of all I would like to thank my colleagues in SV who actually prompted me to write this blog. Because, one of their query made me to think of writing it here so that it might be useful for others as well .. As the name of the blog says 'For Posterity'

So lets look at the steps to configure SSL between Websphere MQ Channels. I have used MQ 7.5 here which is the latest. Any one with MQ version 7.1 can use the below commands as it is. For previous versions there would be slight changes .. 

I have 2 queue managers QM1 and QM2 trying to establish communication over SSL. I assume that the queue managers are already created.

1. Create Queue Manager Key Repository

First step is the creation of queue manager key repository using the below commands

runmqckm -keydb -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -type cms -expire 365 -stash



runmqckm -keydb -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -type cms -expire 365 -stash

2. Create CA key repository

As we are acting as CA in this example, we have to create CA key repository as well. In real time scenario, this step would be handled by a different team

runmqckm -keydb -create -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -type cms -expire 365 -stash

3. Create and distribute the CA certificate

runmqckm -cert -create -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -dn "CN=WMQ CA, OU=WMQ, O=HP, L=Dubai, ST=DXB, C=UAE" -expire 365

4. Check that the CA certificate is listed in the CA key repository

use the below command to make sure that CA certificate is listed in the CA key repository. If the previous step was successful, you will find the label name of the CA certificate as output of the below command

runmqckm -cert -list -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password>

5. Extract the public CA certificate

Extract the CA certificate to pass it to QM1 and QM2

runmqckm -cert -extract -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -target "C:\TestZone\CA_KeyDB\wmqca.crt" -format ascii

6. Add the public CA certificate to QM1 and QM2 key repository

runmqckm -cert -add -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -label wmqca -file "C:\TestZone\CA_KeyDB\wmqca.crt" -format ascii


runmqckm -cert -add -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -label wmqca -file "C:\TestZone\CA_KeyDB\wmqca.crt" -format ascii

7. Check whether public CA certificate is listed in QM1 and QM2  key repository

runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password>


runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password>

8. Create QM1 and QM2's certificate request

Now that we have imported CA public certificate in to QM1 and QM2's repositories, next step is to generate the certificate of QM1 and QM2 and get it signed from CA. 

runmqckm -certreq -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -label ibmwebspheremqqm1 -dn "CN=QM1, OU=WMQ, O=HP, L=Dubai, ST=DXB, C=UAE" -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1req.arm"


runmqckm -certreq -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -label ibmwebspheremqqm2 -dn "CN=QM2, OU=WMQ, O=HP, L=Dubai, ST=DXB, C=UAE" -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2req.arm"

9. Sign QM1 and QM2's certificate using CA certificate


runmqckm -cert -sign -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1req.arm" -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -target "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1cert.arm" -format ascii -expire 365

runmqckm -cert -sign -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2req.arm" -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -target "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2cert.arm" -format ascii -expire 365

10. Add the signed certificates to QM1 and QM2's key repository

runmqckm -cert -receive -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1cert.arm" -format ascii


runmqckm -cert -receive -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2cert.arm" -format ascii

11. Check whether the certificate is listed in QM1 and QM2's key repository

runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password>


runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM1 password>

12. Set SSLKEYR property of QM1 and QM2 

ALTER QMGR SSLKEYR('C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1')


ALTER QMGR SSLKEYR('C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2')

13. Define Channel pair on QM1 and QM2

ON QM1 :
DEFINE CHANNEL(QM1.TO.QM2) CHLTYPE(SDR) CONNAME('localhost(1415)') XMITQ(QM2) SSLCIPH(TRIPLE_DES_SHA_US)
DEFINE CHANNEL(QM2.TO.QM1) CHLTYPE(RCVR) SSLCIPH(TRIPLE_DES_SHA_US)

ON QM2 :
DEFINE CHANNEL(QM2.TO.QM1) CHLTYPE(SDR) CONNAME('localhost(1414)') XMITQ(QM1) SSLCIPH(TRIPLE_DES_SHA_US)
DEFINE CHANNEL(QM1.TO.QM2) CHLTYPE(RCVR) SSLCIPH(TRIPLE_DES_SHA_US)

14. Start Channels and be secured !!

Now start your message channels and start sending messages securely .. To verify your channel configurations use 

DIS CHL(<Channel name>) SSLCIPH.

 If SSLCIPH is set to a non-blank value, SSL is enabled. 

DIS CHS(<Channel name>) SSLPEER

Both ends of the channel should show their peer's DNs 

Hoe this helps .. Please let me know if you have any queries on the above steps .. I will try my best to help you .. 



Monday, 20 January 2014

Communication between a non - clustered QMGR with a QMGR inside Cluster


     This post contains the detailed steps to be followed to establish communication between a Queue Manager (here after called QMGR) inside cluster and a QMGR outside cluster using a gateway QMGR. I had personally spent a lot of time in getting this correct. So thought of putting it together as a post for posterity. Here I am planning to start with 2 QMGRs in Cluster and the create a 3rd QMGR and establish communication with the clustered QMGRs

1.     Create QMGRs QM1 and QM2

                   crtmqm QM1
                   crtmqm QM2

2.     Alter both QM1 and QM2 as fullrespoditory QMGRs of the Cluster QMGRS

                  ALTER QMGR REPOS(QMGRS)

3.     Create listeners QM1.LSR, and QM2.LSR on QM1 and QM2 respectively and start the listeners

 DEF LISTENER(QM1.LSR) TRPTYPE(TCP) PORT(1234)
                   DEF LISTENER(QM2.LSR) TRPTYPE(TCP) PORT(1235)

4.     Create channels between QM1 and QM2 and start the channels

 ON QM1 :

                   DEF CHANNEL(TO.QM2) CHYTYPE(CLUSSDR) CONNAME('localhost(1235)') CLUSTER(QMGRS)
                   DEF CHANNEL(TO.QM1) CHYTYPE(CLUSRCVR) CONNAME('localhost(1234)') CLUSTER(QMGRS)

                   ON QM2:

                    DEF CHANNEL(TO.QM1) CHYTYPE(CLUSSDR) CONNAME('localhost(1234)') CLUSTER(QMGRS)
                    DEF CHANNEL(TO.QM2) CHYTYPE(CLUSRCVR) CONNAME('localhost(1235)') CLUSTER(QMGRS)

5.     Now the cluster set up is ready. Let’s now try to connect a QMGR inside cluster, let’s say QM2 with a QMGR outside cluster. To achieve this, we will create a new QMGR, QM3

 crtmqm QM3

6.     Create a listener QM3.LSR in QM3 and start the listener

 DEF LISTENER(QM3.LSR) TRPTYPE(TCP) PORT(1236)

7.     As QM3 is outside the cluster, we have to use distributed queuing technique to communicate with other QMGRs. In this case, QM1 acts as a gateway to the cluster QMGRS. I this case lets first see how a message can be sent from Outside the cluster, ie, from QM3 to a queue (Q2) on QMGR QM2 inside the Cluster.

Create a sender – receiver channel pair between QM3 and QM1

 ON QM3:

                   DEF QLOCAL(QM1) USAGE(XMITQ)
                   DEF CHANNEL(TO.CLUS) CHLTYPE(SDR) CONNAME('localhost(1234)') XMITQ(QM1)

                   ON QM1 :

                   DEF CHANNEL(TO.CLUS) CHLTYPE(RCVR)

8.     Create a Remote queue Q2 on QM3

 DEF QREMOTE(Q2) RNAME(Q2) RQMNAME(QM2) XMITQ(QM1)

9.     Create a cluster local queue Q2 on QM2

DEF QLOCAL(Q2) CLUSTER(QMGRS)

The setup is complete now. Make sure that all your channels are up and running. Now, if you try to post a message in to the remote queue Q2 on QM3, you can find that it safely lands on the local queue Q2 on QM2.

10.  So that’s the Inbound case. Let’s see how a reply from QM2 QMGR (which is inside cluster ) can fly back to QM3 QMGR (which is outside cluster). Here we have to use the concept of Queue Manager Alias. Here I am going to define a queue manager alias on QM1 (which is the gateway) and publish this info to the cluster.

 DEF QREMOTE(QM3.ALIAS) RNAME('') RQMNAME(QM3) XMITQ(QM3)

11. Create a sender – receiver channel pair from QM1 to QM3

                   ON QM1 :

                   DEF QLOCAL(QM3) USAGE(XMITQ)
                   DEF CHANNEL(FROM.CLUS) CHLTYPE(SDR) CONNAME('localhost(1236)') XMITQ(QM3)

                   ON QM3 :

                   DEF CHANNEL(FROM.CLUS) CHLTYPE(RCVR)

12. Create a local queue Q3 on QM3.

                   DEF QLOCAL(Q3)

Now if an application (App2) connected to QM2 try to post a message to Q3 on QM3, the message will fly all the way to the intended destination. So how did it work ? The queue manager alias defined as a remote queue on QM1 published in the cluster did the trick for us.


Hope this will help beginners who is taking their first steps in MQ. Happy messaging !! 

Thursday, 2 January 2014

DiscoveredEG_ issue

I have been facing an issue with my Execution Groups in my Testing Environment. The EG names were changed to ‘DiscoveredEG_<UUID of the EG> and then finally the message flows deployed in different EGs of that particular broker was not responding to the input messages. We have finally resolved the issue. So I just thought of sharing it so that it may help dome one in future. 

Basically the ‘DiscoveredEG’ issue happens when the periodic synchronization between broker and configuration manager fails. This is a synchronous call and if the response from Broker fails to commit the message to SYSTEM.ADMIN.BROKER.QUEUE, the message will be backed out and hence configuration manager will receive an empty message as response from the broker. It can be because of various reasons like MQ Logs became full and the message will be backed out. As a result configuration manager will update its repository with empty names there by removing the EG names from its repository. When the next successful synchronization happens, EG names in configuration manager repository will be updated, but with ‘DiscoveredEG_<UUID of the EG>’. 

In my case the root cause of the issue was MQ logs. As we had high volume of messages coming in and we didn’t set our MQ logs properly, the COMMIT operation was failing and we were getting error messages in /var/adm/messages. The error message was as given below 
“WebSphere Broker v6102[6512]: [ID 702911 user.error] (<Broker Name>)[1]BIP2074E: A problem was detected with WebSphere MQ while issuing MQCMIT for WebSphere MQ Queue Manager '<Queue Manager Name>'. MQCC=1, MQRC=2003”. We started our analysis with the MQRC code 2003 which helped us to understand that the MQ logs are full. I followed the below steps 

1. I increased the number of both primary and secondary MQ Logs. This can achieved by editing the qm.ini file and restarting the queue manager. 
2. The EG names will be still showing as ‘DiscoveredEG_<UUID of the EG>’ in Toolkit. In order to bring it back to the old names, rename the EGs based on the UUIDs using Toolkit. Take extreme care while doing it because by mistake if you rename with a different name, it will result in the UUID mismatch and more complexities. 
3. After renaming, restart the message flows in the renamed EGs through Toolkit in order to reflect the changed EG names in Broker Database. Once this is completed successfully, ‘DiscoveredEG’ issue is resolved and you will have your EG names back in place.