Beginner's Guide to Ultra Pipelines in Snaplogic

Ultra pipelines are those pipelines which can continuously consume documents from external sources. This is similar to a web service exposed on any platform that can continuously access messages and process them

A typical Ultra Pipelines deployment architecture would look like as in the below diagram

1. Application2. Load Balancer3. Feed Master4. Execution node

Example Pipeline

Here I am demonstrating a simple pipeline that accept that is exposed as a REST POST service that is continuously accepting documents from a consumer application and invoking a REST service at the back end.

Lets look at the expressions in AccessRequestData snap to access data sent in the request

Accessing data for records sent as POST - $content.<field Name>

Accessing data for records sent as GET - $query.queryParam[0]   (Because query params are represented as an array in the request)

This is enough for the ulttra pipeline to work. Now, navigate to the Manager and click on create Task. 

Once you create the ultra task, Snaplogic will provide a URL which is the endpoint for your Ultra task. Use postman or any other tool to verify your service

Containerization - A beginner's thought

We have been hearing about containers for a while now, especially with Docker gaining momentum in the market back in 2013. There is a notion that these are “cloud stuffs” as we often hear these words related to the Cloud technologies and we used to ignore saying “not my cup of tea”. But now, the world is moving towards these technologies and almost every business want to embrace them to capitalize every possible business opportunities and gain advantage over their competitors. Someone might argue about the relevance of this article now especially when Cloud technologies and containerization have already gained a lot of acceptance in the market. This is for those who are taking their baby steps in to the world of Cloud technologies and Containerization.

So, What is Containerization?     

       

The moment I heard the word ‘Container’ for the first time, the first image that came to my mind was the big cargo container. I believe the best and simplest way to start understanding containerization is to take analogy from the logistics business domain. Imagine the operational overhead a logistics company has to go through when they have to transport a variety of goods using different modes of transport like trucks, buses, ships etc. The company has to package good based on the medium of transport and this creates inefficiency. Now, imagine the same scenario where in the company starts using standard containers so that almost anything can be accommodated and transported much faster from one place to another and between the available modes of transfer. The same inefficiency and operational overheads exists when we move an application from an application developer’s machine to a test environment and from there to further higher environments. It could be because of compatibility issues, version mismatch, configuration issues and many other reasons. But, we end up losing time and effort in fixing the same issues from one environment to other. Containerization provides a solution to such inefficiencies. The below figure can help to better visualize about what I am talking about

Containerization is a standard and efficient way of packaging an application, its libraries and dependencies. Containerization is the result of initiatives to improve the efficient use of resources beyond what Virtualization and Virtual machines can offer. We all have been hearing about and using Virtual Machines (VMs) over a decade now. With Virtualization, the operating system and the applications running on it are packaged together as Virtual Machines. But, with Containerization, only the applications and its dependent objects are packaged together as containers. Multiple containers can run on the same Operating system and they share the operating system kernel. Having said that, virtualization and containerization are not completing technologies. Rather, they are complementary to each other.

 The below figure explains the difference between Virtualization and containerization at a high level

Why Containerization?

1.     Containerization enables applications to run anywhere by isolating the application and its dependencies. The container doesn’t care about the host on which it is running and the host doesn’t care about the containers that run on it. The interfaces are standard and hence more predictable.

2.    Containers remove the OS layer from the package. This makes the container a lot more smaller in size than a traditional VM. This enables accommodating more containers on a machine when compared to full blown VMs there by saving operational cost.

3.    As all dependencies of an application are bundled inside a container, it makes the application more portable.

4.    As containerized applications need not wait for the OS to bootup as in the case of VMs, they can be launched much faster than applications deployed in traditional manner

5.    Containerization approach helps in automated deployments there by supporting DevOps practices. Automated spawning of a new application instance based on predefined load levels can be easily achieved with containerized applications.

However, Containerization is not a silver bullet. There are certain areas which still need improvement and they are being taken care by major tech companies in the market and the contributions from active communities around the world.

#Cloud #Containers #IT

Message Broker/IIB: A thought on Logging Transactions

Hi Guys,

Here I am going to discuss mainly 2 approaches for logging transactions and at the end let’s do a comparative study between both the approaches. Here I assume that you are working in an environment where logging each message, along with the payload, at every stage of processing is very important.

Approach 1: Make use of the monitoring events

You can configure your message flows to emit monitoring events at various stages of processing. Either you can make use of the built in record-and-reply option provided by WMB/IIB or you can write custom application/message flows to process the monitoring events emitted by main(service) flows and persist them in a database or file or any medium of your choice.

Approach 2: Subflows to log message

You can write a subflow which constructs the log message and pass it to an application/message flow which process the log message and persist it. The subflows can be plugged to any stage of processing depending on your requirement.

	Monitoring Events	Sub Flow
Decoupling	Provides total decoupling from the main transaction processing message flows. The main flows processing transactions won’t be impacted if there is a failure to emit a monitoring event	Provides partial decoupling. Being a subflow plugged directly in to the message flow, any failure in it could result in the failure of main transaction
Throughput	Has impact on throughput. Under load, I could observe ~ 20% less throughput as compared to Approach 2	Better throughput when compared to Approach 1
Monitoring	Need manual action to switch on the monitoring events after every deployment. If your administrator fails to that, no messages would be logged even though mail message flows would be processing transactions as usual	No monitoring at all
Grouping Business Transactions	Monitoring events provide an option to group business transactions	To achieve similar grouping, additional code has to be written

That’s it for now. Feel free to write your views on the above post.

Integrating IIB 10.0.0.6 with Mongo DB

Integrating IIB 10 with Mongo DB

            Pre-requisites:

1.       Create a free account in Mongo labs

a.       Create a free account in Mongo labs, a Cloud based Mongo DB. Obviously free of cost and you get 500 MB space

b.      Create a Collection (Table in RDBMS)

Broker Configs:

1.       Check your Environment variable MQSI_WORKPATH and switch your work path to MQSI_WORKPATH

2.      Change directory to node_modules

3.      Run the command as below to install the mongodb connector. This might take up to a minute to complete

npm install loopback-connector-mongodb --save

4.      Now a config file datasources.json has to be created. So, switch directory to MQSI_WORKPATH\ connectors\loopback. Create a file with name datasources.json and copy the below contents to the file

{

  "MONGO": {

    "host": "ds031257.mlab.com",

    "port": "31257",

    "database": "esbdev",

    "name": "MONGO",

    "connector": "mongodb",

    "debug": "true"

  }

}

5.      Configure the username and password of mongodb at the broker level using the below command

mqsisetdbparms <Broker name> -n loopback::<securityid> -n username -p password

6.      Restart your broker

7.       Create a simple message flow with http input and reply nodes (it is just my preference) and connect them to a LoopbackRequest node. Please select the input message parsing domain as “JSON”

      

8.      Configure LBR node as below. Leave the other options as default

9.      You are good to go .. deploy the flow .. If it gets successfully deployed, half done ..

10.   Post a Json message and verify the Collection to find the data you sent stored as Key-value pairs (no need of defining any schema as MongoDB is NoSQL). Sample data is given below

 

That's it :-)

Message Broker returns "[Microsoft][ODBC Driver Manager] Driver does not support this function" error while trying to connect to Database

You may have come across a situation where in INSERT/UPDATE/DELETE statements in your message flow returns an exception "[Microsoft][ODBC Driver Manager] Driver does not support this function".

Surprisingly mqsicvp would give a successful response and Test Connection from ODBC configuration panel would get connected successfully. However, only your message flows will fail to connect to Database. To fix this, follow the below steps

1. Go to Control Panel > Administrative Tools > Data Sources (ODBC)
2. Select System DSN tab. Select the datasource of your interest and Click on configure tab
3. Under Advanced tab, select the CheckBox against "EnableSQLDescribeParm"

Now run your message flows. You will find records getting successfully inserted in to the Database tables (provided all other configurations are correct :-) )

Steps to Configure SSL on Websphere MQ Channels

Hi Dears .. First of all I would like to thank my colleagues in SV who actually prompted me to write this blog. Because, one of their query made me to think of writing it here so that it might be useful for others as well .. As the name of the blog says 'For Posterity'

So lets look at the steps to configure SSL between Websphere MQ Channels. I have used MQ 7.5 here which is the latest. Any one with MQ version 7.1 can use the below commands as it is. For previous versions there would be slight changes .. 

I have 2 queue managers QM1 and QM2 trying to establish communication over SSL. I assume that the queue managers are already created.

1. Create Queue Manager Key Repository

First step is the creation of queue manager key repository using the below commands

runmqckm -keydb -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -type cms -expire 365 -stash


runmqckm -keydb -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -type cms -expire 365 -stash

2. Create CA key repository

As we are acting as CA in this example, we have to create CA key repository as well. In real time scenario, this step would be handled by a different team

runmqckm -keydb -create -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -type cms -expire 365 -stash

3. Create and distribute the CA certificate

runmqckm -cert -create -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -dn "CN=WMQ CA, OU=WMQ, O=HP, L=Dubai, ST=DXB, C=UAE" -expire 365

4. Check that the CA certificate is listed in the CA key repository

use the below command to make sure that CA certificate is listed in the CA key repository. If the previous step was successful, you will find the label name of the CA certificate as output of the below command

runmqckm -cert -list -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password>

5. Extract the public CA certificate

Extract the CA certificate to pass it to QM1 and QM2

runmqckm -cert -extract -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -target "C:\TestZone\CA_KeyDB\wmqca.crt" -format ascii

6. Add the public CA certificate to QM1 and QM2 key repository

runmqckm -cert -add -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -label wmqca -file "C:\TestZone\CA_KeyDB\wmqca.crt" -format ascii


runmqckm -cert -add -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -label wmqca -file "C:\TestZone\CA_KeyDB\wmqca.crt" -format ascii

7. Check whether public CA certificate is listed in QM1 and QM2  key repository

runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password>


runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password>

8. Create QM1 and QM2's certificate request

Now that we have imported CA public certificate in to QM1 and QM2's repositories, next step is to generate the certificate of QM1 and QM2 and get it signed from CA. 

runmqckm -certreq -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -label ibmwebspheremqqm1 -dn "CN=QM1, OU=WMQ, O=HP, L=Dubai, ST=DXB, C=UAE" -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1req.arm"


runmqckm -certreq -create -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -label ibmwebspheremqqm2 -dn "CN=QM2, OU=WMQ, O=HP, L=Dubai, ST=DXB, C=UAE" -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2req.arm"

9. Sign QM1 and QM2's certificate using CA certificate


runmqckm -cert -sign -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1req.arm" -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -target "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1cert.arm" -format ascii -expire 365

runmqckm -cert -sign -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2req.arm" -db "C:\TestZone\CA_KeyDB\wmqca.kdb" -pw <CA password> -label wmqca -target "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2cert.arm" -format ascii -expire 365

10. Add the signed certificates to QM1 and QM2's key repository

runmqckm -cert -receive -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password> -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1cert.arm" -format ascii


runmqckm -cert -receive -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM2 password> -file "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2cert.arm" -format ascii

11. Check whether the certificate is listed in QM1 and QM2's key repository

runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1.kdb" -pw <QM1 password>


runmqckm -cert -list -db "C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2.kdb" -pw <QM1 password>

12. Set SSLKEYR property of QM1 and QM2 

ALTER QMGR SSLKEYR('C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM1\ssl\qm1')


ALTER QMGR SSLKEYR('C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\QM2\ssl\qm2')

13. Define Channel pair on QM1 and QM2

ON QM1 :

DEFINE CHANNEL(QM1.TO.QM2) CHLTYPE(SDR) CONNAME('localhost(1415)') XMITQ(QM2) SSLCIPH(TRIPLE_DES_SHA_US)

DEFINE CHANNEL(QM2.TO.QM1) CHLTYPE(RCVR) SSLCIPH(TRIPLE_DES_SHA_US)

ON QM2 :

DEFINE CHANNEL(QM2.TO.QM1) CHLTYPE(SDR) CONNAME('localhost(1414)') XMITQ(QM1) SSLCIPH(TRIPLE_DES_SHA_US)

DEFINE CHANNEL(QM1.TO.QM2) CHLTYPE(RCVR) SSLCIPH(TRIPLE_DES_SHA_US)

14. Start Channels and be secured !!

Now start your message channels and start sending messages securely .. To verify your channel configurations use 

DIS CHL(<Channel name>) SSLCIPH.

 If SSLCIPH is set to a non-blank value, SSL is enabled. 

DIS CHS(<Channel name>) SSLPEER. 

Both ends of the channel should show their peer's DNs 

Hoe this helps .. Please let me know if you have any queries on the above steps .. I will try my best to help you .. 

Communication between a non - clustered QMGR with a QMGR inside Cluster

Hi friends,

     This post contains the detailed steps to be followed to establish communication between a Queue Manager (here after called QMGR) inside cluster and a QMGR outside cluster using a gateway QMGR. I had personally spent a lot of time in getting this correct. So thought of putting it together as a post for posterity. Here I am planning to start with 2 QMGRs in Cluster and the create a 3rd QMGR and establish communication with the clustered QMGRs

1.     Create QMGRs QM1 and QM2

                   crtmqm QM1

                   crtmqm QM2

2.     Alter both QM1 and QM2 as fullrespoditory QMGRs of the Cluster QMGRS

                  ALTER QMGR REPOS(QMGRS)

3.     Create listeners QM1.LSR, and QM2.LSR on QM1 and QM2 respectively and start the listeners

 DEF LISTENER(QM1.LSR) TRPTYPE(TCP) PORT(1234)

                   DEF LISTENER(QM2.LSR) TRPTYPE(TCP) PORT(1235)

4.     Create channels between QM1 and QM2 and start the channels

 ON QM1 :

                   DEF CHANNEL(TO.QM2) CHYTYPE(CLUSSDR) CONNAME('localhost(1235)') CLUSTER(QMGRS)

                   DEF CHANNEL(TO.QM1) CHYTYPE(CLUSRCVR) CONNAME('localhost(1234)') CLUSTER(QMGRS)

                   ON QM2:

                    DEF CHANNEL(TO.QM1) CHYTYPE(CLUSSDR) CONNAME('localhost(1234)') CLUSTER(QMGRS)

                    DEF CHANNEL(TO.QM2) CHYTYPE(CLUSRCVR) CONNAME('localhost(1235)') CLUSTER(QMGRS)

5.     Now the cluster set up is ready. Let’s now try to connect a QMGR inside cluster, let’s say QM2 with a QMGR outside cluster. To achieve this, we will create a new QMGR, QM3

 crtmqm QM3

6.     Create a listener QM3.LSR in QM3 and start the listener

 DEF LISTENER(QM3.LSR) TRPTYPE(TCP) PORT(1236)

7.     As QM3 is outside the cluster, we have to use distributed queuing technique to communicate with other QMGRs. In this case, QM1 acts as a gateway to the cluster QMGRS. I this case lets first see how a message can be sent from Outside the cluster, ie, from QM3 to a queue (Q2) on QMGR QM2 inside the Cluster.

Create a sender – receiver channel pair between QM3 and QM1

 ON QM3:

                   DEF QLOCAL(QM1) USAGE(XMITQ)

                   DEF CHANNEL(TO.CLUS) CHLTYPE(SDR) CONNAME('localhost(1234)') XMITQ(QM1)

                   ON QM1 :

                   DEF CHANNEL(TO.CLUS) CHLTYPE(RCVR)

8.     Create a Remote queue Q2 on QM3

 DEF QREMOTE(Q2) RNAME(Q2) RQMNAME(QM2) XMITQ(QM1)

9.     Create a cluster local queue Q2 on QM2

DEF QLOCAL(Q2) CLUSTER(QMGRS)

The setup is complete now. Make sure that all your channels are up and running. Now, if you try to post a message in to the remote queue Q2 on QM3, you can find that it safely lands on the local queue Q2 on QM2.

10.  So that’s the Inbound case. Let’s see how a reply from QM2 QMGR (which is inside cluster ) can fly back to QM3 QMGR (which is outside cluster). Here we have to use the concept of Queue Manager Alias. Here I am going to define a queue manager alias on QM1 (which is the gateway) and publish this info to the cluster.

 DEF QREMOTE(QM3.ALIAS) RNAME('') RQMNAME(QM3) XMITQ(QM3)

11. Create a sender – receiver channel pair from QM1 to QM3

                   ON QM1 :

                   DEF QLOCAL(QM3) USAGE(XMITQ)

                   DEF CHANNEL(FROM.CLUS) CHLTYPE(SDR) CONNAME('localhost(1236)') XMITQ(QM3)

                   ON QM3 :

                   DEF CHANNEL(FROM.CLUS) CHLTYPE(RCVR)

12. Create a local queue Q3 on QM3.

                   DEF QLOCAL(Q3)

Now if an application (App2) connected to QM2 try to post a message to Q3 on QM3, the message will fly all the way to the intended destination. So how did it work ? The queue manager alias defined as a remote queue on QM1 published in the cluster did the trick for us.

Hope this will help beginners who is taking their first steps in MQ. Happy messaging !!