Smart and Connect the World

RSS header - this is hidden

Don't get lost in the Data Wonderland

Posted by Stephen Copley on Jul 31, 2017 5:28:40 PM

Data moves fast in today's world. Not only do we need to move data from place to place, but we all want to know what the data is doing, what value it can create, and how it will change with time. Most of the companies around the world who work with data and related technologies are doing the incredible task of analyzing the data and drawing inferences that can help shape a better tomorrow. Never before has the importance of data been so critical to our day to day lives. People are doing amazing work with the data - curing diseases, fighting climate change, revolutionizing transportation and energy, making entire populations more effective and so on.

don't get lost in the data wonderland

Figure 1: Don't get lost in the data wonderland

However, there are always some challenges when working with the data - Firstly, analyzing the data and using correct prediction models to actually obtain something meaningful. Second, being able to transform the data beforehand so it's easy to work with and can be used for the correct use case, or business process. It is in this later challenge where RoboMQ comes into the picture.

We will take the example of an employee named Alice at a company named WonderLand.  Alice has a lot of excel sheets that she works with on a daily basis. Excel sheets are one of the quickest and easiest ways to store and organize data, and are used by millions around the world. Mr. Rabbit, is the company's MVP when it comes to analyzing data and helping save the company millions of dollars in the future. But Mr. Rabbit has a problem. He realizes he does not have enough data to make a conclusive decision about selling a product called, "UnBirthday Tea"  in December or not. If somehow Alice can send all her excel sheets, which have all the intel for Mr. Rabbit to make the decision, that would allow Mr. Rabbit to make an informed decision about the product, "UnBirthday Tea".

So let's say Alice does send Mr. Rabbit all her data. Now Mr. Rabbit knows he has to work with Excel sheets and he has done it before. He had previously written scripts  to convert her data into something he can work with. For example, last time he was working with her data, he decided to write a Python script and use Pandas to make it into matrices and dictionaries that he could work with. But he can't work with all of that data at once, so he decides to store the data into his favorite database - SQL or NoSQL. Maybe he is working with graphs and running a Neo4j server on the cloud, so he decides to transform that data and put it there. He then realizes that the month of December is cold and people want to give gifts during the holiday season and the product, "UnBirthday Tea" can have more sales if he has the geographic data about the area where the product is going to be sold.

Now Mrs. Cheshire from the company MadHatter is trying to sell this data for a cost. It is important for Mr. Rabbit to get it, but Mrs. Cheshire has all the data in a SQL Server whereas Mr. Rabbit had already decided to put his data into a graph database. So, should he write a script again? But now Mr. Rabbit is starting to get very concerned that he may be late for a very important date! If he can't do it within a week, he might lose the competitive edge and their rival company will launch a similar product and thereby cause WonderLand to incur a significant loss.

This is where RoboMQ comes in real handy. RoboMQ has thousands of connectors that are generic enough to transfer data from point A to point B (or from one system to another) and transform it in a usable format.  This makes it easier for Mr. Rabbit to just work on the data without ever getting bothered about the transformations. RoboMQ handles these transformation very effectively, so Mr. Rabbit will never have to spend time worrying if things will be late. The way data flows through RoboMQ microservices is akin to a conveyor belt that I encounter at the baggage claim at the Washington DC airports. The data is handled by containers on the conveyor belt, which are the microservices developed by RoboMQ. The transformations make the data much more readable and reliable in the form Mr. Rabbit needs converyor belts

Then Mrs. Cheshire calls up Mr. Rabbit and tells him she has more data which will be delivered daily. Mr. Rabbit asks RoboMQ, can you put in more containers on the conveyor belt? Sure, it's easy peasy, we just scale up the services on our swarms or kubernetes, or ECS. Now you can handle 2 times the data, maybe 3 times, maybe 100, whatever it is that your business requires.

So now, two conveyor belts from Alice and Mrs. Cheshire are making their way to Mr. Rabbit, and he does not need to worry about being late, or even checking his pocket watch. With the help of RoboMQ, Mr. Rabbit and Alice will integrate their data seamlessly to launch WonderLand's new product "UnBirthday Tea" successfully, and maybe even celebrate with a tea party!


Topics: Data Integration, About RoboMQ