Imagine you are an organization sitting on a tons of transactional and other business data which is stored in structured database. Now, you have to analyse it on a daily basis to get vital insights about sales, customers, channels and many other business parameters, what will be your approach?
First thing which will come to your mind is using traditional BI tools or latest in-memory processing tools that can be used to perform analytics and get some good insights, but oh boy, remember that the size of the data is huge, which may not be fitting in memory or will be extremely slow on BI tools. Now what? Implementing big data technology like Hadoop or Spark? But your data, which is big, is still not “Big Data” and is still residing in structured database. So big data solution is an overkill. Also, implementing big data has its own challenges like implementation time/cost, maintenance of clusters, handling failure scenarios etc. RoI on Big Data implementations has been alluding several organizations.
In short, you are in a situation with huge data, which can’t be analysed using in-memory processing and also implementing big data/ cluster computing technology is not feasible. Big ambiguity! Isn’t it? What if there would have been some intermediate way where you neither have to implement cluster computing nor have to worry about running out of memory. Sounds magical!!! Even we felt the same till the time we built the state of the art technology which we call “NSBD” i.e. “Not So Big Data” at G-Square Solutions. NSBD is a stack of columnar databases, ETL tools for transferring data from existing sources, libraries for data munging and analysis, and a set of front-end tools for quickly analyzing the data
NSBD has the capability of handling huge structured data and do some simple as well as very complex tasks like aggregations, group by, sorting, merging, correlations, machine learning etc. which are usually required for doing most of the analysis. NSBD is very fast as compared to traditional databases like MySQL and NoSQL databases like MongoDB. Still need more? Then here you go, some operations in NSBD are faster than in memory processing like R or Pandas also. That means, you can analyse your data keeping it on disk, without putting much load on memory and still getting faster results than in memory processing. Wow! What else is needed for a Data Analysts and a Data Scientists, won’t it make their life easier than ever before?
NSBD is a new born baby and it is still growing, that means you your feedback and feature requests will shape the product. Many things are already there and more things are coming soon, so stay tuned for updates. Let’s adopt the new way of analytics, NSBD.