Over the last days I had the chance to test Datameer Analytics Solution (DAS). DAS is a platform for Hadoop which includes data source integration, an analytics engine and visualisation functionality.
This promise of a fully integrated Big Data analysis process motivated me to test the product.
It really includes all required functionality for data management or ETL, it provides standard tools to analyse data and there are nice ways to build visualisation dashboards. For example, there are connectors for Twitter, IMAP, HDFS, or FTP available. All menus and processes are self-explaining and the complete interface is strongly Excel or spreadsheet oriented. If you are familiar with Excel you can do the analyses on your Big Data out of the box. For a fast on the fly analyses performance you only work with a subset of your data and the analyses you store will then be automatically transformed into a kind of procedure. In the end – or according to a schedule you set – you “run” the analyses on your Big Data: DAS collects the latest data for you, DAS creates MapReduce jobs in the background and updates all your spreadsheets and visualisations. To close the analyses circle you can use the connectors to write your results back to HDFS or a database as HBase or many more technologies.
DAS is really designed for Big Data. If you test it with small data you will be frustrated by the performance – the overhead of creating MapReduce jobs dominates in this situation. But as soon as you start with real Big Data analyses this overhead gets negligible and DAS is taking over a lot of your programming work.
My test infrastructure
The following figure provides a nice overview of the Datameer infrastructure. DAS supports many data sources, it runs on all Hadoop distributions, it provides a REST API and you can add plugins as connectors for other modelling languages such as R (#rstats).
I tested DAS Version 3.1.2 running on our MapR Hadoop cluster version 3.0.2. After getting the latest package version from the Datameer support the installation was straightforward and it worked out of the box. Thanks to Datameer for providing a full test license. There are severalonline tutorials and videos available and there are some tutorial apps. Apps are another great feature of Datameer. You can download Datameer Apps which include connectors, workbooks and visualisations for different analysis examples. And you can create your own app from your analyses and share them with your colleagues or the community.
My test data and analyses
I tested DAS with the famous “Airline on-time performance” data set consisting of flight arrival and departure details for all commercial flights within the USA, from October 1987 to April 2008. I downloaded all the data (including supplements) to maprfs, created connectors for the data and imported the data into a workbook.
In the workbook I tested many classical statistical counting analyses:
- grouping functionality for the airports and counting the number of flights
- grouping for the airlines and calculating different statistics as mean values for the air time
- using joins to add additional information like the airline name to the airline identifier
- doing sorts to extract the most interesting airports depending on different measures
I am not an Excel expert. So it took me some time to get used to this low level process of doing analyses on spreadsheets. But in the end it is a very intuitive process of creating analyses.
Every new analysis will be available in a new tab in your workbook. There are several nice functionalities to support your work. For example there is a “sheet dependencies” overview which provides information about the dependencies between sheets.
Apart from the classical analyses, DAS provides some data mining functionality. It is called “Smart Analystics”. So far, it covers k-means clustering, decision trees, column dependencies and recommendations. It works out of the box but is not yet on the level to be satisfying for real analyses. E.g. for k-means clustering there is no support for choosing the right number of clusters (k) and you can not switch between different distance functions (default is euclidean distance).
Finally, I visualised all my results in a nice “Infographic”. There are many different visualisation tools and parameters available. After playing around with the settings you can create a nice dashboard and share it with your colleagues.
Please be aware that the complete data set is about 5 GB. Importing the data set takes about 30 minutes and running the workbook took more than 3h in my case. In the end I split my analyses into several workbooks to improve the feasibility.
Summary
It was easy to get started with Datameer Analytics Solution (DAS). It is definitely a great tool to do Big Data Analyses without any detailed Hadoop or Big Data knowledge. Furthermore, it covers many use cases and provides all required functionality for your daily analysis process. However, as soon as your analyses get more complex, the limitations of Datameer become apparent and you will probably look for a more powerful tool set or start implementing your Big Data analyses directly on Hadoop.
Finally, Datameer supports many steps in the Big Data Analysis Process, it works efficiently and the usability is straight forward. But Big Data is more than ETL, data analysis and visualising the results. You should never forget to think about your use case and the business value that you want to extract from your data. In the end, this is what should guide you in choosing the tools and/or implementations to use.
If you would like to know more about how Big Data analysis can help your business, feel free to contact us!