pharmine

VisuMap is a high dimensional data visualizer. It provides a number of dimensionality reduction methods like principal component analysis, Sammon mapping, curvilinear component analysis, relational perspective map and SMACOF MDS. It also has a few data clustering methods such as K-mean clustering, agglomerative clustering, self-organizing map and metric sampling.

The website contains some sample maps, sample datasets for you to work on. There are also white papers, and demo videos on the software (which is only available after you register with the website).

To evaluate this software, I used my own dataset. In one of my previous research, I gathered three congeneric groups of compounds: penicillins, cephalosporins, fluoroquinolones. I compute fingerprints (1025 dimensions) using openbabel for these compounds and combined them into one dataset. Then I load the dataset into VisuMap and run it through each of the different dimensionality reduction methods.

Results from Principal component analysis. Yellow squares are cephalosporins, Red circles are penicillins, Blue triangles are fluoroquinolones

Results from Sammon mapping. Yellow squares are cephalosporins, Red circles are penicillins, Blue triangles are fluoroquinolones

Results from Curvilinear component analysis. Yellow squares are cephalosporins, Red circles are penicillins, Blue triangles are fluoroquinolones

Results from Relational perspective map. Yellow squares are cephalosporins, Red circles are penicillins, Blue triangles are fluoroquinolones

Results from SMACOF MDS. Yellow squares are cephalosporins, Red circles are penicillins, Blue triangles are fluoroquinolones

All the pictures above (except PCA) are the 2D maps produced by the various algorithms. Although the software can also produce 3D maps, it is not easy to visualize them as the software does not provide very good controls for rotating the map. I could not get the 3D animation to work in my VMWare machine so I don’t know whether it provides an easy way to view 3D maps. It will be good if the software adopts the way that molecular structure viewer software like Sybyl handles 3D structures (i.e. hold down right mouse button and move the mouse to rotate).

It can be seen from the pictures that the algorithms PCA, Sammon and MDS did a very good job in showing that there are three distinct groups from the obvious separation between the groups (The colours and shapes of the different groups were added in manually to enhance the visual effects. Bear in mind that when you process a dataset with unknown groupings, every point will appear to be the same. Thus the only way to differentitate groups is if there is an obvious separation band). For the other algorithms, the separation between the groups are not as good, although it can be seen that members of each group does not mix with those from other groups. The Sammon and MDS algorithm also correctly showed that penicillins and cephalosporins are closer to each other than they are to fluoroquinolones.

KNIME is very easy to use and is good for preprocessing of datasets and descriptors. Personally, among the various software, I enjoy using KNIME the most. It is a pity that it is weaker in its model building and validation portion. Hopefully the next major version of KNIME will address these issues.

RapidMiner has a very large set of operators, which makes it very suitable for comparing different machine learning/statistical methods. It is also very good for model building and validation. However, the learning curve for the software is rather steep.

Weka (KnowledgeFlow) is somewhat in between KNIME and RapidMiner. Like RapidMiner, it has quite a large number of components and like KNIME, it is relatively simple to use. However, it is not able to perform all the functions that are available in RapidMiner and its graphical user interface is not as friendly as KNIME.

TANAGRA is similar to RapidMiner in terms of the layout for representing an experimental procedure. However it has significantly less operators than RapidMiner. My initial impression of it is that it should be quite good for performing QSAR experiments. However, after using it, it seems like it is lacking in several important features.

Orange is similar to Weka (KnowledgeFlow) in terms of layout. However, like TANAGRA, it seems to be lacking in some important features for QSAR experiments.

A missing feature in all these software is the ability to perform parallel computing, either through job distribution among different computers in the network or through the use of all the cores in multi-core CPUs.

Table 1 shows a comparison of the five software for performing procedures that are widely used in QSAR experiments. The best software appears to be RapidMiner. At a first glance, Weka seems to be redundant since RapidMiner has incorporated most of its algorithms. However, it still contains some algorithms, especially in the area of descriptor selection, which are not available in other software. Although TANAGRA and Orange are the worst performing software among the five, they do have their own merits. For instance, TANAGRA has an interesting collection of statistical tests while Orange has some interesting prototypes like MeSH Term Browser. Personally, I will invest my time to learn KNIME, RapidMiner, and Weka well, and will use these three software for my future research work.

Table 1: Comparison of the four software for performing procedures that are widely used in QSAR experiments.

Lastly, I need to reiterate that the above comments and all the previous posts on these software are very subjective. They are subjective because I have a vested interest in QSAR type of modeling and also because I am not very familar with these software (I have never used them in any of my research projects). Thus there may be factual inaccuracies about my review (i.e. some procedures which I stated that a particular software is unable to do may be false). The authors of these software or readers who are experienced with these software are welcome to comment on these factual inaccuracies and I will update the posts to reflect the truth.

The previous post already provides the steps for model validation using cross-validation. So how do we validate a model using an independent validation set?

Validate model on an independent validation set

1. Put File widget (Data) to canvas and configure it to load a training set from a file.
2. Put Select Attributes widget (Data) to canvas and connect the output port from the File widget to its input port.
• Specify the attributes and class for the training set.
• Click on the Apply button.
3. Put K Nearest Neighbours widget (Classify) to canvas and connect the output port from the Select Attributes widget to its input port.
• Configure it by setting Number of neighbours to 3.
• Click on the Apply button.
4. Put File widget (Data) to canvas and configure it to load an independent validation set from a file.
5. Put Select Attributes widget (Data) to canvas and connect the output port from the second File widget to its input port.
• Specify the attributes and class for the independent validation set.
• Click on the Apply button.
6. Put Test Learners widget (Evaluate) to canvas.
• Connect the output port from K Nearest Neighbours widget to its Learner input port.
• Connect the output port from the first Select Attributes widget to its Data input port.
• Connect the output port from the second Select Attributes widget to its Separate Test Data input port.
• Configure it by choosing Test on test data.

It can be seen that Orange is able to validate a model using either cross-validation or an independent validation set. However, it seems that Orange is unable to save a model and thus the model has to be rebuild each time it is to be used for validating an independent validation set.

1. Put File widget (Data) to canvas and configure it to load a training set from a file.
2. Put Select Attributes widget (Data) to canvas and connect the
   output port from the File widget to its input port.
• Specify the attributes and class for the training set.
• Click on the Apply button.
3. Put K Nearest Neighbours widget (Classify) to canvas and connect the output port from the Select Attributes widget to its input port.
• Configure it by setting Number of neighbours to 3.
• Click on the Apply button.
4. Put Test Learners widget (Evaluate) to canvas.
• Connect the output port from K Nearest Neighbours widget to its Learner input port.
• Connect the output port from the Select Attributes widget to its Data input port.
• Configure it by choosing Cross-validation and setting the Number of folds to 10.

The above procedure shows how Orange can be used to train and assess the performance of a model. However, it is not possible to automatically determine the optimum parameter value (e.g. Number of neighbours to consider (k) in the above procedure) for a machine learning/statistical method. To determine the optimum parameter value, you have to do it manually by setting a parameter value, execute, record the overall error rates, set another parameter value, execute again, record the overall error rates and so on, until you have evaluated all the parameter values that you are interested in. Then the parameter value which gives the lowest overall error rates will be the optimum parameter value of the machine learning/statistical method for the training set.

Orange does not have any wrapper descriptor selection methods.

Orange does not have any capability for scaling descriptors. Zero marks for this one.

1. Put File widget (Data) to canvas and configure it to load a dataset from a file.
2. Put Data Sampler widget (Data) to canvas and connect the output port from the File widget to its input port.
• Configure it by choosing Random sampling and setting the Sample size to 80%.
• Click on the Sample Data button.
3. Put Save widget (Data) on the canvas. Connect the Examples output port from the Data Sampler node to the input node of the Save widget and configure it to save the training set to a file. Then click on the Save current data button.
4. Put Save widget (Data) on the canvas. Connect the Remaining Examples output port from the Data Sampler node to the input node of the Save and configure it to save the testing set to a file. Then click on the Save current data button.

As can be seen from the above procedure, it is very easy to partition a dataset randomly into a training set and testing set. However, Orange does not seems to contain other algorithms, like the Kennard and Stone algorithm, for partitioning datasets.

Orange (Snapshot 11 April 2008)

From their official website, “Orange is a component-based data mining software. It includes a range of preprocessing, modelling and data exploration techniques. It is based on C++ components, that are accessed either directly (not very common), through Python scripts (easier and better), or through GUI objects called Orange Widgets”. Orange is distributed under GPL.

If you install the current version of Orange, you will have a total of 77 widgets, with the following nodes distribution:

• Data: 15
• Classify: 14
• Evaluate: 6
• Visualize: 13
• Associate: 13
• Prototypes: 13
• Regression: 3

However, since I am interested in using it for QSAR experiments, I will only examine those nodes that are relevant. Basically, Orange can read data from five sources: text-delimited files (which include csv files), C4.5 files, and three other formats which I am not familar with. Orange cannot read data from SVM^light files, LIBSVM files or Microsoft Excel files. The lack of support for Microsoft Excel files is no big deal since you can easily convert them to csv format using Microsoft Excel. However, the lack of support for SVM^light and LIBSVM files will inconvenient users who are already using these two popular support vector machine softwares.

Orange has a few filter descriptor selection methods such as ReliefF, Information gain, Gain ratio and Gini gain.

Currently, Orange contains one algorithm for developing regression models and 10 algorithms for constructing classification models. It seems strange that Orange does not have multiple linear regression algorithm, which is the most basic of regression algorithms.

Orange has a Data Sampler widget that provides validation methods like cross-validation and leave-one-out.

Overall, my first impression of Orange is that it has a nice graphical user interface but it seems quite inadequate for QSAR experiments.

The previous post already provides the steps for model validation using cross-validation. TANAGRA does not provide any functionality for loading another dataset into the same diagram, or saving and loading a model. Thus TANAGRA is unable to validate an independent validation set (TANAGRA is able to validate on a testing set only if the testing set is derived using its Sampling operator).

1. Create a new diagram and configure it to load a training set from a file. This will put a Dataset operator on the diagram.
2. Put Define status operator (Feature selection) to diagram under the Dataset operator and configure it to set the correct attributes as Input and Target.
3. Put K-NN operator (Spv learning) to diagram under Define status operator and configure it.
4. Put Cross validation operator (Spv learning assessment) to diagram under K-NN operator and configure it.
5. Execute.

The above procedure shows how TANAGRA can be used to train and assess the performance of a model. However, it is not possible to automatically determine the optimum parameter value (e.g. Number of neighbours to consider (k) in the above procedure) for a machine learning/statistical method. To determine the optimum parameter value, you have to do it manually by setting a parameter value, execute, record the overall error rates, set another parameter value, execute again, record the overall error rates and so on, until you have evaluated all the parameter values that you are interested in. Then the parameter value which gives the lowest overall error rates will be the optimum parameter value of the machine learning/statistical method for the training set.