Validating a classifier

This validation interface can be used to create validation datasets and to measure classifier accuracy.

Generating a validation set

Direct your browser to http://localhost/HistomicsML/validation.html?application=nuclei/.

Under Start a validation set enter a test set name and choose the dataset you want to select validation examples from. Enter the names of your positive and negative classes.

_images/validation-start.png

Note

A strict validation procedure should perform training and validation on separate images. A slide collection can be artificially split

prior to import to produce separate training and validation datasets.

The validation interface is similar to the Prime interface, providing a slide selector, a slide viewer, and a thumbnail gallery of annotated objects. Object boundaryies can be displayed by selecting Show Segmentation. Clicking Select Nuclei will activate the cursor to begin annotating objects. Choose the object class that you want to annotate using the radio button, and then select objects by double-clicking within their boundaries in the slide viewer.

_images/validation-select.png

Note

Thumbnail images of validation set objects are displayed at the top of the screen. The class label of these objects can be toggled with a single-click. A double-click will remove the object from the validation set.

Clicking Add will commit these objects to the validation set. The Save button will save the validation set to the database.

Reviewing a validation set

Annotations in a validation set can be reviewed using the review interface.

At the Validation menu under Continue a validation set, select the validation dataset and validation set name and click Continue. Click Review on the left panel to navigate to the review interface.

The review interface displays the annotated objects organized by class and slide. Thumbnail images of the objects are organized into columns by class. Clicking a thumbnail will bring that object into the field of view in the slide view. The thumbnails can be dragged/dropped to a different column to change the class label, or placed in the Ignore column to discard them from the set. Changes are instantly commited to the database (no additional button clicks are needed).

_images/validation-review.png

Validating a classifier

The Validation interface also enables users to measure classifier accuracy by applying a trained classifier to a validation dataset.

At the Validation menu under Validate classifier, select the training dataset and classifier name, and the validation dataset and validation set name.

The Validate button will generate a .csv file containing: 1. The predicted class labels for each object in the validation set (1 - positive, -1 - negative) 2. The prediction scores from the random forest classifier (ranged 0-1) 3. The false-positive rate 4. The true-positive rate 5. The precision and 6. the total accuracy.