Validating a classifier¶
This validation interface can be used to create validation datasets and to measure classifier accuracy.
Generating a validation set¶
Direct your browser to http://localhost/HistomicsML/validation.html?application=nuclei/.
Under Start a validation set enter a test set name and choose the dataset you want to select validation examples from. Enter the names of your positive and negative classes.
Note
A strict validation procedure should perform training and validation on separate images. A slide collection can be artificially split
prior to import to produce separate training and validation datasets.
The validation interface is similar to the Prime
interface, providing a slide selector, a slide viewer, and a thumbnail gallery of annotated objects.
Object boundaryies can be displayed by selecting Show Segmentation
. Clicking Select Nuclei
will activate the cursor to begin annotating objects.
Choose the object class that you want to annotate using the radio button, and then select objects by double-clicking within their boundaries in the slide viewer.
Note
Thumbnail images of validation set objects are displayed at the top of the screen. The class label of these objects can be toggled with a single-click. A double-click will remove the object from the validation set.
Clicking Add
will commit these objects to the validation set. The Save
button will save the validation set to the database.
Reviewing a validation set¶
Annotations in a validation set can be reviewed using the review interface.
At the Validation
menu under Continue a validation set, select the validation dataset and validation set name and click Continue
. Click Review
on the left panel to navigate to the review interface.
The review interface displays the annotated objects organized by class and slide. Thumbnail images of the objects are organized into columns by class. Clicking a thumbnail will bring that object into the field of view in the slide view. The thumbnails can be dragged/dropped to a different column to change the class label, or placed in the Ignore
column to discard them from the set. Changes are instantly commited to the database (no additional button clicks are needed).
Validating a classifier¶
The Validation
interface also enables users to measure classifier accuracy by applying a trained classifier to a validation dataset.
At the Validation
menu under Validate classifier, select the training dataset and classifier name, and the validation dataset and validation set name.
The Validate
button will generate a .csv file containing: 1. The predicted class labels for each object in the validation set (1 - positive, -1 - negative) 2. The prediction scores from the random forest classifier (ranged 0-1) 3. The false-positive rate 4. The true-positive rate 5. The precision and 6. the total accuracy.