Suggestion for Buiding Model

Hello,

I'm using the Bayesian model components (Learn Molecular Categories). There is the validation step with a Add leave-one-out cross-validation option. It removes one compound, build a model and then look at the score.

I was wondering if it would be possible to change this parameter (new options). So the user could decide the cross-validation parameters (10%, 20%...), which could be a simple random selection (implemented new methods could be longer and more difficult).

It is very convenient to have the information in the model, and the validation of multiples models at once would be great to see the quality by categories.

If you have to validate multpiple models to tell your chemists yes this one is good, but not this one, what is your best approach?

Jeremy
Answers 4
DH
DH 2009-07-01
There is a component named Learn Cross-validated Classification Model which does k-fold cross-validation for any binary learner in Pipeline Pilot. If the learner has tunable parameters, the cross-validation component also selects the parameter values (from among those you specify) that give the best cross-validated ROC score.

You can't use this component with a multi-category learner such as Learn Molecular Categories, but you can use it for Learn Good Molecules, building a separate model for each category. (You'll need to adjust the TestForGood in each case.) The validation results will apply to a model built from the same data with Learn Molecular Categories, as this component itself simply builds a separate binary model for each category.

See the "Learn Cross-validated Auto Price Model" protocol for an example of one of the cross-validation components in action. The component in this case is Learn Cross-validated Numerical Model, which is for regression models, but the component for classification models works the same way.
JB
JB 2009-07-02
Hi Dana,

I was wondering if it was possible to integrate the Learn Cross-validated Classification Model inside the Learn Molecular Categories. Or make it compatible with this component.

Jeremy
DH
DH 2009-07-06
Jeremy,

It is certainly possible, but would require a change to the CV component interface and a decision on the score to use for ranking models. Specifically, if the classes are A, B, C, etc., should the ROC score for the multi-class model be based on A-vs-B type scores (one-vs-one) or A-vs-non-A type scores (one-vs-rest)?

For a description of the one-vs-one approach, see D.J. Hand and R.J. Till, "A Simple Generalization of the Area Under the ROC Curve for Multiple Class Classification Problems," Machine Learning, 45, 171 (2001). Other approaches to multi-class ROC scores are discussed in the Additional Topics section of the following paper: T. Fawcett, "ROC Graphs: Notes and Practical Considerations for Data Mining Researchers," Tech Report HPL-2003-4, HP Laboratories (2003).
JB
JB 2009-07-07
Hi Dana,

Thank you for the articles. I didn't think about the one.vs.one possibility, I just thought the one.vs.rest.

Jeremy