Inspired by this thread, I decided to post some prototype components which you can use to create self-validating learner components -- i.e., learners which automatically perform validation when building a model, with the additional option of using the validation results to tune any adjustable learner parameters. You use the Create Validation Learner component to create a validation learner from any "base" learner component, and then use the new component where you would have used the original learner. But in the new component, you can select validation options, and can enter multiple values for model building parameters that take only single values in the original component. The component then iterates over all possible combinations of these parameter values and chooses the set that gives the "best" model (as measured by RMSE or ROC score).

The attached zip file contains the following components:

Create Validation Learner: Use this to create a custom regression or classification learner with built-in validation

Learn Validated Classfiication Model Prototype: Template component used for adding validation to a classification learner

Learn Validated Numerical Model Prototype: Template component used for adding validation to a regression learner

Y Scrambler: Component used by the above components for performing Y scrambling

My Learn Property: A customer learner created by Create Validation Learner from the Learn Property component

PLS Model Validation: A protocol that demonstrates parameter tuning and Y scrambling using My Learn Property

To use these components, unzip the file and import the Validation Learner Prototypes folder into your user tab.The components were tested in Pipeline Pilot 8.5 CU1. They may or may not work in earlier PP releases (but will definitely not work in pre-version 8.0 releases).

Please give this component set a try and let me know what you think. I'm happy to answer any questions on how and why they work the way they do. But I don't want to say any more right now, because I want to see how clear the scheme is from just the component documentation and the above brief description.