Adaptive learner hyperparameter in RFE [Question]

[bblodfon]

Hi Marc (@be-marc),

This is more of a is-it-possible-somehow question as I know it can't be done via conventional means - but maybe you can think of a way to help me hack this if you like :) In the example I provided in #67 with a random forest learner and rfe fselector, imagine that I had included the parameter mtry.ratio in the learner:

learner = lrn('classif.ranger', num.threads = 10, num.trees = 50, importance = 'permutation', mtry.ratio = 0.01)

Now, the RFE procedure will choose a series of feature subset sizes to check. In my case scenario, I have large task with ~10000 features as a starting point. Using n_features = 2 and feature_fraction = 0.8 I get the following series of subset sizes:

10000  8000  6400  5120  4096  3276  2621  2097  1677  1342  1073   858   687   549   439   351   281   225   180   144   115    92    73    59    47    37    30    24    19    15    12     9     7     6     5     4     3     2

Could I have somehow a simple function that adapts the mtry.ratio in each repetition so that the smaller a feature subset gets, the larger the mtry.ratio becomes? The case being that I would want a small mtry.ratio when the subset size is 10000 (e.g. mtry.ratio = 0.01 => check 100 features per split), but when reaching a subset with a size of 100 or less, then this becomes like an extremely randomized forest, checking 1 feature per split only (so I want to kinda balance this).

[be-marc]

Hey, we recently introduced callbacks to do these hacky things. Something like this should work.

library(mlr3verse)
library(mlr3fselect)

callback = callback_fselect("mtry",
  on_eval_after_design = function(callback, context) {
    x = length(context$design$task[[1]]$feature_names)
    mtry.ratio = 1- ((x - 1) / (60 - 1))
    context$design$learner[[1]]$param_set$set_values(mtry.ratio = mtry.ratio)
  })

instance = FSelectInstanceSingleCrit$new(
  task = tsk("spam"),
  learner = lrn('classif.ranger', num.threads = 10, num.trees = 50, importance = 'permutation', mtry.ratio = 0.1),
  resampling = rsmp("insample"),
  measure = msr("oob_error"),
  terminator = trm("none"),
  callbacks = callback,
  store_models = TRUE
)

fselector = fs("rfe", feature_fraction = 0.8, n_features = 2)

fselector$optimize(instance)

# mtry.ratio increases
mlr3misc::map(as.data.table(instance$archive)$resample_result, function(rr) rr$learner$param_set$values$mtry.ratio)

[bblodfon]

Nice! will try this tomorrow, I guess the callbacks argument can also be used on the AutoFSelector too, right? (didn't see it in the documentation but maybe because it's only a recent thing)

[be-marc]

Download the latest version 0.9.1 from CRAN. The callbacks argument is available in this version.

[bblodfon]

Thanks so much Marc, it worked wonderfully! Also nice work with the callbacks, very useful!