Merge pull request #38 from alan-turing-institute/dev

Extend [compat] for MLJBase for a 0.3.3 release

Project.toml

@@ -1,7 +1,7 @@
 name = "MLJTuning"
 uuid = "03970b2e-30c4-11ea-3135-d1576263f10f"
 authors = ["Anthony D. Blaom <anthony.blaom@gmail.com>"]
-version = "0.3.2"
+version = "0.3.3"
 
 [deps]
 ComputationalResources = "ed09eef8-17a6-5b46-8889-db040fac31e3"
@@ -15,7 +15,7 @@ RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
 [compat]
 ComputationalResources = "^0.3"
 Distributions = "^0.22,^0.23"
-MLJBase = "^0.12.2"
+MLJBase = "^0.12.2,^0.13"
 MLJModelInterface = "^0.2"
 RecipesBase = "^0.8"
 julia = "^1"

README.md

@@ -31,7 +31,7 @@ hyperparameter optimization tasks from there.
 MLJTuning is the place for developers to integrate hyperparameter
 optimization algorithms (here called *tuning strategies*) into MLJ,
 either by adding code to [/src/strategies](/src/strategies), or by
-importing MLJTuning into a third-pary package and implementing
+importing MLJTuning into a third-party package and implementing
 MLJTuning's [tuning strategy
 interface](#implementing-a-new-tuning-strategy).
 
@@ -59,34 +59,25 @@ This repository contains:
 - an abstract **[tuning strategy
   interface]((#implementing-a-new-tuning-strategy))** to allow
   developers to conveniently implement common hyperparameter
-  optimization strategies, such as:
-
-  - [x] search models generated by an arbitrary iterator, eg `models = [model1,
-	model2, ...]` (built-in `Explicit` strategy)
-
-  - [x] grid search (built-in `Grid` strategy)
+  optimization strategies, such as those in Table 1 (already
+  implemented) and the following: Latin hypercubes, bandit, simulated
+  annealing, Bayesian optimization using Gaussian processes,
+  structured tree Parzen estimators, multi-objective (Pareto)
+  optimization, genetic algorithms, AD-powered gradient descent
+  methods
+
+- the **implementations** of the tuning strategy interface given
+  below, which come pre-loaded into 
+  [MLJ](https://github.com/alan-turing-institute/MLJ.jl)
 
-  - [ ] Latin hypercubes
-
-  - [x] random search (built-in `RandomSearch` strategy)
+tuning strategy      | type                | providing package
+---------------------|---------------------|-------------------------
+explicit search      | `Explicit`          | MLJTuning.jl
+grid search          | `Grid`              | MLJTuning.jl
+random search        | `RandomSearch`      | MLTuning.jl 
 
-  - [ ] bandit
-
-  - [ ] simulated annealing
-
-  - [ ] Bayesian optimization using Gaussian processes
-
-  - [ ] structured tree Parzen estimators
-
-  - [ ] multi-objective (Pareto) optimization
+> Table 1. Implemented tuning strategies
 
-  - [ ] genetic algorithms
-
-  - [ ] AD-powered gradient descent methods 
-
-- a selection of **implementations** of the tuning strategy interface,
-  currently all those accessible from
-  [MLJ](https://github.com/alan-turing-institute/MLJ.jl) itself.
 
 - the code defining the MLJ functions `learning_curves!` and `learning_curve` as
   these are essentially one-dimensional grid searches
@@ -138,8 +129,8 @@ begin, on the basis of the specific strategy and a user-specified
 - An *evaluation* is the value returned by some call to the
   `evaluate!` method, when passed the resampling strategy (e.g.,
   `CV(nfolds=9)` and performance measures specified by the user when
-  specifying the tuning task (e.g., `cross_entropy`b,
-  `accuracy`). Recall that such a value is a named tuple of vectors
+  specifying the tuning task (e.g., `cross_entropy`,
+  `accuracy`, `mae`). Recall that such a value is a named tuple of vectors
   with keys `measure`, `measurement`, `per_fold`, and
   `per_observation`. See [Evaluating Model
   Performance](https://alan-turing-institute.github.io/MLJ.jl/dev/evaluating_model_performance/)
@@ -370,7 +361,7 @@ algorithm (available to the `models!` method). Be sure to make this
 object mutable if it needs to be updated by the `models!` method. 
 
 The `state` is a place to record the outcomes of any necessary
-intialization of the tuning algorithm (performed by `setup`) and a
+initialization of the tuning algorithm (performed by `setup`) and a
 place for the `models!` method to save and read transient information
 that does not need to be recorded in the history.
 
@@ -379,11 +370,6 @@ is `fit!` the first time, and not on subsequent calls (unless
 `force=true`). (Specifically, `MLJBase.fit(::TunedModel, ...)` calls
 `setup` but `MLJBase.update(::TunedModel, ...)` does not.)
 
-The `setup` function is called once only, when a `TunedModel` machine
-is `fit!` the first time, and not on subsequent calls (unless
-`force=true`). (Specifically, `MLJBase.fit(::TunedModel, ...)` calls
-`setup` but `MLJBase.update(::TunedModel, ...)` does not.)
-
 The `verbosity` is an integer indicating the level of logging: `0`
 means logging should be restricted to warnings, `-1`, means completely
 silent.
@@ -411,7 +397,7 @@ MLJTuning.models!(tuning::MyTuningStrategy, model, history, state, verbosity)
 ```
 
 This is the core method of a new implementation. Given the existing
-`history` and `state`, it must return a vector ("batch") of *new*
+`history` and `state`, it must return a vector ("batch") of new
 model instances to be evaluated. Any number of models can be returned
 (and this includes an empty vector or `nothing`, if models have been
 exhausted) and the evaluations will be performed in parallel (using
@@ -439,7 +425,7 @@ case).
 If a tuning strategy implementation needs to pass additional
 "metadata" along with each model, to be passed to `result` for
 recording in the history, then instead of model instances, `models!`
-should returne a vector of *tuples* of the form `(m, metadata)`, where
+should return a vector of *tuples* of the form `(m, metadata)`, where
 `m` is a model instance, and `metadata` the associated data. See the
 discussion above on `result`.
 
@@ -507,7 +493,7 @@ model
 
 - `tuning_report(::MyTuningStrategy, ...)` is a method the implementer
   may overload. It should return a named tuple with `history` as one
-  of the keys (the format up to the implementation.) The fallback is
+  of the keys (the format being up to the implementation). The fallback is
   to return the raw history:
 
 ```julia
@@ -546,7 +532,7 @@ MLJTuning.DEFAULT_N` to see check the current value.
 The most rudimentary tuning strategy just evaluates every model
 generated by some iterator, such iterators constituting the only kind
 of supported range. The models generated must all have a common type
-and, in th implementation below, the type information is conveyed by
+and, in the implementation below, the type information is conveyed by
 the specified prototype `model` (which is otherwise ignored).  The
 fallback implementations for `result`, `best` and `report_history`
 suffice.

test/strategies/grid.jl

@@ -152,11 +152,12 @@ end
 
     fit!(tuned, verbosity=0)
     r = report(tuned)
-    @test r.best_report isa NamedTuple{(:machines, :reports)}
+    @test r.best_report isa NamedTuple{(:machines, :report_given_machine)}
     fit!(tuned, verbosity=0)
     rep = report(tuned)
     fp = fitted_params(tuned)
-    @test fp.best_fitted_params isa NamedTuple{(:machines, :fitted_params)}
+    @test fp.best_fitted_params isa
+        NamedTuple{(:machines, :fitted_params_given_machine)}
     b = fp.best_model
     @test b isa SimpleDeterministicCompositeModel