`sperrorest`

is a generic framework which aims to work with all R models/packages. In statistical learning, model setups, their formulas and error measures all depend on the family of the response variable. Various families exist (numeric, binary, multiclass) which again include sub-families (e.g. gaussian or poisson distribution of a numeric response).

This detail needs to be specified via the respective function, e.g. when using `glm()`

with a binary response, one needs to set `family = "binomial"`

to make sure that the model does something meaningful. Most of the time, the same applies to the generic `predict()`

function. For the `glm()`

case, one would need to set `type = "response"`

if the predicted values should reflect probabilities instead of log-odds.

These settings can be specified using `model_args`

and `pred_args`

in `sperrorest()`

. So fine, “why do we need to write all these wrappers and custom model/predict functions then?!”

`model_fun`

expects at least formula argument and a data.frame with the learning sample. All arguments, including the additional ones provided via `model_args`

, are getting passed to `model_fun`

via a `do.call()`

call. However, if `model_fun`

does not have an argument named `formula`

but e.g. `fixed`

(like it is the case for `glmmPQL()`

) the `do.call()`

call will fail because `sperrorest()`

tries to pass an argument named `formula`

but `glmmPQL`

expects an argument named `fixed`

.

In this case, we need to write a wrapper function for `glmmPQL`

(named `glmmPQL_modelfun`

here) which accounts for this naming problem. Here, we are passing the `formula`

argument to our custom model function which then does the actual call to `glmmPQL()`

using the supplied `formula`

object as the `fixed`

argument of `glmmPQL`

.

By default, `glmmPQL()`

has further arguments like `family`

or `random`

. If we want to use these, we pass them to `model_args`

which then appends these to the arguments of `glmmPQL_modelfun`

.

```
glmmPQL_modelfun <- function(formula = NULL, data = NULL, random = NULL,
family = NULL) {
fit <- glmmPQL(fixed = formula, data = data, random = random, family = family)
return(fit)
}
```

Unless specified explicitly, `sperrorest()`

tries to use the generic `predict()`

function. This function works differently depending on the class of the provided fitted model, i.e. many models slightly differ in the naming (and availability) of their arguments. For example, when fitting a Support Vector Machine (SVM) with a binary response variable, package `kernlab`

expects an argument `type = "probabilities"`

in its `predict()`

call to receive predicted probabilities while in package `e1071`

it is `"probability = TRUE"`

. Similar to `model_args`

, this can be accounted for in the `pred_args`

of `sperrorest()`

.

However, `sperrorest()`

expects that the predicted values (of any response type) are stored directly in the returned object of the `predict()`

function. While this is the case for many models, mainly with a numeric response, classification cases often behave differently. Here, the predicted values (classes in this case) are often stored in a sub-object named `class`

or `predicted`

.

Since there is no way to account for this in a general way (when every package may return the predicted values in a different format/column), we need to account for it by providing a custom predict function which returns only the predicted values so that `sperrorest()`

can continue properly. This time we are showing two examples. The first takes again a binary classification using `randomForest`

.

When calling predict on a fitted `randomForest`

model with a binary response variable, the predicted values are actually stored in the resulting object returned by `predict()`

(here called `pred`

). So why do we have trouble here then?

Simply because `pred`

is a matrix containing both probabilities for the `FALSE`

(= 0) and `TRUE`

(= 1) case. `sperrorest()`

needs a vector containing only the predicted values of the `TRUE`

case to pass these further onto `err_fun()`

which then takes care of calculating all the error measures. So the important part is to subset the resulting matrix in the `pred`

object to `TRUE`

cases only and return the result.

```
rf_predfun <- function(object = NULL, newdata = NULL, type = NULL) {
pred <- predict(object = object, newdata = newdata, type = type)
pred <- pred[, 2]
}
```

The same case (binary response) using `svm`

from the `e1071`

package. Here, the predicted probabilities are stored in a sub-object of `pred`

. We can address it using the `attr()`

function. Then again, we only need the `TRUE`

cases for `sperrorest()`

.

```
svm_predfun <- function(object = NULL, newdata = NULL, probability = NULL) {
pred <- predict(object, newdata = newdata, probability = TRUE)
pred <- attr(pred, "probabilities")[, 2]
}
```