Package 'familiar'

Variable importance (vimpTable) object, a list thereof, or one or more paths to these objects.

Method used to aggregate variable importance. The available methods are described in the feature selection methods vignette.

Rank threshold used within several aggregation methods. See the feature selection methods vignette for more details.

unused parameters.

A data.frame or data.table, a path to such tables on a local or network drive, or a path to tabular data that may be converted to these formats.

Unused arguments.

A familiarEnsemble or familiarModel object that is used to check consistency of these objects.

(recommended) Name of the column containing sample or subject identifiers. See batch_id_column above for more details.

If unset, every row will be identified as a single sample.

(recommended) Name of the column containing batch or cohort identifiers. This parameter is required if more than one dataset is provided, or if external validation is performed.

In familiar any row of data is organised by four identifiers:

• The batch identifier batch_id_column: This denotes the group to which a set of samples belongs, e.g. patients from a single study, samples measured in a batch, etc. The batch identifier is used for batch normalisation, as well as selection of development and validation datasets.

• The sample identifier sample_id_column: This denotes the sample level, e.g. data from a single individual. Subsets of data, e.g. bootstraps or cross-validation folds, are created at this level.

• The series identifier series_id_column: Indicates measurements on a single sample that may not share the same outcome value, e.g. a time series, or the number of cells in a view.

• The repetition identifier: Indicates repeated measurements in a single series where any feature values may differ, but the outcome does not. Repetition identifiers are always implicitly set when multiple entries for the same series of the same sample in the same batch that share the same outcome are encountered.

(optional) Name of the column containing series identifiers, which distinguish between measurements that are part of a series for a single sample. See batch_id_column above for more details.

If unset, rows which share the same batch and sample identifiers but have a different outcome are assigned unique series identifiers.

(optional) One or more batch or cohort identifiers to constitute data sets for development. Defaults to all, or all minus the identifiers in validation_batch_id for external validation. Required if external validation is performed and validation_batch_id is not provided.

(optional) One or more batch or cohort identifiers to constitute data sets for external validation. Defaults to all data sets except those in development_batch_id for external validation, or none if not. Required if development_batch_id is not provided.

(optional) Name of the modelled outcome. This name will be used in figures created by familiar.

If not set, the column name in outcome_column will be used for binomial, multinomial, count and continuous outcomes. For other outcomes (survival and competing_risk) no default is used.

(recommended) Name of the column containing the outcome of interest. May be identified from a formula, if a formula is provided as an argument. Otherwise an error is raised. Note that survival and competing_risk outcome type outcomes require two columns that indicate the time-to-event or the time of last follow-up and the event status.

(recommended) Type of outcome found in the outcome column. The outcome type determines many aspects of the overall process, e.g. the available feature selection methods and learners, but also the type of assessments that can be conducted to evaluate the resulting models. Implemented outcome types are:

• binomial: categorical outcome with 2 levels.

• multinomial: categorical outcome with 2 or more levels.

• count: Poisson-distributed numeric outcomes.

• continuous: general continuous numeric outcomes.

• survival: survival outcome for time-to-event data.

If not provided, the algorithm will attempt to obtain outcome_type from contents of the outcome column. This may lead to unexpected results, and we therefore advise to provide this information manually.

Note that competing_risk survival analysis are not fully supported, and is currently not a valid choice for outcome_type.

(recommended) Indicator for events in survival and competing_risk analyses. familiar will automatically recognise 1, true, t, y and yes as event indicators, including different capitalisations. If this parameter is set, it replaces the default values.

(recommended) Indicator for right-censoring in survival and competing_risk analyses. familiar will automatically recognise 0, false, f, n, no as censoring indicators, including different capitalisations. If this parameter is set, it replaces the default values.

(recommended) Indicator for competing risks in competing_risk analyses. There are no default values, and if unset, all values other than those specified by the event_indicator and censoring_indicator parameters are considered to indicate competing risks.

(optional) Class levels for binomial or multinomial outcomes. This argument can be used to specify the ordering of levels for categorical outcomes. These class levels must exactly match the levels present in the outcome column.

(optional) Feature columns that will be removed from the data set. Cannot overlap with features in signature, novelty_features or include_features.

(optional) Feature columns that are specifically included in the data set. By default all features are included. Cannot overlap with exclude_features, but may overlap signature. Features in signature and novelty_features are always included. If both exclude_features and include_features are provided, include_features takes precedence, provided that there is no overlap between the two.

(optional) Method used to set reference levels for categorical features. There are several options:

• auto (default): Categorical features that are not explicitly set by the user, i.e. columns containing boolean values or characters, use the most frequent level as reference. Categorical features that are explicitly set, i.e. as factors, are used as is.

• always: Both automatically detected and user-specified categorical features have the reference level set to the most frequent level. Ordinal features are not altered, but are used as is.

• never: User-specified categorical features are used as is. Automatically detected categorical features are simply sorted, and the first level is then used as the reference level. This was the behaviour prior to familiar version 1.3.0.

Specifies stringency of various checks. This is mostly:

• strict: default value used for summon_familiar. Thoroughly checks input data. Used internally for checking development data.

• external_warn: value used for extract_data and related methods. Less stringent checks, but will warn for possible issues. Used internally for checking data for evaluation and explanation.

• external: value used for external methods such as predict. Less stringent checks, particularly for identifier and outcome columns, which may be completely absent. Used internally for predict.

familiarCollection object, or one or more familiarData objects, that will be internally converted to a familiarCollection object. It is also possible to provide a familiarEnsemble or one or more familiarModel objects together with the data from which data is computed prior to export. Paths to such files can also be provided.

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

Name of the collection.

Arguments passed on to extract_data

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

time_max

Time point which is used as the benchmark for e.g. cumulative risks generated by random forest, or the cut-off value for Uno's concordance index. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

aggregation_method

Method for aggregating variable importances for the purpose of evaluation. Variable importances are determined during feature selection steps and after training the model. Both types are evaluated, but feature selection variable importance is only evaluated at run-time.

See the documentation for the vimp_aggregation_method argument in summon_familiar for information concerning the different available methods.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

rank_threshold

The threshold used to define the subset of highly important features during evaluation.

See the documentation for the vimp_aggregation_rank_threshold argument in summon_familiar for more information.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

metric

One or more metrics for assessing model performance. See the vignette on performance metrics for the available metrics. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_method

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_cut_method

The method used to divide features into separate clusters. The available methods are the same as for the cluster_cut_method configuration parameter: silhouette, fixed_cut and dynamic_cut.

silhouette is available for all cluster methods, but fixed_cut only applies to methods that create hierarchical trees (hclust, agnes and diana). dynamic_cut requires the dynamicTreeCut package and can only be used with agnes and hclust.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_threshold

The threshold level for pair-wise similarity that is required to form feature clusters with the fixed_cut method.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_metric

Metric to determine pairwise similarity between features. Similarity is computed in the same manner as for clustering, and feature_similarity_metric therefore has the same options as cluster_similarity_metric: mcfadden_r2, cox_snell_r2, nagelkerke_r2, spearman, kendall and pearson.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_cluster_method

The method used to perform clustering based on distance between samples. These are the same methods as for the cluster_method configuration parameter: hclust, agnes, diana and pam.

none cannot be used when extracting data for feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_similarity_metric

Metric to determine pairwise similarity between samples. Similarity is computed in the same manner as for clustering, but sample_similarity_metric has different options that are better suited to computing distance between samples instead of between features: gower, euclidean.

The underlying feature data is scaled to the $[0, 1]$ range (for numerical features) using the feature values across the samples. The normalisation parameters required can optionally be computed from feature data with the outer 5% (on both sides) of feature values trimmed or winsorised. To do so append ⁠_trim⁠ (trimming) or ⁠_winsor⁠ (winsorising) to the metric name. This reduces the effect of outliers somewhat.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

icc_type

String indicating the type of intraclass correlation coefficient (1, 2 or 3) that should be used to compute robustness for features in repeated measurements during the evaluation of univariate importance. These types correspond to the types in Shrout and Fleiss (1979). If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

data_element

String indicating which data elements are to be extracted. Default is all, but specific elements can be specified to speed up computations if not all elements are to be computed. This is an internal parameter that is set by, e.g. the export_model_vimp method.

sample_limit

(optional) Set the upper limit of the number of samples that are used during evaluation steps. Cannot be less than 20.

This setting can be specified per data element by providing a parameter value in a named list with data elements, e.g. list("sample_similarity"=100, "permutation_vimp"=1000).

This parameter can be set for the following data elements: sample_similarity and ice_data.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

aggregate_results

(optional) Flag that signifies whether results should be aggregated during evaluation. If estimation_type is bias_correction or bc, aggregation leads to a single bias-corrected estimate. If estimation_type is bootstrap_confidence_interval or bci, aggregation leads to a single bias-corrected estimate with lower and upper boundaries of the confidence interval. This has no effect if estimation_type is point.

The default value is equal to TRUE except when assessing metrics to assess model performance, as the default violin plot requires underlying data.

As with detail_level and estimation_type, a non-default aggregate_results parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"=TRUE, , "model_performance"=FALSE). This parameter exists for the same elements as estimation_type.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

stratification_method

(optional) Method for determining the stratification threshold for creating survival groups. The actual, model-dependent, threshold value is obtained from the development data, and can afterwards be used to perform stratification on validation data.

The following stratification methods are available:

• median (default): The median predicted value in the development cohort is used to stratify the samples into two risk groups. For predicted outcome values that build a continuous spectrum, the two risk groups in the development cohort will be roughly equal in size.

• mean: The mean predicted value in the development cohort is used to stratify the samples into two risk groups.

• mean_trim: As mean, but based on the set of predicted values where the 5% lowest and 5% highest values are discarded. This reduces the effect of outliers.

• mean_winsor: As mean, but based on the set of predicted values where the 5% lowest and 5% highest values are winsorised. This reduces the effect of outliers.

• fixed: Samples are stratified based on the sample quantiles of the predicted values. These quantiles are defined using the stratification_threshold parameter.

• optimised: Use maximally selected rank statistics to determine the optimal threshold (Lausen and Schumacher, 1992; Hothorn et al., 2003) to stratify samples into two optimally separated risk groups.

One or more stratification methods can be selected simultaneously.

This parameter is only relevant for survival outcomes.

dynamic_model_loading

(optional) Enables dynamic loading of models during the evaluation process, if TRUE. Defaults to FALSE. Dynamic loading of models may reduce the overall memory footprint, at the cost of increased disk or network IO. Models can only be dynamically loaded if they are found at an accessible disk or network location. Setting this parameter to TRUE may help if parallel processing causes out-of-memory issues during evaluation.

A familiarData object, or a familiarEnsemble or familiarModel objects that will be internally converted to a familiarData object. Paths to such objects can also be provided.

Arguments passed on to extract_data

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

time_max

Time point which is used as the benchmark for e.g. cumulative risks generated by random forest, or the cut-off value for Uno's concordance index. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

aggregation_method

Method for aggregating variable importances for the purpose of evaluation. Variable importances are determined during feature selection steps and after training the model. Both types are evaluated, but feature selection variable importance is only evaluated at run-time.

See the documentation for the vimp_aggregation_method argument in summon_familiar for information concerning the different available methods.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

rank_threshold

The threshold used to define the subset of highly important features during evaluation.

See the documentation for the vimp_aggregation_rank_threshold argument in summon_familiar for more information.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

metric

One or more metrics for assessing model performance. See the vignette on performance metrics for the available metrics. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_method

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_cut_method

The method used to divide features into separate clusters. The available methods are the same as for the cluster_cut_method configuration parameter: silhouette, fixed_cut and dynamic_cut.

silhouette is available for all cluster methods, but fixed_cut only applies to methods that create hierarchical trees (hclust, agnes and diana). dynamic_cut requires the dynamicTreeCut package and can only be used with agnes and hclust.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_threshold

The threshold level for pair-wise similarity that is required to form feature clusters with the fixed_cut method.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_metric

Metric to determine pairwise similarity between features. Similarity is computed in the same manner as for clustering, and feature_similarity_metric therefore has the same options as cluster_similarity_metric: mcfadden_r2, cox_snell_r2, nagelkerke_r2, spearman, kendall and pearson.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_cluster_method

The method used to perform clustering based on distance between samples. These are the same methods as for the cluster_method configuration parameter: hclust, agnes, diana and pam.

none cannot be used when extracting data for feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_similarity_metric

Metric to determine pairwise similarity between samples. Similarity is computed in the same manner as for clustering, but sample_similarity_metric has different options that are better suited to computing distance between samples instead of between features: gower, euclidean.

The underlying feature data is scaled to the $[0, 1]$ range (for numerical features) using the feature values across the samples. The normalisation parameters required can optionally be computed from feature data with the outer 5% (on both sides) of feature values trimmed or winsorised. To do so append ⁠_trim⁠ (trimming) or ⁠_winsor⁠ (winsorising) to the metric name. This reduces the effect of outliers somewhat.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

icc_type

String indicating the type of intraclass correlation coefficient (1, 2 or 3) that should be used to compute robustness for features in repeated measurements during the evaluation of univariate importance. These types correspond to the types in Shrout and Fleiss (1979). If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

data_element

String indicating which data elements are to be extracted. Default is all, but specific elements can be specified to speed up computations if not all elements are to be computed. This is an internal parameter that is set by, e.g. the export_model_vimp method.

sample_limit

(optional) Set the upper limit of the number of samples that are used during evaluation steps. Cannot be less than 20.

This setting can be specified per data element by providing a parameter value in a named list with data elements, e.g. list("sample_similarity"=100, "permutation_vimp"=1000).

This parameter can be set for the following data elements: sample_similarity and ice_data.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

aggregate_results

(optional) Flag that signifies whether results should be aggregated during evaluation. If estimation_type is bias_correction or bc, aggregation leads to a single bias-corrected estimate. If estimation_type is bootstrap_confidence_interval or bci, aggregation leads to a single bias-corrected estimate with lower and upper boundaries of the confidence interval. This has no effect if estimation_type is point.

The default value is equal to TRUE except when assessing metrics to assess model performance, as the default violin plot requires underlying data.

As with detail_level and estimation_type, a non-default aggregate_results parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"=TRUE, , "model_performance"=FALSE). This parameter exists for the same elements as estimation_type.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

stratification_method

(optional) Method for determining the stratification threshold for creating survival groups. The actual, model-dependent, threshold value is obtained from the development data, and can afterwards be used to perform stratification on validation data.

The following stratification methods are available:

• median (default): The median predicted value in the development cohort is used to stratify the samples into two risk groups. For predicted outcome values that build a continuous spectrum, the two risk groups in the development cohort will be roughly equal in size.

• mean: The mean predicted value in the development cohort is used to stratify the samples into two risk groups.

• mean_trim: As mean, but based on the set of predicted values where the 5% lowest and 5% highest values are discarded. This reduces the effect of outliers.

• mean_winsor: As mean, but based on the set of predicted values where the 5% lowest and 5% highest values are winsorised. This reduces the effect of outliers.

• fixed: Samples are stratified based on the sample quantiles of the predicted values. These quantiles are defined using the stratification_threshold parameter.

• optimised: Use maximally selected rank statistics to determine the optimal threshold (Lausen and Schumacher, 1992; Hothorn et al., 2003) to stratify samples into two optimally separated risk groups.

One or more stratification methods can be selected simultaneously.

This parameter is only relevant for survival outcomes.

dynamic_model_loading

(optional) Enables dynamic loading of models during the evaluation process, if TRUE. Defaults to FALSE. Dynamic loading of models may reduce the overall memory footprint, at the cost of increased disk or network IO. Models can only be dynamically loaded if they are found at an accessible disk or network location. Setting this parameter to TRUE may help if parallel processing causes out-of-memory issues during evaluation.

Name of the familiarData object. If not set, a name is automatically generated.

A familiarEnsemble object, or one or more familiarModel objects that will be internally converted to a familiarEnsemble object. Paths to such objects can also be provided.

Unused arguments.

a familiarModel object

additional arguments passed to coef methods for the underlying model, when available.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

Arguments passed on to extract_data, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

time_max

Time point which is used as the benchmark for e.g. cumulative risks generated by random forest, or the cut-off value for Uno's concordance index. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

aggregation_method

Method for aggregating variable importances for the purpose of evaluation. Variable importances are determined during feature selection steps and after training the model. Both types are evaluated, but feature selection variable importance is only evaluated at run-time.

See the documentation for the vimp_aggregation_method argument in summon_familiar for information concerning the different available methods.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

rank_threshold

The threshold used to define the subset of highly important features during evaluation.

See the documentation for the vimp_aggregation_rank_threshold argument in summon_familiar for more information.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

metric

One or more metrics for assessing model performance. See the vignette on performance metrics for the available metrics. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_method

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_cut_method

The method used to divide features into separate clusters. The available methods are the same as for the cluster_cut_method configuration parameter: silhouette, fixed_cut and dynamic_cut.

silhouette is available for all cluster methods, but fixed_cut only applies to methods that create hierarchical trees (hclust, agnes and diana). dynamic_cut requires the dynamicTreeCut package and can only be used with agnes and hclust.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_threshold

The threshold level for pair-wise similarity that is required to form feature clusters with the fixed_cut method.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_metric

Metric to determine pairwise similarity between features. Similarity is computed in the same manner as for clustering, and feature_similarity_metric therefore has the same options as cluster_similarity_metric: mcfadden_r2, cox_snell_r2, nagelkerke_r2, spearman, kendall and pearson.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_cluster_method

The method used to perform clustering based on distance between samples. These are the same methods as for the cluster_method configuration parameter: hclust, agnes, diana and pam.

none cannot be used when extracting data for feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_similarity_metric

Metric to determine pairwise similarity between samples. Similarity is computed in the same manner as for clustering, but sample_similarity_metric has different options that are better suited to computing distance between samples instead of between features: gower, euclidean.

The underlying feature data is scaled to the $[0, 1]$ range (for numerical features) using the feature values across the samples. The normalisation parameters required can optionally be computed from feature data with the outer 5% (on both sides) of feature values trimmed or winsorised. To do so append ⁠_trim⁠ (trimming) or ⁠_winsor⁠ (winsorising) to the metric name. This reduces the effect of outliers somewhat.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

icc_type

String indicating the type of intraclass correlation coefficient (1, 2 or 3) that should be used to compute robustness for features in repeated measurements during the evaluation of univariate importance. These types correspond to the types in Shrout and Fleiss (1979). If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

data_element

String indicating which data elements are to be extracted. Default is all, but specific elements can be specified to speed up computations if not all elements are to be computed. This is an internal parameter that is set by, e.g. the export_model_vimp method.

sample_limit

(optional) Set the upper limit of the number of samples that are used during evaluation steps. Cannot be less than 20.

This setting can be specified per data element by providing a parameter value in a named list with data elements, e.g. list("sample_similarity"=100, "permutation_vimp"=1000).

This parameter can be set for the following data elements: sample_similarity and ice_data.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

stratification_method

(optional) Method for determining the stratification threshold for creating survival groups. The actual, model-dependent, threshold value is obtained from the development data, and can afterwards be used to perform stratification on validation data.

The following stratification methods are available:

• median (default): The median predicted value in the development cohort is used to stratify the samples into two risk groups. For predicted outcome values that build a continuous spectrum, the two risk groups in the development cohort will be roughly equal in size.

• mean: The mean predicted value in the development cohort is used to stratify the samples into two risk groups.

• mean_trim: As mean, but based on the set of predicted values where the 5% lowest and 5% highest values are discarded. This reduces the effect of outliers.

• mean_winsor: As mean, but based on the set of predicted values where the 5% lowest and 5% highest values are winsorised. This reduces the effect of outliers.

• fixed: Samples are stratified based on the sample quantiles of the predicted values. These quantiles are defined using the stratification_threshold parameter.

• optimised: Use maximally selected rank statistics to determine the optimal threshold (Lausen and Schumacher, 1992; Hothorn et al., 2003) to stratify samples into two optimally separated risk groups.

One or more stratification methods can be selected simultaneously.

This parameter is only relevant for survival outcomes.

dynamic_model_loading

(optional) Enables dynamic loading of models during the evaluation process, if TRUE. Defaults to FALSE. Dynamic loading of models may reduce the overall memory footprint, at the cost of increased disk or network IO. Models can only be dynamically loaded if they are found at an accessible disk or network location. Setting this parameter to TRUE may help if parallel processing causes out-of-memory issues during evaluation.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_auc_data, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_calibration_data, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_confusion_matrix, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

One or more time points that are used to create the outcome columns in expression plots. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarData objects. Only used for survival outcomes.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_feature_expression, as_familiar_collection

feature_similarity

Table containing pairwise distance between sample. This is used to determine cluster information, and indicate which samples are similar. The table is created by the extract_sample_similarity method.

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

feature_cluster_method

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_metric

Metric to determine pairwise similarity between features. Similarity is computed in the same manner as for clustering, and feature_similarity_metric therefore has the same options as cluster_similarity_metric: mcfadden_r2, cox_snell_r2, nagelkerke_r2, spearman, kendall and pearson.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_cluster_method

The method used to perform clustering based on distance between samples. These are the same methods as for the cluster_method configuration parameter: hclust, agnes, diana and pam.

none cannot be used when extracting data for feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

sample_similarity_metric

Metric to determine pairwise similarity between samples. Similarity is computed in the same manner as for clustering, but sample_similarity_metric has different options that are better suited to computing distance between samples instead of between features: gower, euclidean.

The underlying feature data is scaled to the $[0, 1]$ range (for numerical features) using the feature values across the samples. The normalisation parameters required can optionally be computed from feature data with the outer 5% (on both sides) of feature values trimmed or winsorised. To do so append ⁠_trim⁠ (trimming) or ⁠_winsor⁠ (winsorising) to the metric name. This reduces the effect of outliers somewhat.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

The method used to divide features into separate clusters. The available methods are the same as for the cluster_cut_method configuration parameter: silhouette, fixed_cut and dynamic_cut.

silhouette is available for all cluster methods, but fixed_cut only applies to methods that create hierarchical trees (hclust, agnes and diana). dynamic_cut requires the dynamicTreeCut package and can only be used with agnes and hclust.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

The threshold level for pair-wise similarity that is required to form feature clusters with the fixed_cut method.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

Add dendrogram in the data element objects.

Add feature label ordering to data in the data element objects.

Add clustering information to data.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) The method used to aggregate variable importances over different data subsets, e.g. bootstraps. The following methods can be selected:

• mean (default): Use the mean rank of a feature over the subsets to determine the aggregated feature rank.

• median: Use the median rank of a feature over the subsets to determine the aggregated feature rank.

• best: Use the best rank the feature obtained in any subset to determine the aggregated feature rank.

• worst: Use the worst rank the feature obtained in any subset to determine the aggregated feature rank.

• stability: Use the frequency of the feature being in the subset of highly ranked features as measure for the aggregated feature rank (Meinshausen and Buehlmann, 2010).

• exponential: Use a rank-weighted frequence of occurrence in the subset of highly ranked features as measure for the aggregated feature rank (Haury et al., 2011).

• borda: Use the borda count as measure for the aggregated feature rank (Wald et al., 2012).

• enhanced_borda: Use an occurrence frequency-weighted borda count as measure for the aggregated feature rank (Wald et al., 2012).

• truncated_borda: Use borda count computed only on features within the subset of highly ranked features.

• enhanced_truncated_borda: Apply both the enhanced borda method and the truncated borda method and use the resulting borda count as the aggregated feature rank.

(optional) The threshold used to define the subset of highly important features. If not set, this threshold is determined by maximising the variance in the occurrence value over all features over the subset size.

This parameter is only relevant for stability, exponential, enhanced_borda, truncated_borda and enhanced_truncated_borda methods.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_ice, as_familiar_collection

features

Names of the feature or features (2) assessed simultaneously. By default NULL, which means that all features are assessed one-by-one.

feature_x_range

When one or two features are defined using features, feature_x_range can be used to set the range of values for the first feature. For numeric features, a vector of two values is assumed to indicate a range from which n_sample_points are uniformly sampled. A vector of more than two values is interpreted as is, i.e. these represent the values to be sampled. For categorical features, values should represent a (sub)set of available levels.

feature_y_range

As feature_x_range, but for the second feature in case two features are defined.

n_sample_points

Number of points used to sample continuous features.

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

sample_limit

(optional) Set the upper limit of the number of samples that are used during evaluation steps. Cannot be less than 20.

This setting can be specified per data element by providing a parameter value in a named list with data elements, e.g. list("sample_similarity"=100, "permutation_vimp"=1000).

This parameter can be set for the following data elements: sample_similarity and ice_data.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_performance, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

metric

One or more metrics for assessing model performance. See the vignette on performance metrics for the available metrics. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) The method used to aggregate variable importances over different data subsets, e.g. bootstraps. The following methods can be selected:

• mean (default): Use the mean rank of a feature over the subsets to determine the aggregated feature rank.

• median: Use the median rank of a feature over the subsets to determine the aggregated feature rank.

• best: Use the best rank the feature obtained in any subset to determine the aggregated feature rank.

• worst: Use the worst rank the feature obtained in any subset to determine the aggregated feature rank.

• stability: Use the frequency of the feature being in the subset of highly ranked features as measure for the aggregated feature rank (Meinshausen and Buehlmann, 2010).

• exponential: Use a rank-weighted frequence of occurrence in the subset of highly ranked features as measure for the aggregated feature rank (Haury et al., 2011).

• borda: Use the borda count as measure for the aggregated feature rank (Wald et al., 2012).

• enhanced_borda: Use an occurrence frequency-weighted borda count as measure for the aggregated feature rank (Wald et al., 2012).

• truncated_borda: Use borda count computed only on features within the subset of highly ranked features.

• enhanced_truncated_borda: Apply both the enhanced borda method and the truncated borda method and use the resulting borda count as the aggregated feature rank.

(optional) The threshold used to define the subset of highly important features. If not set, this threshold is determined by maximising the variance in the occurrence value over all features over the subset size.

This parameter is only relevant for stability, exponential, enhanced_borda, truncated_borda and enhanced_truncated_borda methods.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_ice, as_familiar_collection

features

Names of the feature or features (2) assessed simultaneously. By default NULL, which means that all features are assessed one-by-one.

feature_x_range

When one or two features are defined using features, feature_x_range can be used to set the range of values for the first feature. For numeric features, a vector of two values is assumed to indicate a range from which n_sample_points are uniformly sampled. A vector of more than two values is interpreted as is, i.e. these represent the values to be sampled. For categorical features, values should represent a (sub)set of available levels.

feature_y_range

As feature_x_range, but for the second feature in case two features are defined.

n_sample_points

Number of points used to sample continuous features.

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

sample_limit

(optional) Set the upper limit of the number of samples that are used during evaluation steps. Cannot be less than 20.

This setting can be specified per data element by providing a parameter value in a named list with data elements, e.g. list("sample_similarity"=100, "permutation_vimp"=1000).

This parameter can be set for the following data elements: sample_similarity and ice_data.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_permutation_vimp, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

metric

One or more metrics for assessing model performance. See the vignette on performance metrics for the available metrics. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_method

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_cut_method

The method used to divide features into separate clusters. The available methods are the same as for the cluster_cut_method configuration parameter: silhouette, fixed_cut and dynamic_cut.

silhouette is available for all cluster methods, but fixed_cut only applies to methods that create hierarchical trees (hclust, agnes and diana). dynamic_cut requires the dynamicTreeCut package and can only be used with agnes and hclust.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_threshold

The threshold level for pair-wise similarity that is required to form feature clusters with the fixed_cut method.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_metric

Metric to determine pairwise similarity between features. Similarity is computed in the same manner as for clustering, and feature_similarity_metric therefore has the same options as cluster_similarity_metric: mcfadden_r2, cox_snell_r2, nagelkerke_r2, spearman, kendall and pearson.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

bootstrap_ci_method

(optional) Method used to determine bootstrap confidence intervals (Efron and Hastie, 2016). The following methods are implemented:

• percentile (default): Confidence intervals obtained using the percentile method.

• bc: Bias-corrected confidence intervals.

Note that the standard method is not implemented because this method is often not suitable due to non-normal distributions. The bias-corrected and accelerated (BCa) method is not implemented yet.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_predictions, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

evaluation_times

One or more time points that are used for in analysis of survival problems when data has to be assessed at a set time, e.g. calibration. If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects. Only used for survival outcomes.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

estimation_type

(optional) Sets the type of estimation that should be possible. This has the following options:

• point: Point estimates.

• bias_correction or bc: Bias-corrected estimates. A bias-corrected estimate is computed from (at least) 20 point estimates, and familiar may bootstrap the data to create them.

• bootstrap_confidence_interval or bci (default): Bias-corrected estimates with bootstrap confidence intervals (Efron and Hastie, 2016). The number of point estimates required depends on the confidence_level parameter, and familiar may bootstrap the data to create them.

As with detail_level, a non-default estimation_type parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="bci", "model_performance"="point"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, and prediction_data.

aggregate_results

(optional) Flag that signifies whether results should be aggregated during evaluation. If estimation_type is bias_correction or bc, aggregation leads to a single bias-corrected estimate. If estimation_type is bootstrap_confidence_interval or bci, aggregation leads to a single bias-corrected estimate with lower and upper boundaries of the confidence interval. This has no effect if estimation_type is point.

The default value is equal to TRUE except when assessing metrics to assess model performance, as the default violin plot requires underlying data.

As with detail_level and estimation_type, a non-default aggregate_results parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"=TRUE, , "model_performance"=FALSE). This parameter exists for the same elements as estimation_type.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that determines whether the raw data or strata are exported.

Time range for which strata should be created. If NULL, the full time range is used.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_risk_stratification_data, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

is_pre_processed

Flag that indicates whether the data was already pre-processed externally, e.g. normalised and clustered. Only used if the data argument is a data.table or data.frame.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

ensemble_method

Method for ensembling predictions from models for the same sample. Available methods are:

• median (default): Use the median of the predicted values as the ensemble value for a sample.

• mean: Use the mean of the predicted values as the ensemble value for a sample.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

detail_level

(optional) Sets the level at which results are computed and aggregated.

• ensemble: Results are computed at the ensemble level, i.e. over all models in the ensemble. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the model performance of the ensemble model for each bootstrap.

• hybrid (default): Results are computed at the level of models in an ensemble. This means that, for example, bias-corrected estimates of model performance are directly computed using the models in the ensemble. If there are at least 20 trained models in the ensemble, performance is computed for each model, in contrast to ensemble where performance is computed for the ensemble of models. If there are less than 20 trained models in the ensemble, bootstraps are created so that at least 20 point estimates can be made.

• model: Results are computed at the model level. This means that, for example, bias-corrected estimates of model performance are assessed by creating (at least) 20 bootstraps and computing the performance of the model for each bootstrap.

Note that each level of detail has a different interpretation for bootstrap confidence intervals. For ensemble and model these are the confidence intervals for the ensemble and an individual model, respectively. That is, the confidence interval describes the range where an estimate produced by a respective ensemble or model trained on a repeat of the experiment may be found with the probability of the confidence level. For hybrid, it represents the range where any single model trained on a repeat of the experiment may be found with the probability of the confidence level. By definition, confidence intervals obtained using hybrid are at least as wide as those for ensemble. hybrid offers the correct interpretation if the goal of the analysis is to assess the result of a single, unspecified, model.

hybrid is generally computationally less expensive then ensemble, which in turn is somewhat less expensive than model.

A non-default detail_level parameter can be specified for separate evaluation steps by providing a parameter value in a named list with data elements, e.g. list("auc_data"="ensemble", "model_performance"="hybrid"). This parameter can be set for the following data elements: auc_data, decision_curve_analyis, model_performance, permutation_vimp, ice_data, prediction_data and confusion_matrix.

confidence_level

(optional) Numeric value for the level at which confidence intervals are determined. In the case bootstraps are used to determine the confidence intervals bootstrap estimation, familiar uses the rule of thumb $n = 20 / ci.level$ to determine the number of required bootstraps.

The default value is 0.95.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

Flag that signifies whether results should be aggregated for export.

(optional) Set the upper limit of the number of samples that are used during evaluation steps. Cannot be less than 20.

This setting can be specified per data element by providing a parameter value in a named list with data elements, e.g. list("sample_similarity"=100, "permutation_vimp"=1000).

This parameter can be set for the following data elements: sample_similarity and ice_data.

The method used to perform clustering based on distance between samples. These are the same methods as for the cluster_method configuration parameter: hclust, agnes, diana and pam.

none cannot be used when extracting data for feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

Add dendrogram in the data element objects.

(optional) Exports the collection if TRUE.

Arguments passed on to as_familiar_collection

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object, or other other objects from which a familiarCollection can be extracted. See details for more information.

Path to folder where extracted data should be saved. NULL will allow export as a structured list of data.tables.

(optional) Indicates type of p-value that is shown. One of holm, hochberg, hommel, bonferroni, BH, BY, fdr, none, p_value or q_value for adjusted p-values, uncorrected p-values and q-values. q-values may not be available.

(optional) Exports the collection if TRUE.

Arguments passed on to extract_univariate_analysis, as_familiar_collection

data

A dataObject object, data.table or data.frame that constitutes the data that are assessed.

cl

Cluster created using the parallel package. This cluster is then used to speed up computation through parallellisation.

feature_cluster_method

The method used to perform clustering. These are the same methods as for the cluster_method configuration parameter: none, hclust, agnes, diana and pam.

none cannot be used when extracting data regarding mutual correlation or feature expressions.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_linkage_method

The method used for agglomerative clustering in hclust and agnes. These are the same methods as for the cluster_linkage_method configuration parameter: average, single, complete, weighted, and ward.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_cluster_cut_method

The method used to divide features into separate clusters. The available methods are the same as for the cluster_cut_method configuration parameter: silhouette, fixed_cut and dynamic_cut.

silhouette is available for all cluster methods, but fixed_cut only applies to methods that create hierarchical trees (hclust, agnes and diana). dynamic_cut requires the dynamicTreeCut package and can only be used with agnes and hclust.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_threshold

The threshold level for pair-wise similarity that is required to form feature clusters with the fixed_cut method.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

feature_similarity_metric

Metric to determine pairwise similarity between features. Similarity is computed in the same manner as for clustering, and feature_similarity_metric therefore has the same options as cluster_similarity_metric: mcfadden_r2, cox_snell_r2, nagelkerke_r2, spearman, kendall and pearson.

If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

icc_type

String indicating the type of intraclass correlation coefficient (1, 2 or 3) that should be used to compute robustness for features in repeated measurements during the evaluation of univariate importance. These types correspond to the types in Shrout and Fleiss (1979). If not provided explicitly, this parameter is read from settings used at creation of the underlying familiarModel objects.

verbose

Flag to indicate whether feedback should be provided on the computation and extraction of various data elements.

message_indent

Number of indentation steps for messages shown during computation and extraction of various data elements.

familiar_data_names

Names of the dataset(s). Only used if the object parameter is one or more familiarData objects.

collection_name

Name of the collection.

A familiarCollection object.

A familiarCollection object.

A familiarCollection object.

A familiarCollection object.

A familiarCollection object.

A familiarCollection object.

Variable importance (vimpTable) object, a list thereof, or one or more paths to these objects. This method extracts the variable importance table from such objects.

State of the returned variable importance table. This affects what contents are shown, and in which format. The variable importance table can be returned with the following states:

• initial: initial state, directly after the variable importance table is filled. The returned variable importance table shows the raw, un-processed data.

• decoded: depending on the variable importance method, the initial variable importance table may contain the scores of individual contrasts for categorical variables. When decoded, scores from all contrasts are aggregated to a single score for each feature.

• declustered: variable importance is determined from fully processed features, which includes clustering. This means that a single feature in the variable importance table may represent multiple original features. When a variable importance table has been declustered, all clusters have been turned into their constituent features.

• ranked (default): The scores have been used to create ranks, with lower ranks indicating better features.

Internally, the variable importance table will go through each state, i.e. an variable importance table in the initial state will be decoded, declustered and then ranked prior to returning the variable importance table.

Unused arguments.

Internally used argument for use with familiarModel objects.

Internally used argument for use with familiarModel objects.

Path to the directory where the configuration file should be created. The directory should exist, and no file named config.xml should be present.