Classes
ModelValidator
Wraps any sklearn-compatible estimator and provides comprehensive validation metrics and Plotly figures for NMR metabolomics workflows.
Args:
estimatorAny sklearn-compatible classifier that implements``fit``, ``predict``, and ``predict_proba``.
XFeature matrix of shape (n_samples, n_features).yTarget labels of shape (n_samples,).cvNumber of stratified k-fold splits used for all CV-basedmethods. Defaults to 5.
random_stateRandom seed for reproducibility. Defaults to 42.class_namesDisplay names for each class. If None, the uniquesorted values of ``y`` are used.
Attributes: roc_results_ (dict): Populated by :meth:`roc_auc`. Maps each class label to a dict containing ``fpr``, ``tpr``, and ``auc``. bootstrap_scores_ (np.ndarray): Populated by :meth:`bootstrap_ci`. Array of per-iteration metric values. bootstrap_ci_ (tuple[float, float]): Populated by :meth:`bootstrap_ci`. Lower and upper confidence bounds.
Examples: >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import make_classification >>> import pandas as pd >>> X_arr, y_arr = make_classification(n_samples=100, random_state=0) >>> X = pd.DataFrame(X_arr) >>> validator = ModelValidator( ... estimator=RandomForestClassifier(random_state=0), ... X=X, ... y=y_arr, ... cv=3, ... ) >>> fig = validator.roc_auc() >>> df = validator.get_cv_summary()
Methods
__init__(self, estimator, X: pd.DataFrame | np.ndarray, y: pd.Series | np.ndarray | list, cv: int=5, random_state: int=42, class_names: list[str] | None=None)
roc_auc(self, fig_height: int=600, fig_width: int=800, font_size: int=14)
Compute and plot ROC curves using stratified k-fold CV.
One curve is produced per class (one-vs-rest) plus a micro-average curve. AUC values are annotated in the legend. Results are stored in ``self.roc_results_``.
Args:
fig_heightHeight of the returned Plotly figure in pixels.Defaults to 600.
fig_widthWidth of the returned Plotly figure in pixels.Defaults to 800.
font_sizeBase font size for the figure. Defaults to 14.Returns: Plotly Figure containing the ROC curve plot.
Examples: >>> fig = validator.roc_auc() >>> fig.show()
nested_cv(self, inner_cv: int=3, param_grid: dict | None=None, scoring: str='balanced_accuracy', n_jobs: int=-1)
Perform nested cross-validation.
The outer loop uses ``self.cv`` stratified folds. When ``param_grid`` is supplied, the inner loop runs a ``GridSearchCV`` with ``inner_cv`` folds for hyperparameter selection. When ``param_grid`` is None, the inner loop is skipped and plain outer CV is performed.
Args:
inner_cvNumber of inner folds for hyperparameter search.Only used when ``param_grid`` is provided. Defaults to 3.
param_gridParameter grid passed to ``GridSearchCV``.If None, no inner loop is performed. Defaults to None.
scoringSklearn scoring string used for both inner andouter evaluation. Defaults to "balanced_accuracy".
n_jobsNumber of parallel jobs. Defaults to -1 (all CPUs).Returns: DataFrame with columns ``fold``, ``train_score``, and ``test_score`` (one row per outer fold). Mean and std of test scores are printed to stdout.
Examples: >>> df = validator.nested_cv(param_grid={"n_estimators": [50, 100]}) >>> print(df)
bootstrap_ci(self, metric: str='balanced_accuracy', n_bootstrap: int=1000, ci: float=0.95, fig_height: int=400, fig_width: int=700, font_size: int=14)
Estimate performance confidence intervals via bootstrap resampling.
The estimator is first fit on the full dataset. For each bootstrap iteration the out-of-bag (OOB) samples are used for evaluation. The resulting distribution and CI bounds are shown as a histogram.
Supported ``metric`` values: - ``"accuracy"`` - ``"balanced_accuracy"`` - ``"f1_macro"`` - ``"roc_auc_ovr"``
Args:
metricPerformance metric to compute on each bootstrapsample. Defaults to "balanced_accuracy".
n_bootstrapNumber of bootstrap iterations. Defaults to 1000.ciConfidence level (e.g. 0.95 for 95% CI). Defaults to 0.95.fig_heightHeight of the returned figure in pixels.Defaults to 400.
fig_widthWidth of the returned figure in pixels.Defaults to 700.
font_sizeBase font size. Defaults to 14.Returns: Plotly Figure showing the bootstrap score distribution with CI bounds annotated.
Raises:
ValueErrorIf ``metric`` is not one of the supported options.Examples: >>> fig = validator.bootstrap_ci(metric="f1_macro", n_bootstrap=500) >>> fig.show()
confusion_matrix(self, normalize: bool=True, fig_height: int=600, fig_width: int=700, font_size: int=14)
Compute and plot a confusion matrix from CV predictions.
Predictions are obtained via ``cross_val_predict`` with a stratified k-fold splitter. When ``normalize=True`` each row is divided by its sum so that cells show proportions.
Args:
normalizeWhether to normalize rows to sum to 1.Defaults to True.
fig_heightHeight of the returned figure in pixels.Defaults to 600.
fig_widthWidth of the returned figure in pixels.Defaults to 700.
font_sizeBase font size. Defaults to 14.Returns: Plotly Figure containing an annotated heatmap of the confusion matrix.
Examples: >>> fig = validator.confusion_matrix(normalize=False) >>> fig.show()
get_cv_summary(self)
Compute cross-validation summary statistics.
Evaluates the estimator using stratified k-fold CV and returns mean and standard deviation for accuracy, balanced accuracy, and macro F1.
Returns: DataFrame with columns ``metric``, ``mean``, and ``std``, one row per metric.
Examples: >>> summary = validator.get_cv_summary() >>> print(summary) metric mean std 0 accuracy 0.920 0.031 1 balanced_accuracy 0.918 0.033 2 f1_macro 0.919 0.032