HpBandSterSearchCV

class hpbandster_sklearn.HpBandSterSearchCV.HpBandSterSearchCV(estimator, param_distributions, *, n_iter=10, optimizer='bohb', nameserver_host='127.0.0.1', nameserver_port=9090, min_budget=None, max_budget=None, resource_name=None, resource_type=None, cv=None, scoring=None, warm_start=True, refit=True, error_score=nan, return_train_score=False, random_state=None, n_jobs=None, verbose=0, **kwargs)

Bases: sklearn.model_selection._search.BaseSearchCV

Hyper parameter search using HpBandSter.

This class provides a scikit-learn compatible wrapper over HpBandSter, implementing the entire HpBandSter search process (Nameserver, Worker, Optimizer).

In addition to scikit-learn estimators, early stopping support is built in for LightGBM, XGBoost and CatBoost estimators.

Parameters:
  • estimator (estimator object) – This is assumed to implement the scikit-learn estimator interface. Either estimator needs to provide a score function, or scoring must be passed.
  • param_distributions (dict or ConfigurationSpace) – Either a ConfigurationSpace object or a dictionary with parameters names (string) as keys and lists of parameters to try. If a list is given, it is sampled uniformly. Using a ConfigurationSpace object is highly recommended. Refer to ConfigSpace documentation.
  • n_iter (int, default=10) – The number of optimizer iterations to perform.
  • optimizer (str or Optimizer type, default='bohb') –

    The HpBandSter optimizer to use. Can be either an Optimizer type (not object!), or one of the folowing strings representing a HpBandSter optimizer.

    • ’bohb’ - BOHB
    • ’random’ or ‘randomsearch’ - RandomSearch
    • ’hyperband’ - HyperBand
    • ’h2bo’ - H2BO
  • nameserver_host (str, default='127.0.0.1') – The hostname to use for the HpBandSter nameserver. Required even when ran locally.
  • nameserver_port (int, default=9090) – The port to use for the HpBandSter nameserver. Required even when ran locally.
  • min_budget (int or float, default=None) –

    The minimum budget (amount of resource) to consider. Must be bigger than 0. If None, will be:

    • n_splits * 2 when resource_name='n_samples' for a regression problem
    • n_classes * n_splits * 2 when resource_name='n_samples' for a classification problem
    • 10 when resource_name != 'n_samples'

    If resource_name is or is determined to be n_samples, an int will translate to that many samples in the dataset and float will translate to that big fraction of a dataset.

  • max_budget (int or float, default=None) –

    The maximum budget (amount of resource) to consider. Must be bigger than 0 and min_budget.

    If None, will be:

    • n_samples (the size X passed in fit) when resource_name='n_samples'
    • 100 when resource_name != 'n_samples'

    If resource_name is or is determined to be n_samples, an int will translate to that many samples in the dataset and float will translate to that big fraction of a dataset.

  • resource_name (‘n_samples’, str or None, default=None) –

    Defines the name of the resource to be increased with each iteration. If None (default), the resource name will be automatically determined to be one of (in order):

    • ’n_estimators’ - if estimator posses that attribute and has warm_start attribute
    • ’max_iter’ - if estimator posses that attribute and has warm_start attribute
    • ’n_samples’ - the number/fraction of samples

    ’n_estimators’ will also be used for LightGBM, XGBoost and CatBoost estimators.

  • resource_type (type, default=None) – Defines the Python type of resource - either int or float. If None, (default), will try to automatically determine the type based on resource_name, min_budget and `max_budget`.
  • cv (int, cross-validation generator or an iterable, default=5) –

    Determines the cross-validation splitting strategy. Possible inputs for cv are:

    • integer, to specify the number of folds in a (Stratified)KFold,
    • CV splitter,
    • An iterable yielding (train, test) splits as arrays of indices.

    For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

    Note

    Due to implementation details, the folds produced by cv must be the same across multiple calls to cv.split(). For built-in scikit-learn iterators, this can be achieved by deactivating shuffling (shuffle=False), or by setting the cv’s random_state parameter to an integer.

  • scoring (str, callable, or None, default=None) – A single string (see scoring_parameter) or a callable (see scoring) to evaluate the predictions on the test set. If None, the estimator’s score method is used.
  • warm_start (bool, default=True) – if estimator has attribute of ‘warm_start’ and ‘warm_start’=True, the fitting process will reuse the solution of the previous call to fit otherwise just fit a whole estimator.
  • refit (bool, default=True) –

    If True, refit an estimator using the best found parameters on the whole dataset. The estimator will be refit with the maximum amount of the resource.

    The refitted estimator is made available at the best_estimator_ attribute and permits using predict directly on this GridSearchCV instance.

  • error_score ('raise' or numeric, default=np.nan) – Value to assign to the score if an error occurs in estimator fitting. If set to ‘raise’, the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.
  • return_train_score (bool, default=False) – If False, the cv_results_ attribute will not include training scores. Computing training scores is used to get insights on how different parameter settings impact the overfitting/underfitting trade-off. However computing the scores on the training set can be computationally expensive and is not strictly required to select the parameters that yield the best generalization performance.
  • random_state (int, RandomState instance or None, default=None) – Pseudo random number generator state used for subsampling the dataset when resources != ‘n_samples’. Also used for random uniform sampling from lists of possible values instead of scipy.stats distributions. Pass an int for reproducible output across multiple function calls.
  • n_jobs (int or None, default=None) – Number of workers to spawn. Each worker runs in a separate thread. None means 1. -1 means maxiumum amount of cores, -2 means one less than maximum and so on. If LOKY_MAX_CPU_COUNT OS enviromental variables is set, it will be used as the maximum number of CPU cores. Otherwise, for better performance, if psutil is installed the maximum value will be the number of physical CPU cores. Otherwise, the number of logical CPU cores will be used.
  • verbose (int) – Controls the verbosity: the higher, the more messages.
  • **kwargs – Keyword arguments to be passed to the Optimizer. Refer to HpBandSter documentation.
n_resources_

The amount of resources used at each iteration.

Type:list of int or float
n_candidates_

The number of candidate parameters that were evaluated at each iteration.

Type:list of int
n_remaining_candidates_

The number of candidate parameters that are left after the last iteration. It corresponds to ceil(n_candidates[-1] / factor)

Type:int
max_resources_

The maximum number of resources that any candidate is allowed to use for a given iteration. Note that since the number of resources used at each iteration must be a multiple of min_resources_, the actual number of resources used at the last iteration may be smaller than max_resources_.

Type:int or float
min_resources_

The amount of resources that are allocated for each candidate at the first iteration.

Type:int or float
resource_name_

The name of the resource.

Type:str
n_iterations_

The actual number of iterations that were run.

Type:int
cv_results_

A dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame. It contains many informations for analysing the results of a search.

Type:dict of numpy (masked) ndarrays
best_estimator_

Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.

Type:estimator or dict
best_score_

Mean cross-validated score of the best_estimator.

Type:float
best_params_

Parameter setting that gave the best results on the hold out data.

Type:dict
best_index_

The index (of the cv_results_ arrays) which corresponds to the best candidate parameter setting.

The dict at search.cv_results_['params'][search.best_index_] gives the parameter setting for the best model, that gives the highest mean score (search.best_score_).

Type:int
scorer_

Scorer function used on the held out data to choose the best parameters for the model.

Type:function or a dict
n_splits_

The number of cross-validation splits (folds/iterations).

Type:int
refit_time_

Seconds used for refitting the best model on the whole dataset.

This is present only if refit is not False.

Type:float

Notes

The parameters selected are those that maximize the score of the held-out data, according to the scoring parameter.

Examples

>>> from sklearn.datasets import load_iris
>>> from sklearn.ensemble import RandomForestClassifier
>>> from hpbandster_sklearn import HpBandSterSearchCV
...
>>> X, y = load_iris(return_X_y=True)
>>> clf = RandomForestClassifier(random_state=0)
>>> np.random.seed(0)
...
>>> param_distributions = {"max_depth": [3, 4],
...                        "min_samples_split": list(range(2, 12))}
>>> search = HpBandSterSearchCV(clf, param_distributions,
...                                resource_name='n_estimators',
...                                random_state=0, n_jobs=1).fit(X, y)
>>> search.best_params_  # doctest: +SKIP
fit(X, y, groups=None, **fit_params)

Run fit with all sets of parameters.

Parameters:
  • X (array-like of shape (n_samples, n_features)) – Training vector, where n_samples is the number of samples and n_features is the number of features.
  • y (array-like of shape (n_samples, n_output) or (n_samples,), default=None) – Target relative to X for classification or regression; None for unsupervised learning.
  • groups (array-like of shape (n_samples,), default=None) – Group labels for the samples used while splitting the dataset into train/test set. Only used in conjunction with a “Group” cv instance (e.g., GroupKFold).
  • **fit_params (dict of str -> object) – Parameters passed to the fit method of the estimator.
Returns:

self – Instance of fitted estimator.
Return type:

object