alpbench.benchmark.Observer

Classes

Observer()	Abstract class for the observer of the active learning process.
PrintObserver(X_test, y_test)	Observer class to get the statistical performance evaluation of the active learning process such as the distribution of the labeled and selected data, and the model performance on the test data.
StatisticalPerformanceObserver(X_test, y_test)	Observer class for statistical performance evaluation of the active learning process such as the distribution of the labeled and selected data, and the model performance on the test data.

class alpbench.benchmark.Observer.Observer

Bases: ABC

Abstract class for the observer of the active learning process. The observer is responsible for keeping track of the indices of the selected data, the labeled data, and the unlabeled data, as well as the model performance on the test data in each iteration..

abstract observe_data(iteration, X_u_selected, y_u_selected, X_l_aug, y_l_aug, X_u_red, D_l_ind)

Abstract method to observe the data in each iteration.

abstract observe_model(iteration, model)

Abstract method to observe the model performances in each iteration.

class alpbench.benchmark.Observer.PrintObserver(X_test, y_test)

Bases: StatisticalPerformanceObserver

Observer class to get the statistical performance evaluation of the active learning process such as the distribution of the labeled and selected data, and the model performance on the test data.

Parameters:

• X_test (np.array) – test data

• y_test (np.array) – test labels

X_test

test data

Type:

np.array

y_test

test labels

Type:

np.array

observe_data(iteration, X_u_selected, y_u_selected, X_l_aug, y_l_aug, X_u_red, D_l_ind)

Observes the data in each iteration.

Parameters:

• iteration (int) – iteration number

• X_u_selected (np.array) – selected data

• y_u_selected (np.array) – selected labels

• X_l_aug (np.array) – labeled data

• y_l_aug (np.array) – labels of labeled data

• X_u_red (np.array) – unlabeled data

• D_l_ind (np.array) – Indices of X_u selected to be labeled

observe_model(iteration, model)

Observes the model performances in each iteration.

Parameters:

• iteration (int) – iteration number

• model (object) – trained model

class alpbench.benchmark.Observer.StatisticalPerformanceObserver(X_test, y_test)

Bases: Observer, ABC

Observer class for statistical performance evaluation of the active learning process such as the distribution of the labeled and selected data, and the model performance on the test data.

Parameters:

• X_test (np.array) – test data

• y_test (np.array) – test labels

X_test

test data

Type:

np.array

y_test

test labels

Type:

np.array

precision

precision of the floating point numbers

Type:

int

compute_labeling_statistics(iteration, X_u_selected, y_u_selected, X_l_aug, y_l_aug, X_u_red, D_l_ind)

Computes the distribution of the data, which involves the ids of the selected data in each iteration, the overall labeled and unlabeled data.

Parameters:

• iteration (int) – iteration number

• X_u_selected (np.array) – selected data

• y_u_selected (np.array) – selected labels

• X_l_aug (np.array) – labeled data

• y_l_aug (np.array) – labeled labels

• X_u_red (np.array) – unlabeled data

• D_l_ind (np.array) – Indices of X_u selected to be labeled

Returns:

dictionary with the distribution of the labeled and selected data

Return type:

eval_scores (dict)

compute_model_performances(iteration, model)

Computes the model performance on the test data, which involves the f1, precision, recall, log loss, accuracy, and AUC scores.

Parameters:

• iteration (int) – iteration number

• model (object) – trained model

Returns:

dictionary with the model performance on the test data

Return type:

eval_scores (dict)

precision = 8