Evaluating an active learning pipeline using the evaluator

In this example we will specify and evaluate an active learning pipeline using the Evaluator, which can be done in only a few lines of code. The query strategy, learning algorithm and setting have to be specified by strings. (Check ALP/evaluation/experimenter/DefaultSetup.py to ensure you are using the same descriptions we are using.)

from sklearn.metrics import accuracy_score

from alpbench.benchmark.BenchmarkConnector import DataFileBenchmarkConnector
from alpbench.evaluation.experimenter.DefaultSetup import ensure_default_setup
from alpbench.pipeline.ALPEvaluator import ALPEvaluator

Establish the database connection

# create benchmark connector and establish database connection
benchmark_connector = DataFileBenchmarkConnector()

# load some default settings and algorithm choices
ensure_default_setup(benchmark_connector)

Fit an ALP

# we choose dataset with **openmlid 31**, the ALP is composed out of a **rf and BALD**
evaluator = ALPEvaluator(
    benchmark_connector=benchmark_connector,
    setting_name="small",
    openml_id=31,
    query_strategy_name="bald",
    learner_name="rf_entropy",
)

alp = evaluator.fit()

# fit / predict and evaluate predictions
X_test, y_test = evaluator.get_test_data()
y_hat = alp.predict(X=X_test)
print("final test acc", accuracy_score(y_test, y_hat))

final test acc 0.7454545454545455