alpbench.benchmark.BenchmarkConnector

Classes

BenchmarkConnector()	Benchmark Connector
CustomEncoder(*[, skipkeys, ensure_ascii, ...])	This class is a custom JSON encoder that is used to encode the parameters of a learner or query strategy.
DataFileBenchmarkConnector([learner_file, ...])	Data File Benchmark Connector
MySQLBenchmarkConnector(host, user, ...)	MySQL Benchmark Connector

class alpbench.benchmark.BenchmarkConnector.BenchmarkConnector

Bases: ABC

Benchmark Connector

This abstract class defines the interface for a benchmark connector. A benchmark connector is responsible for storing and loading all data related to the active learning benchmark. This includes the active learning setting and scenario as well as the learner and query strategy with all used parameters. The data is stored in a database or file and can be accessed by the respective methods of this class.

abstract load_learner(learner_id)

Abstract method that loads the learner with the specified ID from the database.

abstract load_learner_by_name(learner_name)

Abstract method that loads the learner with the specified name from the database.

abstract load_or_create_learner(learner_name, obj)

Abstract method that loads the learner with the specified parameters from the database. If the learner does not exist yet, it is created and then returned to the invoker.

abstract load_or_create_query_strategy(query_strategy_name, obj)

Abstract method that loads the query strategy with the specified parameters from the database. If the query strategy does not exist yet, it is created and then returned to the invoker.

abstract load_or_create_scenario(openml_id, test_split_seed, train_split_seed, seed, setting_id)

Abstract method that loads the scenario with the specified parameters from the database. If the scenario does not exist yet, it is created and then returned to the invoker.

abstract load_or_create_setting(name, labeled_train_size, train_type, test_size, number_of_iterations, number_of_queries, factor)

Abstract method that loads the setting with the specified parameters from the database. If the setting does not exist yet, it is created and then returned to the invoker.

abstract load_query_strategy(query_strategy_id)

Abstract method that loads the query strategy with the specified ID from the database.

abstract load_query_strategy_by_name(query_strategy_name)

Abstract method that loads the query strategy with the specified name from the database.

abstract load_scenario(scenario_id)

Abstract method that loads the scenario with the specified ID from the database.

abstract load_setting(setting_id)

Abstract method that loads the setting with the specified ID from the database.

abstract load_setting_by_name(setting_name)

Abstract method that loads the setting with the specified name from the database.

class alpbench.benchmark.BenchmarkConnector.CustomEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

This class is a custom JSON encoder that is used to encode the parameters of a learner or query strategy.

default(obj)

This method is called by the JSON encoder to encode the specified object.

Parameters:

obj (object) – object to encode

Returns:

the encoded object

Return type:

str

class alpbench.benchmark.BenchmarkConnector.DataFileBenchmarkConnector(learner_file=None, query_strategy_file=None, scenario_file=None, setting_file=None)

Bases: BenchmarkConnector

Data File Benchmark Connector

This class is an implementation of the BenchmarkConnector interface that stores all data in files. This involves the Active Learning Setting and Scenario as well as the Learner and query Strategy with all used parameters. The data is stored in JSON files and can be accessed by the respective methods of this class.

Parameters:

• learner_file (str) – path to the file storing the learner data

• query_strategy_file (str) – path to the file storing the query strategy data

• scenario_file (str) – path to the file storing the scenario data

• setting_file (str) – path to the file storing the setting data

learner_file

path to the file storing the learner data

Type:

str

query_strategy_file

path to the file storing the query strategy data

Type:

str

scenario_file

path to the file storing the scenario data

Type:

str

setting_file

path to the file storing the setting data

Type:

str

cleanup()

dump()

load_learner(learner_id)

This method loads the learner with the specified ID from the database.

Parameters:

learner_id (int) – ID of the learner to load

Returns:

the loaded learner

Return type:

object

load_learner_by_name(learner_name)

This method loads the learner with the specified name from the database.

Parameters:

learner_name (str) – name of the learner to load

Returns:

the loaded learner

Return type:

object

load_or_create_learner(learner_name, obj)

This method checks whether the specified learner already exists in the database. If not, the specified setting is added to the database and then also returned to the invoker.

Parameters:

• learner_name (str) – name of the learner

• obj (object) – the learner to load or create

Returns:

the loaded or created learner

Return type:

object

load_or_create_query_strategy(query_strategy_name, obj)

This method checks whether the specified query strategy already exists in the database. If not, the specified query strategy including its parameterization is added to the database and then also returned to the invoker.

Parameters:

• query_strategy_name (str) – name of the query strategy

• obj (object) – the query strategy to load or create

Returns:

the loaded or created query strategy

Return type:

object

load_or_create_scenario(openml_id, test_split_seed, train_split_seed, seed, setting_id)

This method loads the scenario with the specified parameters from the database. If the scenario does not exist yet, it is created and then returned to the invoker.

Parameters:

• openml_id (int) – ID of the openml dataset

• test_split_seed (int) – seed for the test split

• train_split_seed (int) – seed for the train split

• seed (int) – seed for the scenario

• setting_id (int) – ID of the setting

Returns:

the loaded or created scenario

Return type:

ActiveLearningScenario

load_or_create_setting(name, labeled_train_size, train_type, test_size, number_of_iterations, number_of_queries, factor)

This method checks whether the specified setting already exists. If so, it just fetches the data from the database and returns an instance. If not, the specified setting is added to the database and then also returned.

Parameters:

• name (str) – name of the setting

• labeled_train_size (str) – labeled training size

• train_type (str) – type of training

• test_size (str) – test size

• number_of_iterations (int) – number of iterations

• number_of_queries (int) – number of queries

• factor (int) – factor

Returns:

the loaded or created setting

Return type:

ActiveLearningSetting

load_query_strategy(query_strategy_id)

This method loads the query strategy with the specified ID from the database.

Parameters:

query_strategy_id (int) – ID of the query strategy to load

Returns:

the loaded query strategy

Return type:

object

load_query_strategy_by_name(query_strategy_name)

This method loads the query strategy with the specified name from the database.

Parameters:

query_strategy_name (str) – name of the query strategy to load

Returns:

the loaded query strategy

Return type:

object

load_scenario(scenario_id)

This method loads the scenario with the specified ID from the database.

Parameters:

scenario_id (int) – ID of the scenario to load

Returns:

the loaded scenario

Return type:

ActiveLearningScenario

load_setting(setting_id)

This method loads the setting with the specified ID from the database.

Parameters:

setting_id (int) – ID of the setting to load

Returns:

the loaded setting

Return type:

ActiveLearningSetting

load_setting_by_name(setting_name)

This method loads the setting with the specified name from the database.

Parameters:

setting_name (str) – name of the setting to load

Returns:

the loaded setting

Return type:

ActiveLearningSetting

base_folder = 'db_alpbench/'

learner_file = 'db_alpbench/learner.json'

query_strategy_file = 'db_alpbench/query_strategy.json'

scenario_file = 'db_alpbench/scenario.json'

setting_file = 'db_alpbench/setting.json'

class alpbench.benchmark.BenchmarkConnector.MySQLBenchmarkConnector(host, user, password, database, use_ssl)

Bases: BenchmarkConnector

MySQL Benchmark Connector

This class is an implementation of the BenchmarkConnector interface that stores all data in a MySQL database. This involves the Active Learning Setting and Scenario as well as the Learner and query Strategy with all used parameters. The data is stored in MySQL tables and can be accessed by the respective methods of this class.

Parameters:

• host (str) – host of the MySQL server

• user (str) – user of the MySQL server

• password (str) – password of the MySQL server

• database (str) – database to use

• use_ssl (bool) – whether to use SSL

host

host of the MySQL server

Type:

str

user

user of the MySQL server

Type:

str

password

password of the MySQL server

Type:

str

database

database to use

Type:

str

use_ssl

whether to use SSL

Type:

bool

con

connection to the MySQL database

Type:

mysql.connector.connection.MySQLConnection

scenario_table

name of the table storing the scenarios

Type:

str

setting_table

name of the table storing the settings

Type:

str

learner_table

name of the table storing the learners

Type:

str

query_strategy_table

name of the table storing the query strategies

Type:

str

close()

load_learner(learner_id)

This method loads the learner with the specified ID from the database.

Parameters:

learner_id (int) – ID of the learner to load

Returns:

the loaded learner

Return type:

object

load_learner_by_name(learner_name)

This method loads the learner with the specified name from the database.

Parameters:

learner_name (str) – name of the learner to load

Returns:

the loaded learner

Return type:

object

load_or_create_learner(learner_name, obj)

This method checks whether the specified learner already exists in the database. If not, the specified setting is added to the database and then also returned to the invoker.

Parameters:

• learner_name (str) – name of the learner

• obj (object) – the learner to load or create

Returns:

the loaded or created learner

Return type:

object

load_or_create_query_strategy(query_strategy_name, obj)

This method checks whether the specified query strategy already exists in the database. If not, the specified query strategy including its parameterization is added to the database and then also returned to the invoker.

Parameters:

• query_strategy_name (str) – name of the query strategy

• obj (object) – the query strategy to load or create

Returns:

the loaded or created query strategy

Return type:

object

load_or_create_scenario(openml_id, test_split_seed, train_split_seed, seed, setting_id)

This method loads the scenario with the specified parameters from the database. If the scenario does not exist yet, it is created and then returned to the invoker.

Parameters:

• openml_id (int) – ID of the openml dataset

• test_split_seed (int) – seed for the test split

• train_split_seed (int) – seed for the train split

• seed (int) – seed for the scenario

• setting_id (int) – ID of the setting

Returns:

the loaded or created scenario

Return type:

ActiveLearningScenario

load_or_create_setting(name, labeled_train_size, train_type, test_size, number_of_iterations, number_of_queries, factor)

This method checks whether the specified setting already exists. If so, it just fetches the data from the database and returns an instance. If not, the specified setting is added to the database and then also returned to the invoker.

Parameters:

• name (str) – name of the setting

• labeled_train_size (str) – labeled training size

• train_type (str) – type of training

• test_size (str) – test size

• number_of_iterations (int) – number of iterations

• number_of_queries (int) – number of queries

• factor (int) – factor

Returns:

the loaded or created setting

Return type:

ActiveLearningSetting

load_query_strategy(query_strategy_id)

This method loads the query strategy with the specified ID from the database.

Parameters:

query_strategy_id (int) – ID of the query strategy to load

Returns:

the loaded query strategy

Return type:

object

load_query_strategy_by_name(query_strategy_name)

This method loads the query strategy with the specified name from the database.

Parameters:

query_strategy_name (str) – name of the query strategy to load

Returns:

the loaded query strategy

Return type:

object

load_scenario(scenario_id)

This method loads the scenario with the specified ID from the database.

Parameters:

scenario_id (int) – ID of the scenario to load

Returns:

the loaded scenario

Return type:

ActiveLearningScenario

load_setting(setting_id)

This method loads the setting with the specified ID from the database.

Parameters:

setting_id (int) – ID of the setting to load

Returns:

the loaded setting

Return type:

ActiveLearningSetting

load_setting_by_name(setting_name)

This method loads the setting with the specified name from the database.

Parameters:

setting_name (str) – name of the setting to load

Returns:

the loaded setting

Return type:

ActiveLearningSetting

learner_table = 'salt_learner'

query_strategy_table = 'salt_query_strategy'

scenario_table = 'salt_scenario'

setting_table = 'salt_setting'