decent_bench.metrics.table_metrics

class decent_bench.metrics.table_metrics.TableMetric(statistics: list[Callable[[Sequence[float]], float]])

Bases: ABC

Metric to display in the statistical results table at the end of the benchmarking execution.

Parameters:

statistics – sequence of statistics such as min(), sum(), and average() used for aggregating the data retrieved with get_data_from_trial() into a single value, each statistic gets its own row in the table

abstract property description: str

Metric description to display in the table.

abstractmethod get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → Sequence[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.GlobalCostError(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Global cost error using the agents’ final x.

Global cost error is defined as:

\[| \sum_i (f_i(\mathbf{\bar{x}}) - f_i(\mathbf{x}^\star)) |\]

where \(f_i\) is agent i’s local cost function, \(\mathbf{\bar{x}}\) is the mean x across all agents, and \(\mathbf{x}^\star\) is the optimal x defined in the problem.

description: str = 'global cost error \n[<1e-9 = exact conv.]'

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → tuple[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.GlobalGradientOptimality(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Global gradient optimality using the agents’ final x.

Global gradient optimality is defined as:

\[\| \frac{1}{N} \sum_i \nabla f_i(\mathbf{\bar{x}}) \|^2\]

where N is the number of agents, \(f_i\) is agent i’s local cost function, and \(\mathbf{\bar{x}}\) is the mean x across all agents.

description: str = 'global gradient optimality'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → tuple[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.XError(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

X error per agent as defined below.

\[\{ \|\mathbf{x}_i - \mathbf{x}^\star\|, \|\mathbf{x}_j - \mathbf{x}^\star\|, ... \}\]

where \(\mathbf{x}_i\) is agent i’s final x, \(\mathbf{x}_j\) is agent j’s final x, and \(\mathbf{x}^\star\) is the optimal x defined in the problem.

description: str = 'x error'

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.AsymptoticConvergenceOrder(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Asymptotic convergence order per agent as defined below.

\[\lim_{k \to \infty} \frac{\| \mathbf{x}_{k+1} - \mathbf{x}^\star \|}{\| \mathbf{x}_{k} - \mathbf{x}^\star\|^q} = \mu\]

where \(\mathbf{x}_k\) is the agent’s local x at iteration k, \(\mathbf{x}^\star\) is the optimal x defined in the problem, \(q\) is the asymptotic convergence order, and \(\mu\) is the asymptotic convergence rate.

description: str = 'asymptotic convergence order'

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.AsymptoticConvergenceRate(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Asymptotic convergence rate per agent as defined below.

\[\lim_{k \to \infty} \frac{\| \mathbf{x}_{k+1} - \mathbf{x}^\star \|}{\| \mathbf{x}_{k} - \mathbf{x}^\star\|^q} = \mu\]

where \(\mathbf{x}_k\) is the agent’s local x at iteration k, \(\mathbf{x}^\star\) is the optimal x defined in the problem, \(q\) is the asymptotic convergence order, and \(\mu\) is the asymptotic convergence rate.

description: str = 'asymptotic convergence rate'

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.IterativeConvergenceOrder(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Iterative convergence order per agent as defined below.

\[k = \frac{\mu}{\|\mathbf{x}_k - \mathbf{x}^\star\|^q}\]

where k is the iteration, \(\mu\) is the iterative convergence rate, \(\mathbf{x}_k\) is the agent’s local x at iteration k, \(\mathbf{x}^\star\) is the optimal x defined in the problem, and \(q\) is the iterative convergence order.

As per the definition, iterative convergence is a measure of how many iterations are needed to reach a certain error. This makes iterative convergence order and rate suitable metrics for sublinear algorithms; a sublinear algorithm with iterative convergence order 0.5 generally converges significantly faster than one with order 1.

description: str = 'iterative convergence order'

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.IterativeConvergenceRate(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Iterative convergence rate per agent as defined below.

\[k = \frac{\mu}{\|\mathbf{x}_k - \mathbf{x}^\star\|^q}\]

where k is the iteration, \(\mu\) is the iterative convergence rate, \(\mathbf{x}_k\) is the agent’s local x at iteration k, \(\mathbf{x}^\star\) is the optimal x defined in the problem, and \(q\) is the iterative convergence order.

As per the definition, iterative convergence is a measure of how many iterations are needed to reach a certain error. This makes iterative convergence order and rate suitable metrics for sublinear algorithms; a sublinear algorithm with iterative convergence order 0.5 generally converges significantly faster than one with order 1.

description: str = 'iterative convergence rate'

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrXUpdates(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of iterations/updates of x per agent.

description: str = 'nr x updates'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrEvaluateCalls(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of cost function evaluate calls per agent.

description: str = 'nr evaluate calls'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrGradientCalls(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of cost function gradient calls per agent.

description: str = 'nr gradient calls'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrHessianCalls(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of cost function hessian calls per agent.

description: str = 'nr hessian calls'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrProximalCalls(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of cost function proximal calls per agent.

description: str = 'nr proximal calls'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrSentMessages(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of sent messages per agent.

description: str = 'nr sent messages'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrReceivedMessages(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of received messages per agent.

description: str = 'nr received messages'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

class decent_bench.metrics.table_metrics.NrSentMessagesDropped(statistics: list[Callable[[Sequence[float]], float]])

Bases: TableMetric

Number of sent messages that were dropped per agent.

description: str = 'nr sent messages dropped'

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[float]

Extract trial data to be aggregated into a single value by each of the statistics.

decent_bench.metrics.table_metrics.DEFAULT_TABLE_METRICS

• GlobalCostError - single()

• GlobalGradientOptimality - single()

• XError - min(), average(), max()

• AsymptoticConvergenceOrder - average()

• AsymptoticConvergenceRate - average()

• IterativeConvergenceOrder - average()

• IterativeConvergenceRate - average()

• NrXUpdates - average(), sum()

• NrEvaluateCalls - average(), sum()

• NrGradientCalls - average(), sum()

• NrHessianCalls - average(), sum()

• NrProximalCalls - average(), sum()

• NrSentMessages - average(), sum()

• NrReceivedMessages - average(), sum()

• NrSentMessagesDropped - average(), sum()

decent_bench.metrics.table_metrics.tabulate(resulting_agent_states: dict[DstAlgorithm, list[list[AgentMetricsView]]], problem: BenchmarkProblem, metrics: list[TableMetric], confidence_level: float, table_fmt: Literal['grid', 'latex']) → None

Print table with confidence intervals, one row per metric and statistic, and one column per algorithm.

Parameters:

• resulting_agent_states – resulting agent states from the trial executions, grouped by algorithm

• problem – benchmark problem whose properties, e.g. optimal_x, are used for metric calculations

• metrics – metrics to calculate

• confidence_level – confidence level of the confidence intervals

• table_fmt – table format, grid is suitable for the terminal while latex can be copy-pasted into a latex document