decent_bench.metrics.plot_metrics#

class decent_bench.metrics.plot_metrics.PlotMetric(*, x_log: bool = False, y_log: bool = True)[source]#

Bases: ABC

Metric to plot at the end of the benchmarking execution.

Parameters:

x_log – whether to apply log scaling to the x-axis.
y_log – whether to apply log scaling to the y-axis.

abstract property x_label: str#: Label for the x-axis.

abstract property y_label: str#: Label for the y-axis.

abstractmethod get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → Sequence[tuple[float, float]][source]#: Extract trial data in the form of (x, y) datapoints.

class decent_bench.metrics.plot_metrics.GlobalCostErrorPerIteration(*, x_log: bool = False, y_log: bool = True)[source]#

Bases: PlotMetric

Global cost error (y-axis) per iteration (x-axis).

Global cost error is defined as:

\[| \sum_i (f_i(\mathbf{\bar{x}}) - f_i(\mathbf{x}^\star)) |\]

where \(f_i\) is agent i’s local cost function, \(\mathbf{\bar{x}}\) is the mean x across all agents, and \(\mathbf{x}^\star\) is the optimal x defined in the problem.

All iterations up to and including the last one reached by all agents are taken into account, subsequent iterations are disregarded. This is done to not miscalculate the global cost error which relies on all agents for its calculation.

x_label: str = 'iteration'#

y_label: str = 'global cost error'#

get_data_from_trial(agents: list[AgentMetricsView], problem: BenchmarkProblem) → list[tuple[float, float]][source]#: Extract trial data in the form of (x, y) datapoints.

class decent_bench.metrics.plot_metrics.GlobalGradientOptimalityPerIteration(*, x_log: bool = False, y_log: bool = True)[source]#

Bases: PlotMetric

Global gradient optimality (y-axis) per iteration (x-axis).

Global gradient optimality is defined as:

\[\| \frac{1}{N} \sum_i \nabla f_i(\mathbf{\bar{x}}) \|^2\]

where N is the number of agents, \(f_i\) is agent i’s local cost function, and \(\mathbf{\bar{x}}\) is the mean x across all agents.

All iterations up to and including the last one reached by all agents are taken into account, subsequent iterations are disregarded. This avoids the curve volatility that occurs when fewer and fewer agents are included in the calculation.

x_label: str = 'iteration'#

y_label: str = 'global gradient optimality'#

get_data_from_trial(agents: list[AgentMetricsView], _: BenchmarkProblem) → list[tuple[float, float]][source]#: Extract trial data in the form of (x, y) datapoints.

decent_bench.metrics.plot_metrics.DEFAULT_PLOT_METRICS#

GlobalCostErrorPerIteration (semi-log)
GlobalGradientOptimalityPerIteration (semi-log)

decent_bench.metrics.plot_metrics.plot(resulting_agent_states: dict[DstAlgorithm, list[list[AgentMetricsView]]], problem: BenchmarkProblem, metrics: list[PlotMetric]) → None[source]#

Plot the execution results with one subplot per metric.

Each algorithm’s curve is its mean across the trials. The surrounding envelope is the min and max across the trials.

Parameters:

resulting_agent_states – resulting agent states from the trial executions, grouped by algorithm
problem – benchmark problem whose properties, e.g. optimal_x, are used for metric calculations
metrics – metrics to calculate and plot

Raises:

RuntimeError – if the figure manager can’t be retrieved