Plot a metric across multiple values of a model parameter

This function is to be used on simulations in which generate_model was called using the vary_along parameter. When this is a single (scalar) numeric parameter, a single plot is created in which the x-axis is this parameter. Eventually, this function should handle one or two categorical variables (in which facets are used) and one categorical combined with one continuous variable.

Usage

plot_eval_by(
  sim,
  metric_name,
  varying,
  type = c("aggregated", "raw"),
  center_aggregator = NULL,
  spread_aggregator = NULL,
  use_ggplot2 = TRUE,
  main,
  xlab,
  ylab,
  xlim,
  ylim,
  include_zero = FALSE,
  legend_location = "topright",
  method_col = seq(num_methods),
  method_lty = rep(1, num_methods),
  method_lwd = rep(1, num_methods),
  method_pch = rep(1, num_methods),
  ...
)

Arguments

sim: an object of class Simulation
metric_name: the name of a metric to plot (ignored if custom aggregator is provided)
varying: character vector giving the name of a parameter that is varied across the models in evals. For now, this parameter must be numeric and there cannot be multiple models having the same value of this parameter.
type: if "aggregated" then shows line with error bars (line represents center_aggregator and error bars represent spread_aggregator; by default these are sample mean and estimated standard error); if type is "raw" then shows the raw data as points (with smoother overlayed)
center_aggregator: ignored if type is "raw". When NULL (which is default), the sample mean aggregator is used. User can write specialized aggregators (see definition of class Aggregator) as necessary, for example, when the evaluated metric is not scalar-valued.
spread_aggregator: ignored if type is "raw". When NULL (which is default), the sample mean aggregator is used. User can write specialized aggregators (see definition of class Aggregator) as necessary, for example, when the evaluated metric is not scalar-valued. Set spread_aggregator to NA to hide error bars.
use_ggplot2: whether to use ggplot2 (requires installation of ggplot2)
main: title of plot.
xlab: the x-axis label (default is varying)
ylab: the y-axis label (default is metric_label)
xlim: the x-axis limits to use
ylim: the y-axis limits to use
include_zero: whether ylim should include 0. Ignored if ylim is passed explicitly
legend_location: location of legend. Set to NULL to remove legend.
method_col: color to use for each method
method_lty: line style to use for each method
method_lwd: line thickness to use for each method
method_pch: point style to use for each method (default is that no points, only lines are drawn)
...: additional arguments to pass to plot (only when use_ggplot2 = FALSE).

Details

When type is "raw", the individual evals are shown (one point per model-draw-method triplet) along with a loess smooth. When type is "aggregated", then center_aggregator and spread_aggregator are used. center_aggregator is used to draw a single line per method in which the individual evals computed for each draw has been been aggregated in some way. By default, the mean_aggregator is used, which simply averages the evals computed across all draws. When spread_aggregator is non-NULL, "error bars" are drawn with (half)widths computed using spread_aggregator. By default, the se_aggregator is used, which gives an estimate of the standard error of the sample mean.

The arguments method_col, method_lty, method_lwd, method_pch only apply when use_ggplot2 is FALSE.

Examples

if (FALSE) {
 # suppose previously we had run the following:
 sim <- new_simulation(name = "normal-example",
                       label = "Normal Mean Estimation",
                       dir = tempdir()) %>%
   generate_model(make_my_example_model,
                  n = list(10, 20, 30),
                  vary_along = "n") %>%
   simulate_from_model(nsim = 50, index = 1:3) %>%
   run_method(my_example_method) %>%
   evaluate(my_example_loss)
   # then we could plot this
   plot_eval_by(sim, "myloss", varying = "n", include_zero = TRUE)
 }