Ramble Results

Ramble provides support and tooling around processing results data.

See ramble workspace analyze (and associated workspace documentation) for generating such data.

Most notably, the two supported functionalities are:

  • Uploading of stand alone JSON in ramble-output format

  • Generation of plots and analysis as part of a report

Upload

The ramble results upload command is used to import experiment results from a JSON file, and upload them as specified in the upload block of Ramble’s config file (see config documentation).

This can be used for both results from an arbitrary ramble workspace, or for data that is generated by tools which conform to rambles data schema.

Index

The ramble results index command is used to list attributes in results, including figures of merit and template variables. This is useful for discovering what metrics are available for plotting or analysis.

ramble results index -f ~/ramble-results.json

Report

The ramble results report command is experimental functionality that provides the user a way to generate plots and reports from the output of a workspace. This output is in the form of individual png plots, and a combined pdf report

Multiple plot types are supported, to aid in comparison of runs and common tasks, including:

  • Scaling (line) plots (both strong and weak)

  • Multi-line line plots

  • Comparison bar plots

  • General FOM bar plots

For these plots, common operations are supported such as log axis, data grouping, and idealized line plotting.

Scaling and Multi-line Scaling Plots

Activated by the strong-scaling, weak-scaling, or multi-line subcommands, these plots generate a line plot and attempt to annotate idealized scaling based on the data input.

These commands take two sub-arguments:

  • The y-axis metric: the FOM you want to measure the performance of

  • The x-axis metric: the scaling variable to plot on the x axis

And example call may look something like:

ramble results report --strong-scaling 'simpleFoam Time' n_nodes -f ~/ramble-results.json

(a very similar command should also work for --weak-scaling and --multi-line)

Another common use case might be to plot data by context (such as in OMB), which can be done by simply changing the y-axis metric:

ramble results report --weak-scaling 'Bandwidth' context -f ~/ramble-results.json

Other compelling options include -n to normalize the plot y values, or --logx/--logy for log axes.

The --pandas-where option allows for down-selecting data using Pandas query format, and --split-by can be used to generate separate plots based on a Ramble variable.

Comparison and FOM plots

A comparison plot compares a specific FOM across all valid experiments, and generates a bar plot with one bar per experiment. It can be invoked by:

ramble results report --compare "Total Elapsed Time" -f ~/ramble-results.json

A FOM plot gives the user a quick way to visualize all FOMS in a workspace as a bar chart, for general data discovery. It can be invoked by:

ramble results report --foms -f ~/ramble-results.json