Running experiments

Table of contents

• Running experiments
  • The anatomy of a G-Cubed simulation experiment
    • Solve the model
    • Generate baseline projections
    • Produce layers of simulation projections
    • Review deviations from baseline projections
  • The simulations folder
  • Experiment Python scripts
  • Experiment results

These instructions assume that you have a working G-Cubed deployment.

G-Cubed simulation experiments are run using Python scripts. Those scripts are contained in experiment folders, the child folders of the model’s simulations folder.

The anatomy of a G-Cubed simulation experiment

A typical G-Cubed simulation experiment involves the following steps.

Solve the model

Solving the model involves obtaining a way of producing expectations-consistent projections of all endogenous variables in the system given a set of exogenous variable projections and a set of starting values for state variables. This solution is known as the model solution.

Generate baseline projections

Apply the model solution to a baseline set of exogenous variable projections and a set of starting values for state variables to produce expectations-consistent projections of all endogenous variables in the system. These projections are known as the baseline projections. Model users have a great deal of control over the exogenous variable projections in the baseline, as described in the documentation of how baseline projections are formed.

Produce layers of simulation projections

One or more layers of simulation projections are generated, by applying layers of new adjustments or shocks to exogenous (and optionally state) variables in the model.

Review deviations from baseline projections

Compare the baseline projections to a new set of simulation projections that are obtained after applying the simulation adjustments. The difference between the baseline projections and the simulation projections is known as the deviation projections.

Experiments can be made considerably more complex than this simple outline described above, by adding additional layers of simulation adjustments. Each layer of shocks to exogenous variables is known as a simulation layer. Deviations can be obtained between the projections for any simulation layer and the baseline projections or between any pair of simulation layers. It is also possible to work with deviations between projections from different scenarios or experiments run with the same G-Cubed model.

The process of obtaining a model solution is computationally intensive, requiring several minutes for larger models on modern computers. Fortunately, once obtained, a model solution can be reused across different simulation experiments. A good practice for re-using the model solution is incorporated into the recommmended process of running experiments.

The simulations folder

The simulations folder contains a child folder with the files needed to run the model baseline projections and a child folder for each simulation experiment.

Typically, the child folder for the baseline projections, baseline, contains a python script called setup_baseline.py. It is a good practice to setup and run this baseline first. Run the setup_baseline.py script to create the python script that solves the model and generates the baseline projections and reports on them via your web browser. That solves the model and preserves the solution of the model for use by other experiments. The solved model will be stored in the results/<VERSION>/<BUILD>/baseline/solved_model.joblib file.

In modern G-Cubed deployments, each usage of the model, to generate the baseline projections or to do scenario simulations, needs to be set-up in association with the model being used for the experiment. This involves running a setup_baseline.py or setup_experiment.py script. The setup process ensures that shock files and chartpacks and other resources are matched to the regions, sectors and variables of the specific model being used.

For each experiment, when you run the setup_experiment.py script, it will generate documentation files, shock files, the run_experiment.py script and other resources as required for the experiment. If you want to alter the experiment, within reason, you can do so by editing the YAML configuration details in the setup_experiment.py script.

Experiment Python scripts

Each experiment is a Python file, typically called run_experiment.py. Each experiment is associated with an experiment design and a set of one or more simulation layers (sets of shocks to exogenous variables).

Typically, the script uses a Runner object. There are a number of Runner classes. They run the experiment and save the results to the results folder. A Runner will run an entire experiment, using a model solution and building on a set of baseline projections by sequentially applying the sequence of simulation layers to those baseline projections.

The projections associated with the application of each simulation layer are preserved as a Projections object. To access the full list of projections produced when running an experiment, access the Runner.all_projections property. Each Projections object provides access to the projections implied by the baseline and the cumulative impact of the simulation layers up to and including its own simulation layer. The Runner also provides direct access to the baseline_projections property and the final_projections property for convenience. The baseline_projections is the first Projections in the Runner.all_projections property and the final_projections is the last Projections in the Runner.all_projections` property.

In this way, the Runner object provides a way to analyse the marginal and cumulative impacts of each simulation layer. To assess the marginal impact of a simulation layer work with deviations of the simulation layer’s projections from the projections associated with the previous simulation layer. To assess the cumulative impact of all simulation layers up to and including the simulation layer of interest, work with deviations of that simulation layer’s projections from the baseline projections.

Experiment results

Results, including log files are saved to folders that are organised by model version and build and the experiment name, /results/<VERSION>/<BUILD>/<experiment folder name>.

Results typically include at least:

• The logs in the run.log file
• HTML reports for each layer of projections and for the deviations of each layer of projections from the baseline projections in the html_reports folder
• CSV files containing the data that underlie the HTML reports in the projections_data folder

Open the HTML file in your web browser (or VS Code) to view the results of the experiment.