Derived variables

A derived variable is a reporting series calculated from model projections rather than directly by the G-Cubed projection equations. Examples include GDP growth, a share of GDP, a current-price conversion, a sum across model members and the unemployment satellite model.

The gcubed.derivations subsystem validates these calculations and appends their results to the same dataframe shape used by original model variables. Chartpacks can therefore select a derived row such as GDPRGROWTH(USA) in exactly the same way as a model row such as GDPR(USA).

Derived variables are opt in. If a workflow does not pass derivations= or derived_variables=, projections and reports retain their original model-variable behaviour.

See Available derived variables for the definitions already included in the G-Cubed package.

How the subsystem is organised

The subsystem separates the description of a calculation from its execution:

Part Responsibility
DerivationDefinition Describes one output prefix, domain, metadata, dependencies, validation and level/deviation calculations
DerivationContext Provides a read-oriented view of model metadata, parameters and projection data to calculations
Derivations Validates and orders a collection of definitions, calculates their rows and appends them to reporting frames
Registry helpers Resolve built-in prefix names such as GDPRGROWTH into definitions
Workflow helpers Connect derived calculations to runners, projections, deviations and report generation

A definition declares its output domain with model sets such as regions, sectors or goods. The subsystem resolves those sets and generates row names in a consistent form:

PREFIX(member)
PREFIX(member1,member2)
PREFIX()

For example, a definition with prefix="GDPRGROWTH" and sets=("regions",) produces rows such as GDPRGROWTH(USA).

Calculation and validation flow

For level projections, Derivations:

  1. builds a DerivationContext from a projection object;
  2. validates the definitions against the model sets and required inputs;
  3. orders definitions so derived dependencies are calculated first;
  4. calls each definition’s levels(context) function;
  5. aligns the returned values with its output domain and projection years;
  6. attaches standard charting metadata; and
  7. appends the derived rows after the original charting rows.

Deviation reporting uses both the new and original projection contexts. It first checks that they have compatible years, rows and metadata, calculates and caches their derived levels, and then calls each definition’s deviation function. This is important: percentage changes, percentage-point differences and value differences are not interchangeable, so the definition owns its deviation formula and units.

Definitions declare original model inputs with required_variable_prefixes. If one definition uses another derived result, it declares required_derived_prefixes and reads the upstream values with context.derived_level_rows(...). Missing inputs, missing dependencies and dependency cycles fail during validation instead of producing partial output.

Recipe: use existing derived variables in setup

For the normal build 199 workflow, list built-in prefixes in the derivations property of the setup chartpack. The same chartpack can then select their rows:

chartpack:
  file_name: chartpack.yaml
  title: Growth report
  derivations:
    - GDPRGROWTH
    - CONSUMPTIONGROWTH
  charts:
    - title: Real GDP growth
      variable_prefix: GDPRGROWTH
      selectors:
        - dimension: regions
          members: all
    - title: Consumption growth
      variable_prefix: CONSUMPTIONGROWTH
      selectors:
        - dimension: regions
          members: all

Run setup and then the maintained run script:

python setup.py
python run_experiment.py

Setup uses the selected definitions to resolve the derived chart series. The run script calculates those definitions and passes the resulting level and deviation rows to reporting. A standalone chartpack.yaml only selects rows; by itself it does not cause a derived variable to be calculated.

See Generate chartpacks for the rest of the setup chartpack format.

Recipe: use existing definitions from Python

Use registry prefixes when all required definitions are built into G-Cubed:

from gcubed.derivations import create_derivations

derivations = create_derivations([
    "GDPRGROWTH",
    "CONSUMPTIONGROWTH",
])

Create this object once and pass the same instance to the runner and reporting:

from gcubed.reporting import generate_all_simulation_results
from gcubed.runners.simulation_runner import SimulationRunner

runner = SimulationRunner(
    baseline_projections=baseline_projections,
    experiment_design_file=experiment_design_file,
    derivations=derivations,
)
runner.run()

generate_all_simulation_results(
    chartpack_path=chartpack_path,
    documentation_path=documentation_path,
    template_path=template_path,
    results_directory_path=results_directory_path,
    all_projections=runner.all_projections,
    derivations=derivations,
    show_final_results=False,
)

SimulationRunner, AdvancedRunner, SimpleRunner and BaselineRunner accept an optional derivations= argument. Passing the collection to a runner enables early validation; passing it to reporting performs the calculations and includes the rows in CSV and HTML output.

For code that needs reporting dataframes without a complete report, use the direct helpers:

from gcubed.derivations import append_derived_deviations
from gcubed.derivations import append_derived_projections

baseline_levels = append_derived_projections(
    projections=baseline_projections,
    derivations=derivations,
)

scenario_deviations = append_derived_deviations(
    new_projections=scenario_projections,
    original_projections=baseline_projections,
    derivations=derivations,
)

Create a new simple derived variable

A custom derived variable is normally a no-argument factory that returns a DerivationDefinition. It can live in an experiment script when it is specific to that analysis. This example follows the structure of the built-in GDPRGROWTH factory but creates a regional real-GDP index whose first projection year equals 100:

from typing import Any

import pandas as pd

from gcubed.derivations import DerivationDefinition
from gcubed.derivations.domain import domain_tuples, variable_names


def REAL_GDP_INDEX() -> DerivationDefinition:
    prefix = "REAL_GDP_INDEX"
    sets = ("regions",)

    def levels(context: Any) -> pd.DataFrame:
        domains = domain_tuples(sym_data=context.sym_data, sets=sets)
        input_names = variable_names(prefix="GDPR", domains=domains)
        output_names = variable_names(prefix=prefix, domains=domains)
        years = list(context.projection_years)

        real_gdp = context.charting_projections.loc[
            list(input_names), years
        ].astype(float)
        values = real_gdp.div(real_gdp.iloc[:, 0], axis="index") * 100.0
        values.index = output_names
        return values

    def deviations(
        new_context: Any,
        original_context: Any,
    ) -> pd.DataFrame:
        return levels(new_context) - levels(original_context)

    return DerivationDefinition(
        prefix=prefix,
        sets=sets,
        label="real GDP index",
        units="index (first projection year = 100)",
        deviation_units="index-point deviation",
        required_variable_prefixes=("GDPR",),
        levels=levels,
        deviations=deviations,
        description_markdown="""Regional real GDP index.

        Divides each `GDPR(region)` projection by its first projection-year
        value and multiplies the result by 100.
        """,
    )

The two calculation functions return values only: one row for every resolved region and one column for every projection year. The subsystem generates the charting metadata and checks the returned shape.

Use the custom definition explicitly:

from gcubed.derivations import Derivations

derivations = Derivations([REAL_GDP_INDEX()])

Or mix it with a built-in factory:

from gcubed.derivations import Derivations
from gcubed.derivations.definitions import GDPRGROWTH

derivations = Derivations([
    GDPRGROWTH(),
    REAL_GDP_INDEX(),
])

The custom output rows are named REAL_GDP_INDEX(region), so a chartpack can select an exact row such as REAL_GDP_INDEX(USA) once reporting receives this Derivations collection.

Make a definition available by prefix

Keep experiment-specific factories in the experiment code and construct them explicitly. To contribute a generally useful definition to the G-Cubed package:

  1. add its factory under gcubed.derivations.definitions;
  2. import the factory in gcubed.derivations.definitions.__init__;
  3. add its factory name to that module’s __all__ list; and
  4. add unit tests for metadata, validation, levels, deviations and applicable model domains.

The factory must take no arguments and return a valid DerivationDefinition. The registry calls exported factories and indexes them by the returned definition prefix. Once registered, create_derivations(["REAL_GDP_INDEX"]) can resolve it by name.

Definition contract

A definition’s calculation functions receive DerivationContext objects. The context provides:

  • model configuration, SYM metadata and parameters;
  • a copy of charting_projections;
  • the ordered projection-year columns;
  • optional lower-level database_projections; and
  • cached upstream derived rows for declared dependencies.

Both levels and deviations must return a pandas dataframe with one row for each output-domain tuple and every projection-year column. Use domain_tuples(...) and variable_names(...) to keep inputs and outputs in the same deterministic order. Let the subsystem attach descriptions, units, set members, region and vector="derived" metadata.

In addition to model-native sets, a definition can declare local_sets, a domain_filter, a charting_region_set and custom validation rules. Use those features only when the ordinary model domain and prefix checks are insufficient.

Common failures

Error Meaning
DerivationRegistryError A configured prefix or factory could not be resolved
DerivationValidationError A definition is incompatible with the model or its dependencies
DerivationFrameError A projection object or returned level frame violates the dataframe contract
DerivationDeviationError Two contexts cannot be compared or a deviation frame is invalid
DerivationWorkflowError A runner or reporting adapter received invalid workflow inputs

Unknown built-in prefixes, duplicate definitions and missing model inputs fail early. This is deliberate: a report should not silently omit a requested calculation.

For the complete callable signatures, see the derived-variable Python API.

Table of contents

This site uses Just the Docs, a documentation theme for Jekyll.