Derived variables
A derived variable is a reporting series calculated from model projections rather than directly by the G-Cubed projection equations. Examples include GDP growth, a share of GDP, a gas’s share of CO2-equivalent emissions, a current-price conversion, a sum across model members, a grouped aggregate over selected members and the unemployment satellite model.
The gcubed.derivations subsystem validates these calculations and appends their results to the same dataframe shape used by original model variables. Chartpacks can therefore select a derived row such as GDPRGROWTH(USA) in exactly the same way as a model row such as GDPR(USA).
Derived variables are opt in. If a workflow does not pass derivations= or derived_variables=, projections and reports retain their original model-variable behaviour.
See Available derived variables for the definitions already included in the G-Cubed package. The gas emissions-share guide describes gas composition within activities and totals. The separate world emissions-contribution guide describes regional sector, household, government and total contributions to world emissions, including migration from the removed legacy definitions.
Some models also construct model-local definitions that are not part of the G-Cubed package registry. These remain owned and tested by the model that supplies them; they are not included in the available-derived-variable catalogue.
How the subsystem is organised
The subsystem separates the description of a calculation from its execution:
| Part | Responsibility |
|---|---|
DerivationDefinition | Describes one ordinary output prefix, domain, metadata, dependencies, validation and level/deviation calculations |
AggregateGroup | Describes one synthetic output member and the source members that form it |
GroupedAggregationDefinition | Describes a terminal grouped calculation, independently identifying the definition and its emitted row prefix |
DerivationContext | Provides a read-oriented view of model metadata, parameters and projection data to calculations |
Derivations | Owns ordinary and grouped definitions, runs them in the correct stages and appends their rows to reporting frames |
| Registry helpers | Resolve built-in prefix names such as GDPRGROWTH into definitions |
| Workflow helpers | Connect derived calculations to runners, projections, deviations and report generation |
A definition declares its output domain with model sets such as regions, sectors or goods. The subsystem resolves those sets and generates row names in a consistent form:
PREFIX(member)
PREFIX(member1,member2)
PREFIX()
For example, a definition with prefix="GDPRGROWTH" and sets=("regions",) produces rows such as GDPRGROWTH(USA).
Calculation and validation flow
For level projections, Derivations:
- builds a
DerivationContextfrom a projection object; - validates ordinary and grouped definitions against the model sets and required inputs;
- orders ordinary definitions so their derived dependencies are calculated first;
- calculates and caches all ordinary definition rows;
- creates a combined context containing native model rows and completed ordinary rows;
- calculates terminal grouped definitions from that combined context;
- aligns all returned values with their output domains and projection years;
- attaches standard charting metadata; and
- appends ordinary rows followed by grouped rows after the original model rows.
Model projection rows
↓
Dependency-ordered ordinary definitions
↓
Combined native and ordinary rows
↓
Terminal grouped definitions
↓
Reporting CSVs and charts
Deviation reporting uses both the new and original projection contexts. It first checks that they have compatible years, rows and metadata, calculates and caches their derived levels, and then calls each definition’s deviation function. This is important: percentage changes, percentage-point differences and value differences are not interchangeable, so the definition owns its deviation formula and units.
Definitions declare original model inputs with required_variable_prefixes. If one ordinary definition uses another ordinary result, it declares required_derived_prefixes and reads the upstream values with context.derived_level_rows(...). A grouped definition may also require and aggregate a completed ordinary prefix. Grouped definitions are terminal: they cannot depend on or chain through other grouped outputs. Missing inputs, missing ordinary dependencies and dependency cycles fail during validation instead of producing partial output.
Recipe: use existing derived variables in setup
For the setup workflow, list built-in prefixes in the derivations property of the setup chartpack. The same chartpack can then select their rows:
chartpack:
file_name: chartpack.yaml
title: Growth report
derivations:
- GDPRGROWTH
- CONSUMPTIONGROWTH
charts:
- title: Real GDP growth
variable_prefix: GDPRGROWTH
selectors:
- dimension: regions
members: all
- title: Consumption growth
variable_prefix: CONSUMPTIONGROWTH
selectors:
- dimension: regions
members: all
Run setup and then the maintained run script:
python setup.py
python run_experiment.py
Setup uses the selected definitions to resolve the derived chart series. The run script calculates those definitions and passes the resulting level and deviation rows to reporting. A standalone chartpack.yaml only selects rows; by itself it does not cause a derived variable to be calculated.
See Generate chartpacks for the rest of the setup chartpack format.
Recipe: use existing definitions from Python
Use registry prefixes when all required definitions are built into G-Cubed:
from gcubed.derivations import create_derivations
derivations = create_derivations([
"GDPRGROWTH",
"CONSUMPTIONGROWTH",
])
Create this object once and pass the same instance to the runner and reporting:
from gcubed.reporting import generate_all_simulation_results
from gcubed.runners.simulation_runner import SimulationRunner
runner = SimulationRunner(
baseline_projections=baseline_projections,
experiment_design_file=experiment_design_file,
derivations=derivations,
)
runner.run()
generate_all_simulation_results(
chartpack_path=chartpack_path,
documentation_path=documentation_path,
template_path=template_path,
results_directory_path=results_directory_path,
all_projections=runner.all_projections,
derivations=derivations,
show_final_results=False,
)
SimulationRunner, AdvancedRunner, SimpleRunner and BaselineRunner accept an optional derivations= argument. Passing the collection to a runner enables early validation; passing it to reporting performs the calculations and includes the rows in CSV and HTML output.
For code that needs reporting dataframes without a complete report, use the direct helpers:
from gcubed.derivations import append_derived_deviations
from gcubed.derivations import append_derived_projections
baseline_levels = append_derived_projections(
projections=baseline_projections,
derivations=derivations,
)
scenario_deviations = append_derived_deviations(
new_projections=scenario_projections,
original_projections=baseline_projections,
derivations=derivations,
)
Create a new simple derived variable
A custom derived variable is normally a no-argument factory that returns a DerivationDefinition. It can live in an experiment script when it is specific to that analysis. This example follows the structure of the built-in GDPRGROWTH factory but creates a regional real-GDP index whose first projection year equals 100:
from typing import Any
import pandas as pd
from gcubed.derivations import DerivationDefinition
from gcubed.derivations.domain import domain_tuples, variable_names
def REAL_GDP_INDEX() -> DerivationDefinition:
prefix = "REAL_GDP_INDEX"
sets = ("regions",)
def levels(context: Any) -> pd.DataFrame:
domains = domain_tuples(sym_data=context.sym_data, sets=sets)
input_names = variable_names(prefix="GDPR", domains=domains)
output_names = variable_names(prefix=prefix, domains=domains)
years = list(context.projection_years)
real_gdp = context.charting_projections.loc[
list(input_names), years
].astype(float)
values = real_gdp.div(real_gdp.iloc[:, 0], axis="index") * 100.0
values.index = output_names
return values
def deviations(
new_context: Any,
original_context: Any,
) -> pd.DataFrame:
return levels(new_context) - levels(original_context)
return DerivationDefinition(
prefix=prefix,
sets=sets,
label="real GDP index",
units="index (first projection year = 100)",
deviation_units="index-point deviation",
required_variable_prefixes=("GDPR",),
levels=levels,
deviations=deviations,
description_markdown="""Regional real GDP index.
Divides each `GDPR(region)` projection by its first projection-year
value and multiplies the result by 100.
""",
)
The two calculation functions return values only: one row for every resolved region and one column for every projection year. The subsystem generates the charting metadata and checks the returned shape.
Use the custom definition explicitly:
from gcubed.derivations import Derivations
derivations = Derivations([REAL_GDP_INDEX()])
Or mix it with a built-in factory:
from gcubed.derivations import Derivations
from gcubed.derivations.definitions import GDPRGROWTH
derivations = Derivations([
GDPRGROWTH(),
REAL_GDP_INDEX(),
])
The custom output rows are named REAL_GDP_INDEX(region), so a chartpack can select an exact row such as REAL_GDP_INDEX(USA) once reporting receives this Derivations collection.
Create a grouped derived variable
A grouped definition combines selected members of one source dimension into a synthetic reporting member. It is useful when an aggregate is meaningful for a particular model but should not be added to that model’s SYM sets or solved vectors.
Grouped definitions are deliberately separate from ordinary definitions:
AggregateGroupowns the synthetic output member, its display description and its complete source-member tuple;GroupedAggregationDefinition.keyidentifies the calculation, whileoutput_prefixidentifies the emitted rows;- the output prefix may match a native or ordinary prefix, provided the complete generated row name does not collide with an existing row; and
- grouped definitions run after all ordinary definitions and can therefore consume either native rows or declared ordinary derived rows.
The package provides reusable grouped factories and deviation policies:
| Helper | Purpose |
|---|---|
grouped_sum(...) | Sum source-member levels into each synthetic group |
grouped_growth_of_sum(...) | Sum levels first, then calculate annual log growth |
grouped_level_difference | Report new grouped levels less original grouped levels; this is the default for grouped_sum(...) |
grouped_percent_change | Report the percentage change from original grouped levels |
grouped_percent_of_group_reference(...) | Scale a grouped level difference by a separately grouped original reference series |
This generic example adds a synthetic real-GDP row for two selected partner economies. It is an example only; the group membership must be chosen and tested for the model that owns it.
from gcubed.derivations import AggregateGroup
from gcubed.derivations import Derivations
from gcubed.derivations import GroupedAggregationDefinition
from gcubed.derivations import create_derivations
from gcubed.derivations import grouped_percent_change
from gcubed.derivations import grouped_sum
def PARTNER_REAL_GDP() -> GroupedAggregationDefinition:
partner_group = AggregateGroup(
key="partner_economies",
output_member="PARTNERS",
description="Selected partner economies",
source_members=("USA", "CHN"),
)
return grouped_sum(
key="GDPR_PARTNERS",
output_prefix="GDPR",
source_prefix="GDPR",
source_sets=("regions",),
grouped_source_set="regions",
output_set="aggregate_regions",
groups=(partner_group,),
label="real GDP - selected partner economies",
units="$US billion",
deviation_units="percent deviation",
deviation=grouped_percent_change,
output_set_description="aggregate reporting regions",
charting_region_set="aggregate_regions",
charting_set_aliases={"aggregate_regions": "regions"},
required_variable_prefixes=("GDPR",),
description_markdown="""Real GDP for selected partner economies.
Sums `GDPR(region)` over the model-local partner membership.
""",
)
ordinary_derivations = create_derivations(["GDPRGROWTH"])
derivations = Derivations(
definitions=ordinary_derivations.definitions,
grouped_definitions=(PARTNER_REAL_GDP(),),
)
This collection emits the ordinary GDPRGROWTH(region) rows and a grouped GDPR(PARTNERS) row. The distinct key GDPR_PARTNERS lets the grouped calculation reuse the native GDPR output prefix. The collection still rejects an exact GDPR(PARTNERS) collision and duplicate grouped keys.
The collection exposes the two stages separately as definitions and grouped_definitions; output_definitions returns both in execution-stage order. create_derivations(...) continues to resolve package-registered ordinary prefixes. The more general create_derivations_from_config(...) also accepts explicit ordinary or grouped objects and no-argument factories, then places each resolved definition in the appropriate stage.
output_set="aggregate_regions" preserves the synthetic member’s provenance in an aggregate_regions metadata column. The explicit charting_set_aliases={"aggregate_regions": "regions"} presents it through the normal regions chart selector, and charting_region_set puts PARTNERS in the canonical charting region column.
If the model declares a set that must exactly define the group, supply its name as AggregateGroup.source_set. Validation then requires the declared set membership, including order, to equal source_members. Without source_set, validation still requires every listed source member to belong to the grouped source dimension.
Make grouped rows available during setup
Model-local grouped definitions are not resolved by a built-in prefix list. Construct the complete Derivations collection in the model’s setup wrapper and pass it to chart_catalogue(...):
from gcubed.experiments import chart_catalogue
catalogue = chart_catalogue(
sym_data,
derivations=derivations,
)
Use that catalogue when generating the setup artefacts, and pass the same derivations collection to the runner and report generation. This keeps setup selectors and runtime reporting on the same definition set. A chartpack can then select the synthetic member through the aliased dimension:
- chart:
title: Real GDP
series:
- variable_prefix: GDPR
selectors:
regions: [PARTNERS]
Unavailable grouped values
Grouped calculations preserve data availability by year. If any required source row is missing, non-numeric or non-finite for a year, the grouped result for that year is NaN; available variables and years continue to be reported. Grouped growth also returns NaN for non-positive levels, and its first year remains NaN. Percentage-change and percent-of-reference deviations return NaN when their original denominator is non-positive or unavailable.
This cell-level propagation is different from structural validation. Duplicate source rows, invalid group membership, duplicate keys, missing declared ordinary dependencies and output-row collisions remain errors.
Make an ordinary definition available by prefix
Keep experiment-specific factories in the experiment code and construct them explicitly. To contribute a generally useful definition to the G-Cubed package:
- add its factory under
gcubed.derivations.definitions; - import the factory in
gcubed.derivations.definitions.__init__; - add its factory name to that module’s
__all__list; and - add unit tests for metadata, validation, levels, deviations and applicable model domains.
The factory must take no arguments and return a valid DerivationDefinition. The built-in registry calls exported ordinary factories and indexes them by the returned definition prefix. Once registered, create_derivations(["REAL_GDP_INDEX"]) can resolve it by name. Keep model-specific and still-experimental ordinary or grouped factories outside the package registry and construct their Derivations collection explicitly.
Definition contract
A definition’s calculation functions receive DerivationContext objects. The context provides:
- model configuration, SYM metadata and parameters;
- a copy of
charting_projections; - the ordered projection-year columns;
- optional lower-level
database_projections; and - cached upstream derived rows for declared dependencies.
Ordinary levels and deviations functions must return a pandas dataframe with one row for each output-domain tuple and every projection-year column. Grouped level functions receive the combined context after ordinary calculations. Grouped deviation policies receive the definition, both contexts and the two already-grouped level frames. Use domain_tuples(...) and variable_names(...) to keep inputs and outputs in the same deterministic order. Let the subsystem attach descriptions, units, set members, region and vector="derived" metadata.
In addition to model-native sets, a definition can declare local_sets, a domain_filter, a charting_region_set, charting_set_aliases and custom validation rules. Aliases change selector names and descriptions; the original set columns remain in the output as provenance. Use these features only when the ordinary model domain and prefix checks are insufficient.
Common failures
| Error | Meaning |
|---|---|
DerivationRegistryError | A configured prefix or factory could not be resolved |
DerivationValidationError | A definition is incompatible with the model or its dependencies |
DerivationFrameError | A projection object or returned level frame violates the dataframe contract |
DerivationDeviationError | Two contexts cannot be compared or a deviation frame is invalid |
DerivationWorkflowError | A runner or reporting adapter received invalid workflow inputs |
Unknown built-in prefixes, duplicate ordinary prefixes, duplicate grouped keys, invalid group memberships, output collisions and missing model inputs fail early. Unavailable numeric cells in an otherwise valid grouped calculation are reported as NaN instead of aborting unrelated output. This distinction keeps structural mistakes strict without turning partial source data into misleading aggregates.
For the shared callable signatures, see the derived-variable Python API.
G-Cubed