Usage: pycmor Configuration

This module defines the configuration hierarchy for the pycmor application, using everett’s ~everett.manager.ConfigManager. The configuration hierarchy is as follows (lowest to highest priority):

1. Hardcoded defaults

2. User configuration file

3. Run-specific configuration

4. Environment variables

5. Command-line switches

The configuration hierarchy is defined in the from_pycmor_cfg class method, and cannot be modified outside the class. You should initialize a PycmorConfigManager object (probably in your CMORizer) and grab config values from it by calling with the config key as an argument.

User Configuration File

You can define global configuration options in a user configuration file. The files found at these locations will be used, in highest to lowest priority order:

1. ${PYCMOR_CONFIG_FILE}

2. ${XDG_CONFIG_HOME}/pycmor.yaml

3. ${XDG_CONFIG_HOME}/pycmor/pycmor.yaml

4. ~/.pycmor.yaml

Note that the ${XDG_CONFIG_HOME} environment variable defaults to ~/.config if it is not set.

Configuration Options

You can configure the following:

component pycmor.core.config.PycmorConfig

Configuration summary:

Setting	Parser	Required?
PYCMOR_DASK_CLUSTER	<ChoiceOf(str, [‘local’, ‘slurm’])>
PYCMOR_DASK_CLUSTER_SCALING_FIXED_JOBS	int
PYCMOR_DASK_CLUSTER_SCALING_MAXIMUM_JOBS	int
PYCMOR_DASK_CLUSTER_SCALING_MINIMUM_JOBS	int
PYCMOR_DASK_CLUSTER_SCALING_MODE	<ChoiceOf(str, [‘adapt’, ‘fixed’])>
PYCMOR_DIMENSIONLESS_MAPPING_TABLE	str
PYCMOR_ENABLE_DASK	pycmor.core.config._parse_bool
PYCMOR_ENABLE_FLOX	pycmor.core.config._parse_bool
PYCMOR_ENABLE_OUTPUT_SUBDIRS	pycmor.core.config._parse_bool
PYCMOR_FILE_TIMESPAN	str
PYCMOR_PARALLEL	pycmor.core.config._parse_bool
PYCMOR_PARALLEL_BACKEND	str
PYCMOR_PIPELINE_WORKFLOW_ORCHESTRATOR	<ChoiceOf(str, [‘native’, ‘prefect’])>
PYCMOR_PREFECT_TASK_RUNNER	<ChoiceOf(str, [‘thread_pool’, ‘dask’])>
PYCMOR_QUIET	pycmor.core.config._parse_bool
PYCMOR_RAISE_ON_NO_RULE	pycmor.core.config._parse_bool
PYCMOR_WARN_ON_NO_RULE	pycmor.core.config._parse_bool
PYCMOR_XARRAY_DEFAULT_MISSING_VALUE	float
PYCMOR_XARRAY_ENGINE	<ChoiceOf(str, [‘netcdf4’, ‘h5netcdf’, ‘zarr’])>
PYCMOR_XARRAY_SKIP_UNIT_ATTR_FROM_DRV	pycmor.core.config._parse_bool
PYCMOR_XARRAY_TIME_DTYPE	<ChoiceOf(str, [‘float64’, ‘datetime64[ns]’])>
PYCMOR_XARRAY_TIME_ENABLE_SET_AXIS	pycmor.core.config._parse_bool
PYCMOR_XARRAY_TIME_REMOVE_FILL_VALUE_ATTR	pycmor.core.config._parse_bool
PYCMOR_XARRAY_TIME_SET_LONG_NAME	pycmor.core.config._parse_bool
PYCMOR_XARRAY_TIME_SET_STANDARD_NAME	pycmor.core.config._parse_bool
PYCMOR_XARRAY_TIME_TAXIS_STR	str
PYCMOR_XARRAY_TIME_UNLIMITED	pycmor.core.config._parse_bool

Configuration options:

PYCMOR_DASK_CLUSTER

Parser:

<ChoiceOf(str, [‘local’, ‘slurm’])>

Default:

“local”

Required:

No

Dask cluster to use. See: https://docs.dask.org/en/stable/deploying.html

PYCMOR_DASK_CLUSTER_SCALING_FIXED_JOBS

Parser:

int

Default:

“5”

Required:

No

Number of jobs to create for Jobqueue-backed Dask Cluster

PYCMOR_DASK_CLUSTER_SCALING_MAXIMUM_JOBS

Parser:

int

Default:

“10”

Required:

No

Maximum number of jobs to create for Jobqueue-backed Dask Clusters (adaptive)

PYCMOR_DASK_CLUSTER_SCALING_MINIMUM_JOBS

Parser:

int

Default:

“1”

Required:

No

Minimum number of jobs to create for Jobqueue-backed Dask Clusters (adaptive)

PYCMOR_DASK_CLUSTER_SCALING_MODE

Parser:

<ChoiceOf(str, [‘adapt’, ‘fixed’])>

Default:

“adapt”

Required:

No

Flexible dask cluster scaling

PYCMOR_DIMENSIONLESS_MAPPING_TABLE

Parser:

str

Default:

“/home/docs/checkouts/readthedocs.org/user_builds/pymor/checkouts/stable/src/pycmor/data/dimensionless_mappings.yaml”

Required:

No

Where the dimensionless unit mapping table is defined.

PYCMOR_ENABLE_DASK

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to enable Dask-based processing

PYCMOR_ENABLE_FLOX

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to enable flox for group-by operation. See: https://flox.readthedocs.io/en/latest/

PYCMOR_ENABLE_OUTPUT_SUBDIRS

Parser:

pycmor.core.config._parse_bool

Default:

“no”

Required:

No

Whether to create subdirectories under output_dir when saving data-sets.

PYCMOR_FILE_TIMESPAN

Parser:

str

Default:

“1YS”

Required:

No

Default timespan for grouping output files together.

Use the special flag 'file_native' to use the same grouping as in the input files. Otherwise, use a pandas-flavoured string, see: https://tinyurl.com/38wxf8px

PYCMOR_PARALLEL

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to run in parallel.

PYCMOR_PARALLEL_BACKEND

Parser:

str

Default:

“dask”

Required:

No

Which parallel backend to use.

PYCMOR_PIPELINE_WORKFLOW_ORCHESTRATOR

Parser:

<ChoiceOf(str, [‘native’, ‘prefect’])>

Default:

“prefect”

Required:

No

Which workflow orchestrator to use for running pipelines

PYCMOR_PREFECT_TASK_RUNNER

Parser:

<ChoiceOf(str, [‘thread_pool’, ‘dask’])>

Default:

“thread_pool”

Required:

No

Which runner to use for Prefect flows.

PYCMOR_QUIET

Parser:

pycmor.core.config._parse_bool

Default:

“False”

Required:

No

Whether to suppress output.

PYCMOR_RAISE_ON_NO_RULE

Parser:

pycmor.core.config._parse_bool

Default:

“no”

Required:

No

Whether or not to raise an error if no rule is found for every single DataRequestVariable

PYCMOR_WARN_ON_NO_RULE

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether or not to issue a warning if no rule is found for every single DataRequestVariable

PYCMOR_XARRAY_DEFAULT_MISSING_VALUE

Parser:

float

Default:

“1e+30”

Required:

No

Which missing value to use for xarray. Default is 1e30.

PYCMOR_XARRAY_ENGINE

Parser:

<ChoiceOf(str, [‘netcdf4’, ‘h5netcdf’, ‘zarr’])>

Default:

“netcdf4”

Required:

No

Which engine to use for xarray.

PYCMOR_XARRAY_SKIP_UNIT_ATTR_FROM_DRV

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to skip setting the unit attribute from the DataRequestVariable, this can be handled via Pint

PYCMOR_XARRAY_TIME_DTYPE

Parser:

<ChoiceOf(str, [‘float64’, ‘datetime64[ns]’])>

Default:

“float64”

Required:

No

The dtype to use for time axis in xarray.

PYCMOR_XARRAY_TIME_ENABLE_SET_AXIS

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to enable setting the axis for the time axis in xarray.

PYCMOR_XARRAY_TIME_REMOVE_FILL_VALUE_ATTR

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to remove the fill_value attribute from the time axis in xarray.

PYCMOR_XARRAY_TIME_SET_LONG_NAME

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to set the long name for the time axis in xarray.

PYCMOR_XARRAY_TIME_SET_STANDARD_NAME

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether to set the standard name for the time axis in xarray.

PYCMOR_XARRAY_TIME_TAXIS_STR

Parser:

str

Default:

“T”

Required:

No

Which axis to set for the time axis in xarray.

PYCMOR_XARRAY_TIME_UNLIMITED

Parser:

pycmor.core.config._parse_bool

Default:

“yes”

Required:

No

Whether the time axis is unlimited in xarray.

Usage

Here are some examples of how to use the configuration manager:

>>> pycmor_cfg = {}
>>> config = PycmorConfigManager.from_pycmor_cfg(pycmor_cfg)

>>> engine = config("xarray_engine")
>>> print(f"Using xarray backend: {engine}")
Using xarray backend: netcdf4

>>> parallel = config("parallel")
>>> print(f"Running in parallel: {parallel}")
Running in parallel: True

You can define a user file at ${XDG_CONFIG_DIR}/pycmor/pycmor.yaml:

>>> import pathlib
>>> import yaml
>>> cfg_file = pathlib.Path("~/.config/pycmor/pycmor.yaml").expanduser()
>>> cfg_file.parent.mkdir(parents=True, exist_ok=True)
>>> cfg_to_dump = {"xarray_engine": "zarr"}
>>> with open(cfg_file, "w") as f:
...     yaml.dump(cfg_to_dump, f)
>>> config = PycmorConfigManager.from_pycmor_cfg()
>>> engine = config("xarray_engine")
>>> print(f"Using xarray backend: {engine}")
Using xarray backend: zarr

See also

-