Package reference

configtools - Utilities for reading, parsing and validating config files

The configtools module exists of tools for loading and verifying config files for extractors.

Extractor configurasions are conventionally written in hyphen-cased YAML. These are typically loaded and serialized as dataclasses in Python.

Config loader

Base classes

The configtools module contains several prebuilt config classes for many common parameters. The class BaseConfig is intended as a starting point for a custom configuration schema, containing parameters for config version, CDF connection and logging.

Example:

@dataclass
class ExtractorConfig:
    state_store: Optional[StateStoreConfig]
    ...

@dataclass
class SourceConfig:
    ...


@dataclass
class MyConfig(BaseConfig):
    extractor: ExtractorConfig
    source: SourceConfig

Exceptions

metrics - Automatic pushers of performance metrics

statestore - Storing extractor state between runs locally or remotely

uploader - Batching upload queues with automatic upload triggers

util - Miscellaneous utilities