Cognite extractor-utils package

The extractor-utils package is an extension of the Cognite Python SDK intended to simplify the development of data extractors for Cognite Data Fusion.

Installation

To install this package:

pip install cognite-extractor-utils

If the Cognite SDK is not already installed, the installation will automatically fetch and install it as well.

To get going, consult the Quickstart

Contents

• Quickstart
  • Starting a new extractor project
  • Developing your first extractor
    • Defining a config schema
    • Read CSV files
    • Uploading data to CDF
      • Using an upload queue
      • Tying it all together
      • Extraction pipeline runs
    • Packaging the final extractor
      • Python is available at production environment
      • Python is not available at production environment
• Remote configuration
  • Detecting config changes
• Package reference
  • Base class for extractors
    • Extractor
      • Extractor.get_current_config()
      • Extractor.get_current_statestore()
      • Extractor.reload_config_callback()
      • Extractor.run()
  • configtools - Utilities for reading, parsing and validating config files
    • Config loader
      • load_yaml()
      • load_yaml_dict()
      • KeyVaultAuthenticationMethod
      • KeyVaultLoader
    • Base classes
      • BaseConfig
      • CogniteConfig
      • EitherIdConfig
      • ConnectionConfig
      • AuthenticatorConfig
      • CertificateConfig
      • LoggingConfig
      • MetricsConfig
      • RawDestinationConfig
      • StateStoreConfig
      • RawStateStoreConfig
      • LocalStateStoreConfig
      • TimeIntervalConfig
      • FileSizeConfig
      • ConfigType
    • Exceptions
      • InvalidConfigError
  • metrics - Automatic pushers of performance metrics
    • AbstractMetricsPusher
      • AbstractMetricsPusher.start()
      • AbstractMetricsPusher.stop()
    • BaseMetrics
    • CognitePusher
      • CognitePusher.start()
      • CognitePusher.stop()
    • PrometheusPusher
      • PrometheusPusher.clear_gateway()
      • PrometheusPusher.start()
      • PrometheusPusher.stop()
    • safe_get()
  • statestore - Storing extractor state between runs locally or remotely
    • AbstractStateStore
      • AbstractStateStore.delete_state()
      • AbstractStateStore.expand_state()
      • AbstractStateStore.get_state()
      • AbstractStateStore.initialize()
      • AbstractStateStore.outside_state()
      • AbstractStateStore.post_upload_handler()
      • AbstractStateStore.set_state()
      • AbstractStateStore.start()
      • AbstractStateStore.stop()
      • AbstractStateStore.synchronize()
    • LocalStateStore
      • LocalStateStore.delete_state()
      • LocalStateStore.expand_state()
      • LocalStateStore.get_state()
      • LocalStateStore.initialize()
      • LocalStateStore.outside_state()
      • LocalStateStore.post_upload_handler()
      • LocalStateStore.set_state()
      • LocalStateStore.start()
      • LocalStateStore.stop()
      • LocalStateStore.synchronize()
    • NoStateStore
      • NoStateStore.delete_state()
      • NoStateStore.expand_state()
      • NoStateStore.get_state()
      • NoStateStore.initialize()
      • NoStateStore.outside_state()
      • NoStateStore.post_upload_handler()
      • NoStateStore.set_state()
      • NoStateStore.start()
      • NoStateStore.stop()
      • NoStateStore.synchronize()
    • RawStateStore
      • RawStateStore.delete_state()
      • RawStateStore.expand_state()
      • RawStateStore.get_state()
      • RawStateStore.initialize()
      • RawStateStore.outside_state()
      • RawStateStore.post_upload_handler()
      • RawStateStore.set_state()
      • RawStateStore.start()
      • RawStateStore.stop()
      • RawStateStore.synchronize()
  • uploader - Batching upload queues with automatic upload triggers
    • AbstractUploadQueue
      • AbstractUploadQueue.start()
      • AbstractUploadQueue.stop()
      • AbstractUploadQueue.upload()
    • TimestampedObject
      • TimestampedObject.created
      • TimestampedObject.payload
    • AssetUploadQueue
      • AssetUploadQueue.add_to_upload_queue()
      • AssetUploadQueue.upload()
    • EventUploadQueue
      • EventUploadQueue.add_to_upload_queue()
      • EventUploadQueue.upload()
    • BytesUploadQueue
      • BytesUploadQueue.add_to_upload_queue()
    • ChunkedStream
      • ChunkedStream.current_chunk
      • ChunkedStream.next_chunk()
      • ChunkedStream.readable()
      • ChunkedStream.tell()
      • ChunkedStream.writelines()
    • FileUploadQueue
      • FileUploadQueue.add_to_upload_queue()
    • IOFileUploadQueue
      • IOFileUploadQueue.add_io_to_upload_queue()
      • IOFileUploadQueue.upload()
    • RawUploadQueue
      • RawUploadQueue.add_to_upload_queue()
      • RawUploadQueue.upload()
    • SequenceUploadQueue
      • SequenceUploadQueue.add_to_upload_queue()
      • SequenceUploadQueue.set_sequence_column_definition()
      • SequenceUploadQueue.set_sequence_metadata()
      • SequenceUploadQueue.upload()
    • TimeSeriesUploadQueue
      • TimeSeriesUploadQueue.add_to_upload_queue()
      • TimeSeriesUploadQueue.upload()
    • default_time_series_factory()
  • util - Miscellaneous utilities
    • EitherId
      • EitherId.content()
      • EitherId.type()
    • add_extraction_pipeline()
    • cognite_exceptions()
    • ensure_assets()
    • ensure_time_series()
    • httpx_exceptions()
    • requests_exceptions()
    • retry()
    • throttled_loop()