Cognite extractor-utils package

The extractor-utils package is an extension of the Cognite Python SDK intended to simplify the development of data extractors for Cognite Data Fusion.

Installation

To install this package:

pip install cognite-extractor-utils

If the Cognite SDK is not already installed, the installation will automatically fetch and install it as well.

To get going, consult the Quickstart

Contents

• Quickstart
  • Starting a new extractor project
  • Developing your first extractor
    • Defining a config schema
    • Read CSV files
    • Uploading data to CDF
      • Using an upload queue
      • Tying it all together
      • Extraction pipeline runs
    • Packaging the final extractor
      • Python is available at production environment
      • Python is not available at production environment
• Remote configuration
  • Detecting config changes
• Package reference
  • Base class for extractors
    • Extractor
      • Extractor.get_current_config()
      • Extractor.get_current_statestore()
      • Extractor.reload_config_callback()
      • Extractor.run()
  • configtools - Utilities for reading, parsing and validating config files
    • Config loader
      • load_yaml()
    • Base classes
      • BaseConfig
      • CogniteConfig
      • LoggingConfig
      • MetricsConfig
      • RawDestinationConfig
      • StateStoreConfig
      • RawStateStoreConfig
      • LocalStateStoreConfig
    • Exceptions
  • throttle - Tools for throttling
  • metrics - Automatic pushers of performance metrics
    • AbstractMetricsPusher
      • AbstractMetricsPusher.start()
      • AbstractMetricsPusher.stop()
    • BaseMetrics
    • CognitePusher
      • CognitePusher.start()
      • CognitePusher.stop()
    • PrometheusPusher
      • PrometheusPusher.clear_gateway()
      • PrometheusPusher.start()
      • PrometheusPusher.stop()
    • safe_get()
  • statestore - Storing extractor state between runs locally or remotely
    • AbstractStateStore
      • AbstractStateStore.delete_state()
      • AbstractStateStore.expand_state()
      • AbstractStateStore.get_state()
      • AbstractStateStore.initialize()
      • AbstractStateStore.outside_state()
      • AbstractStateStore.post_upload_handler()
      • AbstractStateStore.set_state()
      • AbstractStateStore.start()
      • AbstractStateStore.stop()
      • AbstractStateStore.synchronize()
    • LocalStateStore
      • LocalStateStore.delete_state()
      • LocalStateStore.expand_state()
      • LocalStateStore.get_state()
      • LocalStateStore.initialize()
      • LocalStateStore.outside_state()
      • LocalStateStore.post_upload_handler()
      • LocalStateStore.set_state()
      • LocalStateStore.start()
      • LocalStateStore.stop()
      • LocalStateStore.synchronize()
    • NoStateStore
      • NoStateStore.delete_state()
      • NoStateStore.expand_state()
      • NoStateStore.get_state()
      • NoStateStore.initialize()
      • NoStateStore.outside_state()
      • NoStateStore.post_upload_handler()
      • NoStateStore.set_state()
      • NoStateStore.start()
      • NoStateStore.stop()
      • NoStateStore.synchronize()
    • RawStateStore
      • RawStateStore.delete_state()
      • RawStateStore.expand_state()
      • RawStateStore.get_state()
      • RawStateStore.initialize()
      • RawStateStore.outside_state()
      • RawStateStore.post_upload_handler()
      • RawStateStore.set_state()
      • RawStateStore.start()
      • RawStateStore.stop()
      • RawStateStore.synchronize()
  • uploader - Batching upload queues with automatic upload triggers
  • util - Miscellaneous utilities
    • EitherId
      • EitherId.content()
      • EitherId.type()
    • add_extraction_pipeline()
    • ensure_assets()
    • ensure_time_series()
    • retry()
    • set_event_on_interrupt()
    • throttled_loop()