Cognite extractor-utils package

The extractor-utils package is an extension of the Cognite Python SDK intended to simplify the development of data extractors for Cognite Data Fusion.

Installation

To install this package:

pip install cognite-extractor-utils

If the Cognite SDK is not already installed, the installation will automatically fetch and install it as well.

To get going, consult the Quickstart

Contents

• Quickstart
  • Starting a new extractor project
  • Developing your first extractor
    • Defining a config schema
    • Read CSV files
    • Uploading data to CDF
      • Using an upload queue
      • Tying it all together
      • Extraction pipeline runs
    • Packaging the final extractor
      • Python is available at production environment
      • Python is not available at production environment
• Remote configuration
  • Detecting config changes
• Package reference
  • Base class for extractors
    • Extractor
      • Extractor.get_current_config()
      • Extractor.get_current_statestore()
      • Extractor.reload_config_callback()
      • Extractor.run()
  • configtools - Utilities for reading, parsing and validating config files
    • Config loader
      • load_yaml()
      • load_yaml_dict()
      • KeyVaultAuthenticationMethod
      • KeyVaultLoader
    • Base classes
      • BaseConfig
      • CogniteConfig
      • EitherIdConfig
      • ConnectionConfig
      • AuthenticatorConfig
      • CertificateConfig
      • LoggingConfig
      • MetricsConfig
      • RawDestinationConfig
      • StateStoreConfig
      • RawStateStoreConfig
      • LocalStateStoreConfig
      • TimeIntervalConfig
      • FileSizeConfig
      • ConfigType
    • Exceptions
      • InvalidConfigError
  • metrics - Automatic pushers of performance metrics
    • AbstractMetricsPusher
      • AbstractMetricsPusher.start()
      • AbstractMetricsPusher.stop()
    • BaseMetrics
    • CognitePusher
      • CognitePusher.start()
      • CognitePusher.stop()
    • PrometheusPusher
      • PrometheusPusher.clear_gateway()
      • PrometheusPusher.start()
      • PrometheusPusher.stop()
    • safe_get()
  • statestore - Storing extractor state between runs locally or remotely
    • AbstractHashStateStore
      • AbstractHashStateStore.get_state()
      • AbstractHashStateStore.has_changed()
      • AbstractHashStateStore.initialize()
      • AbstractHashStateStore.set_state()
      • AbstractHashStateStore.start()
      • AbstractHashStateStore.stop()
      • AbstractHashStateStore.synchronize()
    • AbstractStateStore
      • AbstractStateStore.delete_state()
      • AbstractStateStore.expand_state()
      • AbstractStateStore.get_state()
      • AbstractStateStore.initialize()
      • AbstractStateStore.outside_state()
      • AbstractStateStore.post_upload_handler()
      • AbstractStateStore.set_state()
      • AbstractStateStore.start()
      • AbstractStateStore.stop()
      • AbstractStateStore.synchronize()
    • LocalHashStateStore
      • LocalHashStateStore.get_state()
      • LocalHashStateStore.has_changed()
      • LocalHashStateStore.initialize()
      • LocalHashStateStore.set_state()
      • LocalHashStateStore.start()
      • LocalHashStateStore.stop()
      • LocalHashStateStore.synchronize()
    • LocalStateStore
      • LocalStateStore.delete_state()
      • LocalStateStore.expand_state()
      • LocalStateStore.get_state()
      • LocalStateStore.initialize()
      • LocalStateStore.outside_state()
      • LocalStateStore.post_upload_handler()
      • LocalStateStore.set_state()
      • LocalStateStore.start()
      • LocalStateStore.stop()
      • LocalStateStore.synchronize()
    • NoStateStore
      • NoStateStore.delete_state()
      • NoStateStore.expand_state()
      • NoStateStore.get_state()
      • NoStateStore.initialize()
      • NoStateStore.outside_state()
      • NoStateStore.post_upload_handler()
      • NoStateStore.set_state()
      • NoStateStore.start()
      • NoStateStore.stop()
      • NoStateStore.synchronize()
    • RawHashStateStore
      • RawHashStateStore.get_state()
      • RawHashStateStore.has_changed()
      • RawHashStateStore.initialize()
      • RawHashStateStore.set_state()
      • RawHashStateStore.start()
      • RawHashStateStore.stop()
      • RawHashStateStore.synchronize()
    • RawStateStore
      • RawStateStore.delete_state()
      • RawStateStore.expand_state()
      • RawStateStore.get_state()
      • RawStateStore.initialize()
      • RawStateStore.outside_state()
      • RawStateStore.post_upload_handler()
      • RawStateStore.set_state()
      • RawStateStore.start()
      • RawStateStore.stop()
      • RawStateStore.synchronize()
  • uploader - Batching upload queues with automatic upload triggers
    • AbstractUploadQueue
      • AbstractUploadQueue.start()
      • AbstractUploadQueue.stop()
      • AbstractUploadQueue.upload()
    • TimestampedObject
      • TimestampedObject.created
      • TimestampedObject.payload
    • AssetUploadQueue
      • AssetUploadQueue.add_to_upload_queue()
      • AssetUploadQueue.upload()
    • EventUploadQueue
      • EventUploadQueue.add_to_upload_queue()
      • EventUploadQueue.upload()
    • BytesUploadQueue
      • BytesUploadQueue.add_to_upload_queue()
    • ChunkedStream
      • ChunkedStream.current_chunk
      • ChunkedStream.next_chunk()
      • ChunkedStream.readable()
      • ChunkedStream.tell()
      • ChunkedStream.writelines()
    • FileUploadQueue
      • FileUploadQueue.add_to_upload_queue()
    • IOFileUploadQueue
      • IOFileUploadQueue.add_io_to_upload_queue()
      • IOFileUploadQueue.upload()
    • RawUploadQueue
      • RawUploadQueue.add_to_upload_queue()
      • RawUploadQueue.upload()
    • SequenceUploadQueue
      • SequenceUploadQueue.add_to_upload_queue()
      • SequenceUploadQueue.set_sequence_column_definition()
      • SequenceUploadQueue.set_sequence_metadata()
      • SequenceUploadQueue.upload()
    • TimeSeriesUploadQueue
      • TimeSeriesUploadQueue.add_to_upload_queue()
      • TimeSeriesUploadQueue.upload()
    • default_time_series_factory()
  • util - Miscellaneous utilities
    • BufferedReadWithLength
      • BufferedReadWithLength.close()
      • BufferedReadWithLength.closed
      • BufferedReadWithLength.detach()
      • BufferedReadWithLength.fileno()
      • BufferedReadWithLength.flush()
      • BufferedReadWithLength.isatty()
      • BufferedReadWithLength.mode
      • BufferedReadWithLength.name
      • BufferedReadWithLength.peek()
      • BufferedReadWithLength.raw
      • BufferedReadWithLength.read()
      • BufferedReadWithLength.read1()
      • BufferedReadWithLength.readable()
      • BufferedReadWithLength.readinto()
      • BufferedReadWithLength.readinto1()
      • BufferedReadWithLength.readline()
      • BufferedReadWithLength.readlines()
      • BufferedReadWithLength.seek()
      • BufferedReadWithLength.seekable()
      • BufferedReadWithLength.tell()
      • BufferedReadWithLength.truncate()
      • BufferedReadWithLength.writable()
      • BufferedReadWithLength.write()
      • BufferedReadWithLength.writelines()
    • EitherId
      • EitherId.content()
      • EitherId.type()
    • add_extraction_pipeline()
    • cognite_exceptions()
    • datetime_to_timestamp()
    • ensure_assets()
    • ensure_time_series()
    • httpx_exceptions()
    • iterable_to_stream()
    • now()
    • requests_exceptions()
    • retry()
    • throttled_loop()
    • timestamp_to_datetime()
    • truncate_byte_len()