Cognite extractor-utils package
The extractor-utils package is an extension of the Cognite Python SDK intended to simplify the development of data extractors for Cognite Data Fusion.
Installation
To install this package:
pip install cognite-extractor-utils
If the Cognite SDK is not already installed, the installation will automatically fetch and install it as well.
To get going, consult the Quickstart
Contents
- Quickstart
- Remote configuration
- Package reference
- Base class for extractors
configtools- Utilities for reading, parsing and validating config filesmetrics- Automatic pushers of performance metricsstatestore- Storing extractor state between runs locally or remotelyAbstractHashStateStoreAbstractStateStoreAbstractStateStore.delete_state()AbstractStateStore.expand_state()AbstractStateStore.get_state()AbstractStateStore.initialize()AbstractStateStore.outside_state()AbstractStateStore.post_upload_handler()AbstractStateStore.set_state()AbstractStateStore.start()AbstractStateStore.stop()AbstractStateStore.synchronize()
LocalHashStateStoreLocalStateStoreNoStateStoreRawHashStateStoreRawStateStore
uploader- Batching upload queues with automatic upload triggersAssetUploadQueueBytesUploadQueueBytesUploadQueue.add_entry_failure_logger()BytesUploadQueue.add_io_to_upload_queue()BytesUploadQueue.add_to_upload_queue()BytesUploadQueue.flush_failure_logger()BytesUploadQueue.get_failure_logger()BytesUploadQueue.initialize_failure_logging()BytesUploadQueue.start()BytesUploadQueue.stop()BytesUploadQueue.upload()
CDMTimeSeriesUploadQueueEventUploadQueueFileUploadQueueFileUploadQueue.add_entry_failure_logger()FileUploadQueue.add_io_to_upload_queue()FileUploadQueue.add_to_upload_queue()FileUploadQueue.flush_failure_logger()FileUploadQueue.get_failure_logger()FileUploadQueue.initialize_failure_logging()FileUploadQueue.start()FileUploadQueue.stop()FileUploadQueue.upload()
IOFileUploadQueueRawUploadQueueSequenceUploadQueueTimeSeriesUploadQueuedefault_time_series_factory()AbstractUploadQueueTimestampedObjectAssetUploadQueueEventUploadQueueBytesUploadQueueChunkedStreamFileUploadQueueIOByteStreamIOFileUploadQueueRawUploadQueueBaseTimeSeriesUploadQueueCDMTimeSeriesUploadQueueCdmDatapointsPayloadSequenceUploadQueueTimeSeriesUploadQueuedefault_time_series_factory()
util- Miscellaneous utilitiesBufferedReadWithLengthBufferedReadWithLength.close()BufferedReadWithLength.closedBufferedReadWithLength.detach()BufferedReadWithLength.fileno()BufferedReadWithLength.flush()BufferedReadWithLength.isatty()BufferedReadWithLength.modeBufferedReadWithLength.nameBufferedReadWithLength.peek()BufferedReadWithLength.rawBufferedReadWithLength.read()BufferedReadWithLength.read1()BufferedReadWithLength.readable()BufferedReadWithLength.readinto()BufferedReadWithLength.readinto1()BufferedReadWithLength.readline()BufferedReadWithLength.readlines()BufferedReadWithLength.seek()BufferedReadWithLength.seekable()BufferedReadWithLength.tell()BufferedReadWithLength.truncate()BufferedReadWithLength.writable()BufferedReadWithLength.write()BufferedReadWithLength.writelines()
EitherIdadd_extraction_pipeline()cognite_exceptions()datetime_to_timestamp()ensure_assets()ensure_time_series()httpx_exceptions()iterable_to_stream()now()requests_exceptions()retry()throttled_loop()timestamp_to_datetime()truncate_byte_len()