Quickstart

This quickstart guide will take you through the steps of developing your very first extractor based on the Cognite extractor-utils framework.

If you want to see other examples of extractors, see our example extractor repository

Starting a new extractor project

The easiest way to set up a new extractor project is to use cogex. To install it, run

pip install cognite-extractor-manager

in a shell. To initialize a new extractor project, run

cogex init

in the directory you want your extractor project in.

Running cogex init will first prompt you for some information about your extractor, and then set up a poetry environment, git repository, commit hooks with type and style checks and load a template.

Developing your first extractor

In this tutorial, we will go through the steps of creating a simple extractor that reads CSV files locally on your machine and uploads them to CDF RAW.

Chapters

• Defining a config schema
• Read CSV files
• Uploading data to CDF
  • Using an upload queue
  • Tying it all together
  • Extraction pipeline runs
• Packaging the final extractor
  • Python is available at production environment
  • Python is not available at production environment