Explainer

Open Data Contract Standard (ODCS) explained

ODCS is an open standard for data contracts in YAML. Who is behind it, which sections a contract has, and which tools can read it.

By Steffen Sande9 min read

A data contract describes what a producer of data promises to deliver: schema, meaning, owner, quality and service levels. You can write such a contract in a format of your own, but then every tool and every team has to learn your particular format. The Open Data Contract Standard, ODCS for short, is a shared format for the same content.

This article explains what ODCS is, how a contract in the standard is put together, and what you gain from using an open standard rather than a format of your own.

What ODCS is

ODCS is an open, vendor-neutral standard for data contracts. It is maintained by Bitol, an open project under Linux Foundation AI & Data, and is licensed under Apache 2.0. Bitol is also behind a sister standard for data products, the Open Data Product Standard (ODPS).

The standard is developed in the open on GitHub, where changes are proposed and discussed as RFCs before they are adopted. The current version is v3.2.0. The version is written in the contract itself, in the apiVersion field, so that a tool knows which edition of the standard the file follows.

An ODCS contract is a single YAML file. Its structure is described by an official JSON Schema for each version, which means a machine can check whether a file is a valid contract before anyone reads it. ODCS is one of the most widely used standards for data contracts today.

The sections of a contract

An ODCS contract is divided into sections with fixed names. Only four fields are required: apiVersion, kind, id and version. You add the rest as you need it, and a small contract can easily start with a schema and an owner and grow from there.

Fundamentals

At the top of the file are the fields that identify the contract. apiVersion gives the version of the standard, and kind is always DataContract. id is a stable identifier for the contract, while version is the version of the contract itself, which you bump when its content changes. status says where the contract is in its lifecycle, for example draft, active or deprecated.

This is also where description belongs, with subfields for purpose (purpose), intended use (usage) and known limitations (limitations). These are the fields a new consumer reads first.

Schema

The schema section describes the structure of the data. It holds a list of objects, typically tables, and each object has a list of properties (properties), typically columns.

ODCS distinguishes between logical and physical types. The logical type (logicalType) is a portable description such as string, integer, number, date or timestamp. The physical type (physicalType) is what the database actually uses, for example varchar or uuid. The distinction lets the same contract describe data that lives in different systems.

For each property you can state, among other things, whether it is required (required), whether it is part of the primary key (primaryKey), a description, and a classification (classification) of how sensitive the data is. Relationships between tables can be described here too.

Quality

Quality rules go in quality, either on a whole object or on a single property. The standard has four kinds of rule:

  • Text (text): a rule described in words, for people.
  • Library (library): predefined metrics such as nullValues, duplicateValues and rowCount, combined with a threshold, for example that the number of duplicates must be 0.
  • SQL (sql): a query that returns a value, and a range the value must stay within.
  • Custom (custom): rules written for a specific data quality tool.

Library rules are the easiest to move between tools, because their meaning is defined by the standard rather than by any one tool.

Service levels

Expectations for the delivery live in slaProperties. Each entry is a property with a value and optionally a unit, for example how fresh the data must be, how often it is updated, and how long it is retained.

Team and roles

team says who owns and maintains the dataset, with a list of members and their roles. roles describes access: which roles exist for reading or changing the data, and who has to approve an access request.

Servers

servers describes where the data actually lives: the type of system, for example postgres, and the connection details needed to reach it. A contract can list several servers, for example one for test and one for production. This is the section that lets a tool connect and check that the data matches the contract.

Support, price and extensions

Some sections are less central, but useful when a contract is shared widely. support lists channels where consumers can get help, such as a chat channel or an issue tracker. price describes what the data costs, where that applies. tags provides keywords for search and grouping.

authoritativeDefinitions points from the contract or a property to a definition somewhere else, for example a term in a business glossary. customProperties is the standard’s opening for fields it does not define itself, so you can add your own information without breaking the format.

An example: the customer contract

The example below is a small contract for the customer dataset in an online shop. Where the article on data contracts showed the schema and quality rules for order, this one focuses on the sections around the schema: description, server, owner and service levels. The file is valid against the official JSON Schema for ODCS v3.2.0.

apiVersion: v3.2.0
kind: DataContract
id: customer
version: 2.1.0
status: active
description:
  purpose: One row per registered customer of the online shop.
servers:
  - server: production
    type: postgres
    host: db.example.com
    port: 5432
    database: shop
    schema: sales
team:
  name: Customer team
  members:
    - username: customer-team@example.com
      role: Owner
schema:
  - name: customer
    properties:
      - name: customer_id
        logicalType: string
        primaryKey: true
        required: true
      - name: email
        logicalType: string
        classification: confidential
slaProperties:
  - property: latency
    value: 1
    unit: d

The contract is at version 2.1.0 and is active. The data lives in the sales schema of a Postgres database, and the customer team is the owner. The customer table has a required primary key, customer_id, and an email address classified as confidential. The service level says the data must be at most one day old.

Notice what the contract leaves out. It has no quality rules and no description of each column yet. That is perfectly fine: the contract is valid, and it can be extended in a new version once the producer and consumers agree on more.

Why an open standard means less lock-in

When your contracts follow a format only one tool understands, the contracts are in practice part of that tool. If you want to switch tools, you have to convert every contract, and something is often lost along the way. That makes switching expensive, even when another tool would suit you better.

An open standard turns this around. The contract is a file you own, with a structure that is described publicly and maintained by a neutral foundation, not by a vendor. That has some concrete consequences:

  • You can switch tools without rewriting your contracts. A tool that reads ODCS can read the contracts you already have.
  • Several tools can use the same files. One tool can be used to write contracts, another to test the data against them, and a third to show them in a catalog.
  • Contracts can live in version control. A YAML file can be reviewed, diffed and tracked in Git like any other code, regardless of which tool created it.
  • Changes to the format happen in the open. New versions of the standard are discussed publicly, and you can follow, or contribute to, what comes next.

A standard does not remove all the work of switching tools. Extensions in customProperties may be written for one particular tool, and two tools may interpret parts of the standard differently. But the core of the contract, schema, owner, quality and service levels, sits in a format that belongs to no single vendor.

Tools that support ODCS

One open tool with built-in support for ODCS is the Data Contract CLI, a command-line tool that uses ODCS as its own contract format. Among other things, it can:

  • create a new contract with datacontract init
  • check that a file follows the standard’s JSON Schema with datacontract lint
  • connect to the servers in the contract and test that the data meets it with datacontract test
  • show the differences between two versions of a contract with datacontract changelog
  • import a contract from an existing system and export the contract to other formats

The Data Contract CLI validates contracts written for v3.2.0, as well as the older versions v3.1.0 and v3.0.2. Its documentation also mentions the Data Contract Editor, an editor for writing contracts.

If you are evaluating a tool, it is worth checking which version of ODCS it supports. If a tool only supports an older version, fields that are new in v3.2.0 may not be understood.

Where to start

The easiest way to get to know ODCS is to write a contract for one dataset you know well, as the example above does for customer. Start with the required fields, the schema and the owner, and check the file against the JSON Schema, for example with datacontract lint. Once it is valid, you can add quality rules and service levels one section at a time.

The official documentation and the JSON Schemas for each version are on the ODCS GitHub page, and news about the standard is published at bitol.io.

Sources