Guide

Contract first or data first?

You can write the data contract before the data exists, or build it from data you have. Here is when each path fits, and what they share.

By Steffen Sande8 min read

When a team decides to use data contracts, a practical question comes up quickly: where do we start? Some want to write the contract before a single table is built. Others want to start with the tables they already have and describe them. Both work, and the guide Adopting Data Contracts describes them as two ways to adopt data contracts: contract first and data first. Its advice is to pick the one that fits the situation.

This article compares the two paths, using the web shop and its customer, order and product datasets as the example. Neither path is better than the other. They suit different starting points, and they end in the same place.

Contract first: start with the requirements

With contract first, you write the contract before the data product exists. The starting point is the requirements of the people who will use the data, not whatever happens to sit in a database today.

Say the web shop’s finance department needs a new dataset: daily revenue per product category, for a report they send to management every month. Nobody has built it yet. Before the data platform team writes any code, they sit down with finance and agree on what the dataset should contain. One row per day and category. The amount excludes sales tax. The data is ready before the working day starts.

That agreement is written down as a contract. In the Open Data Contract Standard, a first draft might look like this:

apiVersion: v3.2.0
kind: DataContract
id: daily-revenue
version: 0.1.0
status: draft
description:
  purpose: Daily revenue per product category for the finance monthly report.
schema:
  - name: daily_revenue
    properties:
      - name: date
        logicalType: date
        required: true
      - name: product_category
        logicalType: string
        required: true
      - name: amount_excl_tax
        logicalType: number
        required: true

The contract has the status draft, because it is not yet a promise anyone has made. The version is 0.1.0 for the same reason. Once producer and consumer agree, the status is set to active and the version is bumped.

From here the work follows a fixed order. The contract is checked against the standard, for example with datacontract lint from the Data Contract CLI. Then the data platform team builds the dataset to the contract, while finance can start on their report at the same time, because they know what they will get. The contract can also generate starter code for both sides, for example dbt models for the producer and dbt sources for the consumer. Finally, what was built is tested against the contract.

When contract first fits

Contract first fits best when three things are true:

  • The data product is new. There is no table to describe, so the contract is the first thing that can be written down.
  • The consumers are known. Someone can say what they need. Without consumers to ask, the requirements are guesswork, and so is the contract.
  • Agreeing first is worth it. Changes are cheapest before anything is built. Renaming a column in a draft costs one line. Doing the same after three reports use the column costs a contract change with notice and a transition period.

The downside is that it takes a while before anything is in production. The contract has to be written, discussed and approved before the first row is delivered. And even a good contract can meet reality badly: the source system may not have a category on every order, or the amount excluding tax may need to be calculated in a way nobody thought of. Then the contract has to be adjusted while it is still a draft, which is exactly what the draft is for.

Data first: start with what exists

With data first, you build the contract from data that already exists. The starting point is the actual schema, not the requirements.

In the web shop, customer, order and product have existed for years. They have consumers, reports and integrations built on them, but no contract. Starting from a blank page makes little sense here. The tables exist, the columns have names and types, and those are what consumers actually use today.

The first step is therefore to bring the schema in, for example with the Data Contract CLI. It can import a schema from sources such as SQL DDL, BigQuery, AWS Glue and Avro, and the documentation shows how to create a contract straight from an existing table. The result is a contract with tables, columns and types, but without what the database does not hold: what the columns mean, who owns them, and what has been promised.

That is the real work in data first. The team goes through the contract and adds what is missing:

  • Descriptions. Is amount in order with or without sales tax? The schema does not say, but someone knows.
  • Owner. Who answers when something is wrong with customer?
  • Quality rules. One rule at a time, for example that order_id is unique, with a check for each new rule that the data still meets it.
  • Status. Which tables are in use, and which are on their way out?

The guide Adopting Data Contracts describes data first as the fastest way to get started and get feedback from consumers. That makes sense: the contract describes something that already works, and consumers can see right away whether it matches what they use.

When data first fits

Data first fits best when the sources already exist and are in use. If you run a data platform in production, that probably applies to many of its tables.

The downside is that, at first, the contract only describes what is, not what is wanted. If order has a column with an unclear name or the wrong type, it comes along into the contract. That is not wrong, because the contract should describe what consumers actually build on. But it means an import is a starting point, not a finished contract. Without descriptions, an owner and rules, it is just the schema, written down in a new format.

Both end with a contract in Git

The difference between the two paths is where you start. Where you end up is the same: a contract that is valid against the standard, that states what the producer promises, and that lives in version control alongside the code.

From that point on, there is no difference between a contract that was written first and one that was imported. Both change through the same process. A change is proposed, reviewed and approved in Git, and tools can show what changed between two versions of the contract, for example with datacontract changelog from the Data Contract CLI. That lets reviewers see whether the change touches something consumers depend on. Both are tested against the data in the same way, and both can show the history of what was promised when.

Nothing stops you from using both paths at once. The existing tables get contracts through data first, and new data products are written contract first. In the web shop, customer, order and product can be imported and described, while the new daily revenue dataset is written as a draft before anything is built. The new dataset builds on the old ones, so you end up with contracts that fit together.

How to choose

The question is not which path is right, but which one fits the dataset in front of you. Two questions settle most of it:

  • Does the data already exist? If so, start with data first. It is faster, and it describes what consumers actually use.
  • Do you know who will use the data, and what they need? If the data does not exist yet and you have consumers to ask, start with contract first. It gives you an agreement before anything is built.

If you have neither, it is worth finding the consumers before you write the contract. Otherwise you are writing down your own assumptions and calling them an agreement.

Whichever path you choose, the first step is the same: one dataset. Pick an existing table that many people use, or a new dataset with a consumer waiting for it, and write the contract for it. How a contract is structured, and which fields the standard has, is covered in the article on the Open Data Contract Standard.

Sources