The rules a review checks

The published half of our method. Every rule in the current check suite has a page here. Each runs in every review that has the artefact it needs, and we name the rule behind each finding. How a review is assessed covers scoring, severity and confidence.

Can someone tell what a model, source or column means without reading its SQL?

• dbt models with no description
• dbt columns with no description
• dbt models that declare no columns
• dbt sources with no description
• dbt source groups with no description
• Warehouse columns missing from dbt YAML
• dbt metrics with no description
• Power BI measures with no description
• dbt models with several dates and no default
• Undocumented columns on dbt models behind dashboards

Does the repository run CI, carry tests, a README and an owner file, pin its packages, and keep credentials out?

• Credential files committed to a data repository
• Data repository with no CI pipeline
• Archived repositories under review
• Data repository with no test files
• Data repository with no README
• No CODEOWNERS file in a data repository
• Public data repository with no licence
• Data repository with no recent pushes
• dbt packages.yml with no package-lock.yml
• dbt_project.yml with no require-dbt-version
• Passwords and keys written into SQL files
• Admin-level grants in Terraform
• Public buckets and datasets in Terraform
• Secrets written into Terraform files

Is every model tested, and does a test confirm one row per key?

• dbt models with no tests
• dbt model grain: no uniqueness test
• dbt key columns: no not_null test
• SQL tables with no assertion
• Dagster assets with no asset check
• OpenLineage datasets with no quality assertion

Is each source checked for freshness, and was it fresh when last checked?

• dbt sources with no freshness check
• dbt freshness checks with no loaded_at_field
• dbt sources past their freshness threshold
• dbt freshness thresholds that are never run

Does every model read through ref() and source(), with no hard-coded tables and no cycles?

• dbt dependency cycles in the model DAG
• dbt models with a hardcoded table name
• dbt leaf models with no declared exposure
• Deep dbt lineage: long chains of models
• dbt models with too many direct dependents
• dbt models with no ref() or source()
• Large dbt sources that nothing references
• Large dbt models with no declared reader
• Circular dependencies between SQL files
• SQL tables written by more than one file
• Long chains of SQL tables built on tables
• Datasets written by more than one OpenLineage job

Do incremental models, contracts and materialisations match what the warehouse actually holds?

• dbt incremental models with no unique_key
• dbt public models with no enforced contract
• dbt sources declared twice for one table
• dbt staging models that read other staging models
• dbt staging models that read marts
• dbt marts that read raw sources directly
• dbt models mixing ref() and source()
• dbt sources read directly by several models
• dbt models reading several raw sources at once
• dbt models joined back onto their own parent
• dbt exposures built on views or raw sources
• dbt exposures on access: private models
• dbt YAML columns missing from the warehouse
• dbt models built differently from their config
• dbt models that do not exist in the warehouse
• dbt models that never say what one row is

Can a model be read and changed safely: explicit columns, a readable body, no copied logic?

• Duplicate logic across dbt models
• select * in dbt models
• dbt models with too many CTEs
• dbt layer naming: inconsistent model prefixes
• dbt tests declared away from their models
• select * in SQL transformation files
• SQL files with too many CTEs
• Duplicate SQL files in a warehouse repo

Did the last run build every model and pass every test?

• dbt tests that are failing
• dbt models that failed to build
• dbt models skipped behind a failure
• dbt table models dominating build time
• dbt tests slower than the model they test
• Pipeline jobs whose latest run failed

Can each model, source and exposure be routed to a named owner who can be reached?

• dbt models with no owner
• dbt sources with no owner
• dbt groups with no contactable owner
• dbt exposures with no contactable owner
• Dagster assets with no owner

Does anybody read what the project builds, and are the most-read models tested?

• dbt models nobody queries
• Widely read dbt models with no tests
• One dbt model dominating warehouse scans
• Snowflake warehouses that never suspend

Are AI and ML dependencies pinned, and is model use evaluated and observed?

• Unpinned AI and ML library versions
• Model training with no experiment tracking
• LLM apps with no evaluation suite
• LLM calls with no tracing or cost tracking
• Jupyter notebooks committed with outputs

Do scheduled jobs name an owner, retry, alert on failure, and avoid replaying history by accident?

• Airflow DAGs with no owner
• Airflow DAGs with no retries
• Airflow DAGs with no failure alert
• Airflow catchup left on by default
• Passwords and API keys hardcoded in DAG files
• Airflow sensors with no timeout
• dbt projects with no declared orchestrator
• dbt models behind dashboards on fragile schedules

Does the BI layer read the models dbt builds, and do its joins count correctly?

• dbt semantic models reading unbuilt columns
• dbt semantic model keys with no unique test
• Looker views reading tables dbt does not know
• Looker views reading raw dbt sources
• Looker SQL derived tables doing dbt's job
• Looker joined views with no primary key
• Looker joins with no declared relationship
• Power BI relationships filtering both ways
• Power BI tables reading unmodelled tables
• Tableau data sources using custom SQL
• Tableau data sources reading unmodelled tables
• One BI measure name, several definitions
• BI measures that redefine a dbt metric
• LookML joins that inflate sums (fan-out)