DBT : Organizing DBT Models in Subdirectories: A Guide to YAML Configuration

DBT (Data Build Tool) is an essential tool for data engineers and analysts to build, test, and document data pipelines using SQL. DBT projects rely on YAML files for configuration and organization. In this article, we will explore how to organize your DBT models into subdirectories and appropriately configure your YAML files.

1. Organizing Models in Subdirectories

To create a more structured and maintainable DBT project, you can organize your models into subdirectories within the models folder. Grouping related models into subdirectories allows for easier navigation and project management.

1. Creating a More Structured Project: When you work on a data transformation project using DBT, you often have multiple SQL models that represent different aspects of your data analysis or reporting. To maintain order and clarity, it’s a good idea to structure your project in an organized way.
2. Organizing Models into Subdirectories: The recommendation is to group related models into subdirectories within the “models” folder of your DBT project. This means that instead of having all your SQL models in a single flat directory, you create subfolders to categorize and group models based on their purpose, source, or any other relevant criteria.
3. Benefits:
   • Easier Navigation: Organizing models into subdirectories makes it much easier to find specific models when you’re working on them or need to reference them.
   • Project Management: It simplifies project management because related models are grouped together, which can be especially helpful in larger projects.
   • Improved Clarity: A structured organization improves the clarity of your project’s layout, making it more understandable for you and your team members.

For example, you can create the following subdirectories for a project:

models/
    ├── customers/
    │   ├── dim_customers.sql
    │   └── fct_customer_orders.sql
    ├── orders/
    │   ├── dim_orders.sql
    │   └── fct_order_items.sql
    └── products/
        ├── dim_products.sql
        └── fct_product_sales.sql

version: 2

models:
  - name: dim_customers
    description: Dimension table containing customer information
    columns:
      - name: customer_id
        description: Unique identifier for customers
        tests:
          - unique
          - not_null

  - name: fct_customer_orders
    description: Fact table containing customer order information
    columns:
      - name: order_id
        description: Unique identifier for orders
        tests:
          - unique
          - not_null

name: my_project
version: 1.0

profile: my_profile

source-paths:
  - models/customers
  - models/orders
  - models/products

test-paths:
  - tests