Home / Getting Started / Data Layer

Data Layer

The data layer is a combination of schemas, config, and annotations that hof understands and operates on. The primary goals are:

Support git-style checkpointing, history, and diff features, flexible to your datamodel schema
Provide consistent data models for downstream consumers
Enable downstream features like automatic database migrations, client/server version skew, and change detection for infrastructure as code

The hof datamodel command and schema/dm schema form the foundations and are designed so you can customize, extend, or replace as needed.

the built-in base models, fields, and enrichers
the shape and hierarchy for diff and history tracking

Note, hof dm is shorthand for hof datamodel.

Schemas

The core of hof datamodel is a set of schemas for adding metadate to a value. These indicate the various node types that give structure to your datamodel. The enables a flexible model that can still be used by the git-like features for tracking history, showing diffs, and generating migration code.

There are also schemas for common datamodel formats (like SQL) and enrichers for different languages (like Go & Python).

Core Schema

These core schemas are metadata that hof recognizes and treats specially to enable the hof datamodel commands.

Datamodel Schemas

package dm

import (
	"github.com/hofstadter-io/hof/schema"
)

// This is a complete Value tracked as one
// useful for schemas, config, and NoSQL
Object: {
	schema.Hof// needed for reFerences
	#hof: datamodel: root: true

	TrackHistory

	// all fields will be tracked
}

// This is like object, but supports cue values
// (todo, should support full lattice)
Value: {
	Object
	#hof: datamodel: cue: true
}

// This is a general datamodel useful in many applications
// It can be expanded and enriched to cover more
// Useful for SQL, APIs, forms, and similar
Datamodel: {
	schema.DHof  // needed for references
	#hof: datamodel: root: true
}

// Schema for a snapshot, can include anything else
Snapshot: {
	Timestamp: string | *""
}

// convenience type for embedding history
History: [...Snapshot]

TrackHistory: {
	#hof: datamodel: history: true // needed for CUE compat
	"Snapshot": Snapshot
	"History":  History
}

Common Formats

There are currently two core formats

schema/dm/fields/common.cue for common field types.
schema/dm/sql is the base for a relational datamodel.

Common Field Schema

package fields

DataTypes: ID |
	UUID |
	CUID |
	Bool |
	String |
	Int |
	Float |
	Enum |
	Password |
	Email

ID: UUID & {Default: "" | *"uuid_generate_v4()"}

Field: {
	Name:   string
	Plural: string | *"\(Name)s"
	Type:   string
	Reln?:  string
}

UUID: Field & {
	Type:     "uuid"
	Nullable: bool | *false
	Unique:   bool | *true
	Default?: string
	Validation: {
		Format: "uuid"
	}
}

CUID: Field & {
	Type:     "cuid"
	Nullable: bool | *false
	Unique:   bool | *true
}

Bool: Field & {
	Type:     "bool"
	Default:  string | *"false"
	Nullable: bool | *false
}

String: Field & {
	Type:     "string"
	Length:   int | *64
	Unique:   bool | *false
	Nullable: bool | *false
	Default?: string
	Validation: {
		Max: int | *Length
	}
}

Int: Field & {
	Type:     "int"
	Nullable: bool | *false
	Default?: int
}

Float: Field & {
	Type:     "float"
	Nullable: bool | *false
	Default?: float
}

Enum: Field & {
	Type: "string"
	Vals: [...string]
	Nullable: bool | *false
	Default?: string
}

Password: String & {
	Bcrypt: true
}

Email: String & {
	Validation: {
		Format: "email"
	}
	Unique: true
}

Date: Field & {
	Type: "date"
}

Time: Field & {
	Type: "time"
}

Datetime: Field & {
	Type: "datetime"
}

Enrichers

Enrichers extend or enhance a the schema to add language or library specifics. For example:

add language specific type used during code generation
map field types to library specific types

Enrichers are the most common type of datamodel customization as the target of code generation depends on your preferred tech stack.

See some examples here: schema/dm/enrichers

Commands and Example

This example will show you the basics of a datamodel and the hof dm commands.

hof dm -h (snippet)

# Example Usage   (dm is short for datamodel)

  $ hof dm list   (print known data models)
  NAME         TYPE       VERSION  STATUS  ID
  Config       object     -        ok      Config
  MyDatamodel  datamodel  -        ok      datamodel-abc123

  $ hof dm tree   (print the structure of the datamodels)
  $ hof dm diff   (prints a tree based diff of the datamodel)
  $ hof dm checkpoint -m "a message about this checkpoint"
  $ hof dm log    (prints the log of changes from latest to oldest)

  You can also use the -d & -e flags to subselect datamodels and nested values

# Learn more:
  - https://docs.hofstadter.io/getting-started/data-layer/
  - https://docs.hofstadter.io/data-modeling/

Usage:
  hof datamodel [command]

Aliases:
  datamodel, dm

Available Commands:
  checkpoint  create a snapshot of the data model
  diff        show the current diff or between datamodel versions
  list        print available datamodels
  log         show the history for a datamodel
  tree        print datamodel structure as a tree

Create a Datamodel

We’ll use a relational datamodel, typical of a database, for our example.

To create a datamodel, simply write some CUE. To see your datamodel in hof, run the following commands

hof dm list
hof eval datamodel.cue
hof tui view datamodel.cue

hof dm list

NAME       TYPE       VERSION  STATUS      ID        
Datamodel  datamodel  -        no-history  Datamodel

hof eval datamodel.cue

// Traditional database model which maps onto tables & columns.
Datamodel: {
	// schema for #hof: ...
	#hof: {
		// #hof version
		apiVersion: "v1beta1"

		// typical metadata
		metadata: {}

		// hof/datamodel
		datamodel: {
			// define the root of a datamodel
			root: true

			// instruct history to be tracked
			history: true

			// instruct ordrered version of the fields
			// to be injected as a peer value
			ordered: false

			// tell hof this is a node of interest for
			// the inspection commands (list,info)
			node: false

			// tell hof to track this as a raw CUE value
			// (partially implemented)
			cue: false
		}
	}
	Snapshot: {
		Timestamp: ""
	}

	// these are the models for the application
	// they can map onto database tables and apis
	Models: {
		User: {
			// for easy access
			Name:   "User"
			Plural: "Users"

			// These are the fields of a model
			// they can map onto database columnts and form fields
			Fields: {
				ID: {
					Name:     "ID"
					Plural:   "IDs"
					Type:     "uuid"
					Nullable: false
					Unique:   true
					Validation: {
						Format: "uuid"
					}
					#hof: {
						metadata: {
							name: "ID"
						}
					}
				}
				CreatedAt: {
					Name:   "CreatedAt"
					Plural: "CreatedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "CreatedAt"
						}
					}
				}
				UpdatedAt: {
					Name:   "UpdatedAt"
					Plural: "UpdatedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "UpdatedAt"
						}
					}
				}
				DeletedAt: {
					Name:   "DeletedAt"
					Plural: "DeletedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "DeletedAt"
						}
					}
				}
				email: {
					Name:     "email"
					Plural:   "emails"
					Type:     "string"
					Length:   64
					Unique:   true
					Nullable: false
					Validation: {
						Max:    64
						Format: "email"
					}
					#hof: {
						metadata: {
							name: "email"
						}
					}
				}
				username: {
					Name:     "username"
					Plural:   "usernames"
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "username"
						}
					}
				}
				password: {
					Name:     "password"
					Plural:   "passwords"
					Bcrypt:   true
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "password"
						}
					}
				}
				verified: {
					Name:     "verified"
					Plural:   "verifieds"
					Type:     "bool"
					Default:  "false"
					Nullable: false
					#hof: {
						metadata: {
							name: "verified"
						}
					}
				}
				active: {
					Name:     "active"
					Plural:   "actives"
					Type:     "bool"
					Default:  "false"
					Nullable: false
					#hof: {
						metadata: {
							name: "active"
						}
					}
				}
				persona: {
					Name:   "persona"
					Plural: "personas"
					Type:   "string"
					Vals: ["guest", "user", "admin", "owner"]
					Nullable: false
					Default:  "user"
					#hof: {
						metadata: {
							name: "persona"
						}
					}
				}
				#hof: {
					datamodel: {
						node:    true
						ordered: true
					}
				}
			}

			// if we want Relations as a separate value
			// we can process the fields to extract them
			// schema for #hof: ...
			#hof: {
				// #hof version
				apiVersion: "v1beta1"

				// typical metadata
				metadata: {
					name: "User"
				}

				// hof/datamodel
				datamodel: {
					// define the root of a datamodel
					root: false

					// instruct history to be tracked
					history: true

					// instruct ordrered version of the fields
					// to be injected as a peer value
					ordered: false

					// tell hof this is a node of interest for
					// the inspection commands (list,info)
					node: false

					// tell hof to track this as a raw CUE value
					// (partially implemented)
					cue: false
				}
			}
			Snapshot: {
				Timestamp: ""
			}
			History: []
		}
		#hof: {
			datamodel: {
				node:    true
				ordered: true
			}
		}
	}

	// OrderedModels: [...Model] will be
	// inject here for order stability
	History: []
}

datamodel.cue

package datamodel

import (
	"github.com/hofstadter-io/hof/schema/dm/sql"
	"github.com/hofstadter-io/hof/schema/dm/fields"
)

// Traditional database model which maps onto tables & columns.
Datamodel: sql.Datamodel & {
	Models: {
		User: {
			Fields: {
				ID:        fields.UUID
				CreatedAt: fields.Datetime
				UpdatedAt: fields.Datetime
				DeletedAt: fields.Datetime

				email:    fields.Email
				username: fields.String
				password: fields.Password
				verified: fields.Bool
				active:   fields.Bool

				persona: fields.Enum & {
					Vals: ["guest", "user", "admin", "owner"]
					Default: "user"
				}
			}
		}
	}
}

Checkpoints and History

Like a database and SQL migration files, you can checkpoint the history of your datamodels. This is an optional feature, but will allow you to automatically generate database migrations and code that can upgrade requests or downgrade responses, allowing for client/server version skew.

To checkpoint a datamodel, run hof dm checkpoint -s ... -m "..."

> hof dm checkpoint --suffix initial_user_model --message "initial user model for the application"
creating checkpoint: 20240507010604_initial_user_model "initial user model for the application"

At the root of your CUE module, you should now find a .hof/dm/... directory

> tree .hof 
.hof
└── dm
    └── Datamodel
        ├── 20240507010604_initial_user_model.cue
        └── Models
            └── User
                └── 20240507010604_initial_user_model.cue

5 directories, 2 files

The hof SQL datamodel tracks both the full datamodel and the individual models. This is done to ease the authoring of code generation templates that create database migrations and version skew functions. Generally, hof supports user defined datamodel hierarchy and history tracking.

Update a Datamodel

Next, we will add a UserProfile to the model.

hof eval datamodel.cue

// Traditional database model which maps onto tables & columns.
Datamodel: {
	// schema for #hof: ...
	#hof: {
		// #hof version
		apiVersion: "v1beta1"

		// typical metadata
		metadata: {}

		// hof/datamodel
		datamodel: {
			// define the root of a datamodel
			root: true

			// instruct history to be tracked
			history: true

			// instruct ordrered version of the fields
			// to be injected as a peer value
			ordered: false

			// tell hof this is a node of interest for
			// the inspection commands (list,info)
			node: false

			// tell hof to track this as a raw CUE value
			// (partially implemented)
			cue: false
		}
	}
	Snapshot: {
		Timestamp: ""
	}

	// these are the models for the application
	// they can map onto database tables and apis
	Models: {
		User: {
			// for easy access
			Name:   "User"
			Plural: "Users"

			// These are the fields of a model
			// they can map onto database columnts and form fields
			Fields: {
				ID: {
					Name:     "ID"
					Plural:   "IDs"
					Type:     "uuid"
					Nullable: false
					Unique:   true
					Validation: {
						Format: "uuid"
					}
					#hof: {
						metadata: {
							name: "ID"
						}
					}
				}
				CreatedAt: {
					Name:   "CreatedAt"
					Plural: "CreatedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "CreatedAt"
						}
					}
				}
				UpdatedAt: {
					Name:   "UpdatedAt"
					Plural: "UpdatedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "UpdatedAt"
						}
					}
				}
				DeletedAt: {
					Name:   "DeletedAt"
					Plural: "DeletedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "DeletedAt"
						}
					}
				}
				email: {
					Name:     "email"
					Plural:   "emails"
					Type:     "string"
					Length:   64
					Unique:   true
					Nullable: false
					Validation: {
						Max:    64
						Format: "email"
					}
					#hof: {
						metadata: {
							name: "email"
						}
					}
				}
				username: {
					Name:     "username"
					Plural:   "usernames"
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "username"
						}
					}
				}
				password: {
					Name:     "password"
					Plural:   "passwords"
					Bcrypt:   true
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "password"
						}
					}
				}
				verified: {
					Name:     "verified"
					Plural:   "verifieds"
					Type:     "bool"
					Default:  "false"
					Nullable: false
					#hof: {
						metadata: {
							name: "verified"
						}
					}
				}
				active: {
					Name:     "active"
					Plural:   "actives"
					Type:     "bool"
					Default:  "false"
					Nullable: false
					#hof: {
						metadata: {
							name: "active"
						}
					}
				}
				persona: {
					Name:   "persona"
					Plural: "personas"
					Type:   "string"
					Vals: ["guest", "user", "admin", "owner"]
					Nullable: false
					Default:  "user"
					#hof: {
						metadata: {
							name: "persona"
						}
					}
				}

				// relation fields
				Profile: {
					Name:     "Profile"
					Plural:   "Profiles"
					Type:     "uuid"
					Nullable: false
					Unique:   true
					Validation: {
						Format: "uuid"
					}

					// relation type, open to be flexible
					Relation: {
						Name:  "Profile"
						Type:  "has-one"
						Other: "Models.UserProfile"
					}

					// we can enrich this for various types
					// in our app or other reusable datamodels
					#hof: {
						metadata: {
							name: "Profile"
						}
					}
				}
				#hof: {
					datamodel: {
						node:    true
						ordered: true
					}
				}
			}

			// if we want Relations as a separate value
			// we can process the fields to extract them
			// schema for #hof: ...
			#hof: {
				// #hof version
				apiVersion: "v1beta1"

				// typical metadata
				metadata: {
					name: "User"
				}

				// hof/datamodel
				datamodel: {
					// define the root of a datamodel
					root: false

					// instruct history to be tracked
					history: true

					// instruct ordrered version of the fields
					// to be injected as a peer value
					ordered: false

					// tell hof this is a node of interest for
					// the inspection commands (list,info)
					node: false

					// tell hof to track this as a raw CUE value
					// (partially implemented)
					cue: false
				}
			}
			Snapshot: {
				Timestamp: ""
			}
			History: []
		}
		#hof: {
			datamodel: {
				node:    true
				ordered: true
			}
		}
		UserProfile: {
			// for easy access
			Name:   "UserProfile"
			Plural: "UserProfiles"

			// These are the fields of a model
			// they can map onto database columnts and form fields
			Fields: {
				About: {
					Name:   "About"
					Plural: "Abouts"
					SQL: {
						Type: "character varying(64)"
					}
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "About"
						}
					}
				}
				Avatar: {
					Name:   "Avatar"
					Plural: "Avatars"
					SQL: {
						Type: "character varying(64)"
					}
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "Avatar"
						}
					}
				}
				Social: {
					Name:   "Social"
					Plural: "Socials"
					SQL: {
						Type: "character varying(64)"
					}
					Type:     "string"
					Length:   64
					Unique:   false
					Nullable: false
					Validation: {
						Max: 64
					}
					#hof: {
						metadata: {
							name: "Social"
						}
					}
				}
				ID: {
					Name:     "ID"
					Plural:   "IDs"
					Type:     "uuid"
					Nullable: false
					Unique:   true
					Default:  "uuid_generate_v4()"
					Validation: {
						Format: "uuid"
					}
					#hof: {
						metadata: {
							name: "ID"
						}
					}
				}
				CreatedAt: {
					Name:   "CreatedAt"
					Plural: "CreatedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "CreatedAt"
						}
					}
				}
				Owner: {
					Name:     "Owner"
					Plural:   "Owners"
					Type:     "uuid"
					Nullable: false
					Unique:   true
					Validation: {
						Format: "uuid"
					}

					// relation type, open to be flexible
					Relation: {
						Name:  "Owner"
						Type:  "belongs-to"
						Other: "Models.User"
					}

					// we can enrich this for various types
					// in our app or other reusable datamodels
					#hof: {
						metadata: {
							name: "Owner"
						}
					}
				}
				UpdatedAt: {
					Name:   "UpdatedAt"
					Plural: "UpdatedAts"
					Type:   "datetime"
					#hof: {
						metadata: {
							name: "UpdatedAt"
						}
					}
				}
				#hof: {
					datamodel: {
						node:    true
						ordered: true
					}
				}
			}

			// if we want Relations as a separate value
			// we can process the fields to extract them
			// schema for #hof: ...
			#hof: {
				// #hof version
				apiVersion: "v1beta1"

				// typical metadata
				metadata: {
					name: "UserProfile"
				}

				// hof/datamodel
				datamodel: {
					// define the root of a datamodel
					root: false

					// instruct history to be tracked
					history: true

					// instruct ordrered version of the fields
					// to be injected as a peer value
					ordered: false

					// tell hof this is a node of interest for
					// the inspection commands (list,info)
					node: false

					// tell hof to track this as a raw CUE value
					// (partially implemented)
					cue: false
				}
			}
			Snapshot: {
				Timestamp: ""
			}
			History: []
		}
	}

	// OrderedModels: [...Model] will be
	// inject here for order stability
	History: []
}

datamodel.cue

package datamodel

import (
	"github.com/hofstadter-io/hof/schema/dm/sql"
	"github.com/hofstadter-io/hof/schema/dm/fields"
)

// Traditional database model which maps onto tables & columns.
Datamodel: sql.Datamodel & {
	Models: {
		User: {
			Fields: {
				ID:        fields.UUID
				CreatedAt: fields.Datetime
				UpdatedAt: fields.Datetime
				DeletedAt: fields.Datetime

				email:    fields.Email
				username: fields.String
				password: fields.Password
				verified: fields.Bool
				active:   fields.Bool

				persona: fields.Enum & {
					Vals: ["guest", "user", "admin", "owner"]
					Default: "user"
				}

				// relation fields
				Profile: fields.UUID
				Profile: Relation: {
					Name:  "Profile"
					Type:  "has-one"
					Other: "Models.UserProfile"
				}
			}
		}

		UserProfile: {
			Fields: {
				// note how we are using sql fields here
				sql.CommonFields

				About:  sql.Varchar
				Avatar: sql.Varchar
				Social: sql.Varchar

				Owner: fields.UUID
				Owner: Relation: {
					Name:  "Owner"
					Type:  "belongs-to"
					Other: "Models.User"
				}
			}
		}
	}
}

View a Datamodel

With our modified datamodel, we can explore some hof dm commands for inspecting it.

We can see that it has changes with hof dm list, note the dirty status.

hof dm list

NAME       TYPE       VERSION  STATUS  ID        
Datamodel  datamodel  -        dirty   Datamodel

We can also see the diff of those changes. hof uses a structural diff on the CUE value which allows for the hierarchical history.

hof dm diff

package diff

Datamodel: Models: {
	User: Fields: "+": {
		// relation fields
		Profile: {
			Name:     "Profile"
			Plural:   "Profiles"
			Type:     "uuid"
			Nullable: false
			Unique:   true
			Validation: Format: "uuid"

			// relation type, open to be flexible
			Relation: {
				Name:  "Profile"
				Type:  "has-one"
				Other: "Models.UserProfile"
			}

			// we can enrich this for various types
			// in our app or other reusable datamodels
		}
	}
	"+": UserProfile: {
		// for easy access
		Name:   "UserProfile"
		Plural: "UserProfiles"

		// These are the fields of a model
		// they can map onto database columnts and form fields
		Fields: {
			About: {
				Name:   "About"
				Plural: "Abouts"
				SQL: Type: "character varying(64)"
				Type:     "string"
				Length:   64
				Unique:   false
				Nullable: false
				Validation: Max: 64
			}
			Avatar: {
				Name:   "Avatar"
				Plural: "Avatars"
				SQL: Type: "character varying(64)"
				Type:     "string"
				Length:   64
				Unique:   false
				Nullable: false
				Validation: Max: 64
			}
			Social: {
				Name:   "Social"
				Plural: "Socials"
				SQL: Type: "character varying(64)"
				Type:     "string"
				Length:   64
				Unique:   false
				Nullable: false
				Validation: Max: 64
			}
			ID: {
				Name:     "ID"
				Plural:   "IDs"
				Type:     "uuid"
				Nullable: false
				Unique:   true
				Default:  "uuid_generate_v4()"
				Validation: Format: "uuid"
			}
			CreatedAt: {
				Name:   "CreatedAt"
				Plural: "CreatedAts"
				Type:   "datetime"
			}
			Owner: {
				Name:     "Owner"
				Plural:   "Owners"
				Type:     "uuid"
				Nullable: false
				Unique:   true
				Validation: Format: "uuid"

				// relation type, open to be flexible
				Relation: {
					Name:  "Owner"
					Type:  "belongs-to"
					Other: "Models.User"
				}

				// we can enrich this for various types
				// in our app or other reusable datamodels
			}
			UpdatedAt: {
				Name:   "UpdatedAt"
				Plural: "UpdatedAts"
				Type:   "datetime"
			}
		}

		// if we want Relations as a separate value
		// we can process the fields to extract them
		Snapshot: Timestamp: ""
		History: []
	}
}

package diff

User: Fields: "+": {
	// relation fields
	Profile: {
		Name:     "Profile"
		Plural:   "Profiles"
		Type:     "uuid"
		Nullable: false
		Unique:   true
		Validation: Format: "uuid"

		// relation type, open to be flexible
		Relation: {
			Name:  "Profile"
			Type:  "has-one"
			Other: "Models.UserProfile"
		}

		// we can enrich this for various types
		// in our app or other reusable datamodels
	}
}

With a new checkpoint…

hof dm checkpoint...

> hof dm checkpoint -s add_user_profile -m "add a user profile and give ownership to the user"
creating checkpoint: 20240507014051 "add a user profile and give ownership to the user"

we can also view the history log

> hof dm log
20240507014051_add_user_profile: "add a user profile and give ownership to the user"
  Datamodel         ~ has changes
    Models
      User          ~ has changes
      UserProfile   + new value

20240507010604_initial_user_model: "initial user model"
  Datamodel         + new value
    Models
      User          + new value

If we inspect the .hof/dm directory, we will see there are three new files. One for the datamodel change, and one for each model that was changed.

> tree .hof
.hof
└── dm
    └── Datamodel
        ├── 20240507010604_initial_user_model.cue
        ├── 20240507014051_add_user_profile.cue
        └── Models
            ├── User
            │   ├── 20240507010604_initial_user_model.cue
            │   └── 20240507014051_add_user_profile.cue
            └── UserProfile
                └── 20240507014051_add_user_profile.cue

6 directories, 5 files

Task Engine

LLM Chat

Edit this page Create page issue Create project issue