Catalog API¶

Catalog is the main public Python facade. It owns user-facing argument handling, schema selection, and delegation into internal application services. Search results are returned as ogcat.CatalogRecordSet by default; the record-set helpers are documented with search.

Record deletion is trash-style by default: Catalog.delete() tombstones a record and hides it from normal search, Catalog.restore() makes the record active again, and Catalog.purge() permanently removes a tombstoned record after removing managed catalog-local artifacts. Purge is best-effort across artifacts; incomplete cleanup raises PurgeIncompleteError after retaining the tombstone with purge outcome metadata.

class ogcat.Catalog(root, spec, repository, hook_manager=<factory>, audit_sink=None, audit_user_id=None)[source]¶

Bases: object

User-facing API bound to one catalog root.

Parameters:

root (Path) – Root directory containing catalog.json, db.json, and managed files.
spec (CatalogSpec) – Catalog specification loaded from or written to catalog.json.
repository (CatalogRepository) – Record storage backend.
hook_manager (HookManager) – Long-lived hook registry.
audit_sink (AuditSink | None) – Sink for structured operation audit events.
audit_user_id (str | None) – User id recorded on audit events.

root: Path¶

spec: CatalogSpec¶

repository: CatalogRepository¶

hook_manager: HookManager¶

audit_sink: AuditSink | None¶

audit_user_id: str | None¶

classmethod create(root, spec, *, plugins=None, hooks=None, audit_sink=None, audit_user_id=None)[source]¶

Create a catalog directory and write its specification.

Parameters:

root (str | Path) – Directory to create or reuse for the catalog.
spec (CatalogSpec) – Catalog specification to persist.
plugins (PluginRegistry | Iterable[object] | None) – Optional plugin registry, or iterable of hook objects, used to build a hook manager.
hooks (HookManager | Iterable[object] | None) – Optional hook manager, or iterable of hook objects. Pass either plugins or hooks.
audit_sink (AuditSink | None) – Optional audit sink. Defaults to a catalog-local JSONL sink under .ogcat/logs/events.jsonl.
audit_user_id (str | None) – Optional user id to record on audit events.

Return type:

Catalog

Returns:

Open catalog instance bound to root.

Raises:

ValueError – If the configured backend is unsupported, or both plugins and hooks are supplied.

classmethod open(root, *, plugins=None, hooks=None, audit_sink=None, audit_user_id=None)[source]¶

Open an existing catalog from disk.

Parameters:

root (str | Path) – Existing catalog root containing catalog.json.
plugins (PluginRegistry | Iterable[object] | None) – Optional plugin registry, or iterable of hook objects, used to build a hook manager.
hooks (HookManager | Iterable[object] | None) – Optional hook manager, or iterable of hook objects. Pass either plugins or hooks.
audit_sink (AuditSink | None) – Optional audit sink. Defaults to a catalog-local JSONL sink under .ogcat/logs/events.jsonl.
audit_user_id (str | None) – Optional user id to record on audit events.

Return type:

Catalog

Returns:

Open catalog instance bound to root.

Raises:

FileNotFoundError – If catalog.json is missing.
ValueError – If the configured backend is unsupported, or both plugins and hooks are supplied.

add_file(path, metadata=None, operation=None, record_type=None, primary_location='uuid', create_template_replica=True)[source]¶

Add a local file or file-like directory store using managed copy or move.

Parameters:

path (str | Path) – Source file or file-like directory store to ingest.
metadata (Mapping[Any, Any] | None) – JSON-compatible user metadata.
operation (str | None) – "copy" or "move". Defaults to the catalog spec.
record_type (str | None) – Optional named schema to validate against.
primary_location (Literal['uuid', 'template']) – "uuid" stores the primary artifact under a UUID path. "template" stores the primary artifact at the rendered template path.
create_template_replica (bool) – Whether UUID-primary file adds create a human-readable template symlink replica. Ignored for template-primary adds.

Return type:

CatalogRecord

Returns:

Persisted catalog record.

Raises:

TypeError – If metadata is not a dictionary.
ValueError – If validation fails, the operation is unsupported, or record_type names an unknown schema.

plan_artifact_storage(path=None, *, record_type=None, metadata=None, locator=None, target_kind='file', write_mode=None, ogcat_owned=True, storage_root=None, primary_location='uuid')[source]¶

Plan artifact storage without writing data or a catalog record.

Parameters:

path (str | Path | None) – Optional local source path used for naming and copy/move plans.
record_type (str | None) – Optional named schema to validate and use for naming.
metadata (Mapping[Any, Any] | None) – JSON-compatible user metadata.
locator (ArtifactLocator | None) – Optional pre-resolved target locator. When omitted, schema naming templates are rendered under storage_root or this catalog’s managed files root.
target_kind (Literal['file', 'directory']) – Whether the target is a file-like or directory-like artifact.
write_mode (Optional[Literal['copy', 'move', 'write', 'reference']]) – Desired materialisation mode. Defaults to "write" for owned artifacts and "reference" otherwise.
ogcat_owned (bool) – Whether ogcat should treat the target as managed.
storage_root (str | Path | None) – Optional local root or fsspec URL root for rendered template targets.
primary_location (Literal['uuid', 'template']) – "uuid" plans a UUID primary path. "template" plans the rendered schema template as the primary path. Ignored when locator is supplied.

Return type:

StoragePlan

Returns:

Planned storage decision.

add_artifact(*, record_type, locator=None, storage_plan=None, metadata=None, storage_mode=None, original_path=None, original_filename=None, suffixes=None, derived_metadata=None, naming_metadata=None, time_added=None, source=None, artifact_writer=None, transaction=None)[source]¶

Add an artifact record and optionally materialise planned storage.

This is the minimal general record API. add_file() remains the managed ingest convenience wrapper that prepares a path-backed locator and delegates through the same lifecycle.

Parameters:

record_type (str) – Logical type of record to create.
locator (ArtifactLocator | None) – Artifact locator to store with the record. Required unless storage_plan is supplied.
storage_plan (StoragePlan | None) – Optional planned storage decision to use instead of a standalone locator.
metadata (Mapping[Any, Any] | None) – JSON-compatible user metadata.
storage_mode (str | None) – Optional description such as "external".
original_path (str | Path | None) – Optional source path or URI.
original_filename (str | None) – Optional source filename.
suffixes (list[str] | None) – Optional suffix list for the source artifact.
derived_metadata (Mapping[Any, Any] | None) – Optional derived metadata to persist.
naming_metadata (Mapping[Any, Any] | None) – Optional naming metadata to persist.
time_added (str | None) – Optional timestamp override.
source (OperationSource | None) – Optional operation source for hooks and writers.
artifact_writer (ArtifactWriter | None) – Optional writer that materialises data before the record is written.
transaction (UnitOfWork | None) – Optional caller-owned unit of work.

Return type:

CatalogRecord

Returns:

Persisted or staged catalog record.

Raises:

TypeError – If metadata or writer inputs are invalid.
ValueError – If validation fails or the transaction belongs to a different repository.

add_reference(reference=None, *, uri=None, urlpath=None, record_type='external_reference', metadata=None, original_path=None, original_filename=None, suffixes=None, derived_metadata=None, naming_metadata=None, time_added=None, source=None, transaction=None)[source]¶

Record an existing path or locator without materialising storage.

add_reference() is a convenience wrapper around add_artifact() for artifacts that already exist. It records a reference only: no file is copied, moved, created, or required to live under the catalog’s managed files root.

Parameters:

reference (str | Path | ArtifactLocator | None) – Local filesystem path, URI-like string, or explicit artifact locator.
uri (str | None) – Optional explicit URI reference. Pass exactly one of reference, uri, or urlpath.
urlpath (str | None) – Optional explicit fsspec-style URL-path reference. Pass exactly one of reference, uri, or urlpath.
record_type (str) – Logical type of record to create.
metadata (Mapping[Any, Any] | None) – JSON-compatible user metadata.
original_path (str | Path | None) – Optional source path or URI override. Inferred for local path references when omitted.
original_filename (str | None) – Optional source filename override. Inferred for local path references when omitted.
suffixes (list[str] | None) – Optional source suffix list override. Inferred for local path references when omitted.
derived_metadata (Mapping[Any, Any] | None) – Optional derived metadata to persist.
naming_metadata (Mapping[Any, Any] | None) – Optional naming metadata to persist.
time_added (str | None) – Optional timestamp override.
source (OperationSource | None) – Optional operation source for hooks.
transaction (UnitOfWork | None) – Optional caller-owned unit of work.

Return type:

CatalogRecord

Returns:

Persisted or staged reference record.

add_collection(collection=None, *, uri=None, urlpath=None, record_type='collection', metadata=None, collection_pattern='*', member_format=None, member_suffixes=None, reader_hint=None, original_path=None, original_filename=None, suffixes=None, derived_metadata=None, naming_metadata=None, time_added=None, source=None, transaction=None)[source]¶

Record a directory-backed logical collection as one artifact.

Collection semantics are explicit: a plain directory reference remains a directory unless callers opt into this method.

Parameters:

collection (str | Path | ArtifactLocator | None) – Local directory, URI-like string, or explicit artifact locator for the collection root.
uri (str | None) – Optional explicit URI collection root. Pass exactly one of collection, uri, or urlpath.
urlpath (str | None) – Optional fsspec-style URL-path collection root. Pass exactly one of collection, uri, or urlpath.
record_type (str) – Logical type of record to create.
metadata (Mapping[Any, Any] | None) – JSON-compatible user metadata.
collection_pattern (str) – Relative pattern describing intended members, for example "*.nc".
member_format (str | None) – Optional format label for collection members.
member_suffixes (Sequence[str] | None) – Optional suffixes expected for collection members.
reader_hint (str | None) – Optional human-readable downstream reader hint.
original_path (str | Path | None) – Optional source path or URI override. Inferred for local collection roots when omitted.
original_filename (str | None) – Optional source filename override. Inferred for local collection roots when omitted.
suffixes (list[str] | None) – Optional source suffix list override. Inferred for local collection roots when omitted.
derived_metadata (Mapping[Any, Any] | None) – Optional derived metadata to persist alongside collection classification.
naming_metadata (Mapping[Any, Any] | None) – Optional naming metadata to persist.
time_added (str | None) – Optional timestamp override.
source (OperationSource | None) – Optional operation source for hooks.
transaction (UnitOfWork | None) – Optional caller-owned unit of work.

Return type:

CatalogRecord

Returns:

Persisted or staged collection record.

Raises:

ValueError – If a local collection root is not an existing directory or if collection metadata is invalid.

transaction()[source]¶

Create a best-effort unit of work for composed catalog operations.

The current TinyDB backend uses staged writes and compensating rollback actions. This context manager does not provide true database transactions or ACID semantics.

Return type:: Iterator[UnitOfWork]

audit_events(*, user_id=None, operation_id=None, record_id=None, level=None, event_type=None, limit=None)[source]¶

Return catalog-local audit events matching optional filters.

Parameters:

user_id (str | None) – Optional user id filter.
operation_id (str | None) – Optional operation id filter.
record_id (str | None) – Optional record id filter.
level (str | None) – Optional event severity filter.
event_type (str | None) – Optional lifecycle event type filter.
limit (int | None) – Optional number of most recent matching events to return.

Return type:

list[AuditEvent]

Returns:

Matching audit events in log order.

add_artifacts(artifacts)[source]¶

Add multiple artifact records.

Each item should provide the same keyword-style fields accepted by add_artifact(). Items are added one at a time so hooks and artifact writers run consistently for each record. Earlier items remain committed if a later item fails.

Parameters:: artifacts (list[dict[str, object]]) – List of dictionaries accepted by add_artifact().
Return type:: list[CatalogRecord]
Returns:: Persisted records in input order.

plan_view(root, template, *, mode='symlink', query=None, where=None, contains=None, regex=None, match=None, exists=None, missing=None, ignore_case=False)[source]¶

Plan a generated replica view without mutating records or files.

Parameters:

root (str | Path) – Root directory for the generated view.
template (str) – Combined path template relative to root.
mode (Literal['symlink']) – Replica materialisation mode. Only "symlink" is supported.
query (SearchQuery | None) – Optional pre-built search query.
where (Mapping[str, object] | None) – Equality filters.
contains (Mapping[str, object] | None) – Substring or list-membership filters.
regex (Mapping[str, str] | None) – Regular-expression filters.
match (Mapping[str, str] | None) – Glob or substring filters.
exists (Sequence[str] | None) – Fields that must be present.
missing (Sequence[str] | None) – Fields that must be absent.
ignore_case (bool) – Whether string comparisons should be case-insensitive.

Return type:

ReplicaViewPlan

Returns:

Dry-run replica view plan.

search(query=None, *, where=None, contains=None, regex=None, match=None, exists=None, missing=None, ignore_case=False, include_deleted=False, only_deleted=False, as_record_set=True)[source]¶

Search catalog records using backend-neutral query semantics.

Parameters:

query (SearchQuery | None) – Optional pre-built search query.
where (Mapping[str, object] | None) – Equality filters.
contains (Mapping[str, object] | None) – Substring or list-membership filters.
regex (Mapping[str, str] | None) – Regular-expression filters.
match (Mapping[str, str] | None) – Glob or substring filters.
exists (Sequence[str] | None) – Fields that must be present.
missing (Sequence[str] | None) – Fields that must be absent.
ignore_case (bool) – Whether string comparisons should be case-insensitive.
include_deleted (bool) – Include tombstoned records alongside active records.
only_deleted (bool) – Return only tombstoned records.
as_record_set (bool) – Return a CatalogRecordSet. Pass False for a list.

Return type:

list[CatalogRecord] | CatalogRecordSet

Returns:

Matching records, as a record-set view by default or a list when requested.

Raises:

ValueError – If include_deleted and only_deleted are both true.

get_one(query=None, *, where=None, contains=None, regex=None, match=None, exists=None, missing=None, ignore_case=False, include_deleted=False, only_deleted=False, allow_many=False)[source]¶

Return one matching record, raising clear errors for ambiguous searches.

Return type:: CatalogRecord

record_set(records)[source]¶

Wrap records in a sequence-like container.

Parameters:: records (Sequence[CatalogRecord]) – Records to expose through CatalogRecordSet helpers.
Return type:: CatalogRecordSet
Returns:: Record set using this catalog’s field resolution order.

describe(*, include_deleted=False)[source]¶

Return a serialisable summary of catalog configuration and contents.

Return type:: dict[str, object]

list_metadata_fields(record_type=None)[source]¶

Return serialisable schema-declared metadata field descriptions.

Return type:: list[MetadataDict]

list_record_fields(*, include_deleted=False)[source]¶

Return discoverable field paths present in stored records.

Return type:: list[str]

unique_values(field, *, include_deleted=False)[source]¶

Return unique scalar values present for a field across stored records.

Return type:: list[JsonValue]

get_schema(record_type=None)[source]¶

Return a serialisable schema description.

Return type:: dict[str, object]

list_record_schemas()[source]¶

Return available named record schema names.

Return type:: list[str]

get(record_id)[source]¶

Get a record by id after coercing public input with str().

Return type:: CatalogRecord | None

path(record_id)[source]¶

Return the stored path for a path-backed record, if present.

Return type:: Path | None

delete(record_id, *, reason=None, transaction=None)[source]¶

Tombstone a record so it is hidden from normal search results.

The record, artifact descriptors, and locators remain stored so the deletion can be audited, inspected, restored, or purged later.

Parameters:

record_id (object) – Existing record id.
reason (str | None) – Optional human-readable deletion reason.
transaction (UnitOfWork | None) – Optional caller-owned transaction. When supplied, the previous record version is restored if the transaction rolls back.

Return type:

CatalogRecord

Returns:

Tombstoned catalog record.

Raises:

KeyError – If the record id does not exist.
ValueError – If the record is already deleted or the transaction belongs to another repository.

restore(record_id, *, reason=None, transaction=None)[source]¶

Restore a tombstoned record to normal search visibility.

Parameters:

record_id (object) – Existing record id.
reason (str | None) – Optional human-readable restore reason.
transaction (UnitOfWork | None) – Optional caller-owned transaction. When supplied, the previous record version is restored if the transaction rolls back.

Return type:

CatalogRecord

Returns:

Restored catalog record.

purge(record_id, *, force=False)[source]¶

Permanently remove a record and its managed catalog-local artifacts.

Purge is irreversible and best-effort across artifacts. It only removes path-backed artifacts under this catalog’s managed files or objects roots; external or user-owned locators are skipped and audited. If cleanup is incomplete, the tombstoned record is retained with purge outcome metadata and the method raises PurgeIncompleteError.

Parameters:

record_id (object) – Existing record id.
force (bool) – Allow purging an active record. By default, records must be tombstoned with delete() first.

Raises:

KeyError – If the record id does not exist.
ValueError – If the record is active and force is false.
PurgeIncompleteError – If managed cleanup is incomplete and the tombstone is retained.

Return type:

None

update_metadata(record_id, metadata, mode='replace', *, transaction=None)[source]¶

Update a record’s user metadata through validation and storage.

Parameters:

record_id (object) – Existing record id.
metadata (Mapping[Any, Any]) – Replacement metadata or top-level metadata updates.
mode (Literal['replace', 'shallow_merge']) – "replace" replaces the whole user metadata dictionary. "shallow_merge" applies a top-level dictionary update; nested dictionaries are replaced as values, not recursively merged.
transaction (UnitOfWork | None) – Optional caller-owned transaction. When supplied, the previous record version is restored if the transaction rolls back.

Return type:

CatalogRecord

Returns:

Updated catalog record.

Raises:

TypeError – If metadata is not a dictionary.
ValueError – If the update mode is unsupported, or schema metadata validation fails.
KeyError – If the record id does not exist.

update_derived_metadata(record_id, derived_metadata, mode='replace', *, transaction=None)[source]¶

Update a record’s derived metadata through normalization and storage.

Parameters:

record_id (object) – Existing record id.
derived_metadata (Mapping[Any, Any]) – Replacement derived metadata or top-level updates.
mode (Literal['replace', 'shallow_merge']) – "replace" replaces the whole derived metadata dictionary. "shallow_merge" applies a top-level dictionary update; nested dictionaries are replaced as values, not recursively merged.
transaction (UnitOfWork | None) – Optional caller-owned transaction. When supplied, the previous record version is restored if the transaction rolls back.

Return type:

CatalogRecord

Returns:

Updated catalog record.

Raises:

TypeError – If derived metadata is not a dictionary.
ValueError – If the update mode is unsupported.
KeyError – If the record id does not exist.

add_record_schema(name, schema, *, overwrite=False)[source]¶

Add or replace a record schema in the catalog spec.

Parameters:

name (str) – Record schema name.
schema (RecordSchema | dict[str, object]) – Schema object or serialised schema dictionary.
overwrite (bool) – Whether an existing schema may be replaced.

Raises:

ValueError – If the schema already exists and overwrite is false, or if the resulting spec is invalid.
TypeError – If schema is not a valid schema object.

Return type:

None

set_default_record_schema(name)[source]¶

Set the default record schema by name.

Return type:: None

update_spec(**fields)[source]¶

Update simple catalog spec fields and persist catalog.json.

Supported fields are catalog_name, default_operation, and field_resolution_order. Storage root changes require a dedicated migration operation and are intentionally rejected here.

Return type:: None