Internal architecture reference¶

These modules describe maintainer-facing seams below the public Python API. They are documented so refactors and plugin-boundary discussions have concrete interfaces to point at. They should not be treated as stable public API unless a public concept page or release note promotes a specific name.

Application orchestration¶

Catalog delegates add-operation setup into an internal application service, which in turn builds operation-runner requests.

Application services below the public catalog facade.

class ogcat.catalog_application.CatalogApplication(catalog)[source]¶

Bases: object

Coordinate catalog operations below the public Python API.

catalog: Catalog¶

add_file(*, source, metadata, schema, schema_record_type, record_type, directory_template, filename_template, operation, primary_location, create_template_replica, time_added)[source]¶

Run the managed local-file add operation.

Return type:: CatalogRecord

add_artifact(*, transaction, commit, record_type, locator, metadata, storage_mode, original_path, original_filename, suffixes, derived_metadata, naming_metadata, time_added, source, artifact_writer, storage_plan, schema)[source]¶

Run the general add-artifact operation.

Return type:: CatalogRecord

run_add_operation(*, transaction, commit, operation_type, record_type, schema, schema_record_type, metadata, storage_mode, original_path, original_filename, suffixes, derived_metadata, naming_metadata, time_added, source, locator_factory, materialization_intent, storage_plan_factory=None, derived_metadata_collector=None, secondary_artifact_operations=())[source]¶

Build and run a shared add-operation request.

Return type:: CatalogRecord

delete(*, record_id, reason, transaction, commit)[source]¶

Run the record tombstone operation.

Return type:: CatalogRecord

restore(*, record_id, reason, transaction, commit)[source]¶

Run the record restore operation.

Return type:: CatalogRecord

purge(*, record_id, force, transaction, commit)[source]¶

Run the permanent record purge operation.

Return type:: None

Internal operation runner interfaces and the add-operation implementation.

Catalog owns public API argument handling, schema selection, and transaction creation. Operation runners own the operation lifecycle once those inputs are prepared. The module-level OperationRunner ABC is intentionally generic so future operation families, such as artifact updates, can implement the same run() command interface without pretending they are add operations.

AddOperationRunner is the concrete runner for the current add lifecycle. It uses a template-style flow: run() fixes the ordering of validation, locator resolution, storage planning, artifact writing, metadata collection, record staging, commit, and rollback, while private phase methods keep each step separately testable and replaceable by future sibling runners.

class ogcat.operation_runner.OperationAuditEmitter(*args, **kwargs)[source]¶

Bases: Protocol

Callable used by the runner to emit catalog operation audit events.

class ogcat.operation_runner.MetadataValidationReporter(*args, **kwargs)[source]¶

Bases: Protocol

Callable used by the runner to validate operation metadata.

class ogcat.operation_runner.ArtifactRecordBuilder(*args, **kwargs)[source]¶

Bases: Protocol

Callable used by the runner to build a catalog record.

class ogcat.operation_runner.OperationServices(catalog_root, hook_manager, schema_name, metadata_validation_report, build_artifact_record, emit_operation_audit, emit_hook_lifecycle_audit)[source]¶

Bases: object

Catalog-owned services shared by internal operation runners.

catalog_root: Path¶

hook_manager: HookManager¶

schema_name: Callable[[str | None], str]¶

metadata_validation_report: MetadataValidationReporter¶

build_artifact_record: ArtifactRecordBuilder¶

emit_operation_audit: OperationAuditEmitter¶

emit_hook_lifecycle_audit: Callable[[HookLifecycleEvent], None]¶

class ogcat.operation_runner.AddOperationRequest(transaction, commit, operation_type, record_type, schema, schema_record_type, metadata, storage_mode, original_path, original_filename, suffixes, derived_metadata, naming_metadata, time_added, source, locator_factory, materialization_intent, storage_plan_factory=None, derived_metadata_collector=None, secondary_artifact_operations=())[source]¶

Bases: object

Inputs required to run one catalog add operation.

transaction: UnitOfWork¶

commit: bool¶

operation_type: str¶

record_type: str¶

schema: RecordSchema¶

schema_record_type: str | None¶

metadata: dict[str, str | int | float | bool | None | list[str | int | float | bool | None | list[JsonValue] | dict[str, JsonValue]] | dict[str, str | int | float | bool | None | list[JsonValue] | dict[str, JsonValue]]]¶

storage_mode: str | None¶

original_path: str | Path | None¶

original_filename: str | None¶

suffixes: list[str] | None¶

derived_metadata: dict[str, str | int | float | bool | None | list[str | int | float | bool | None | list[JsonValue] | dict[str, JsonValue]] | dict[str, str | int | float | bool | None | list[JsonValue] | dict[str, JsonValue]]]¶

naming_metadata: dict[str, str | int | float | bool | None | list[str | int | float | bool | None | list[JsonValue] | dict[str, JsonValue]] | dict[str, str | int | float | bool | None | list[JsonValue] | dict[str, JsonValue]]] | None¶

time_added: str | None¶

source: OperationSource¶

locator_factory: Callable[[OperationContext], ArtifactLocator]¶

materialization_intent: MaterializationIntent¶

storage_plan_factory: Callable[[OperationContext, ArtifactLocator], StoragePlan | None] | None¶

derived_metadata_collector: Callable[[OperationContext, ArtifactLocator], None] | None¶

secondary_artifact_operations: tuple[SecondaryArtifactOperation, ...]¶

class ogcat.operation_runner.RecordLifecycleOperationRequest(transaction, commit, operation_type, record, reason=None, force=False, managed_roots=())[source]¶

Bases: object

Inputs required to run one record lifecycle operation.

transaction: UnitOfWork¶

commit: bool¶

operation_type: str¶

record: CatalogRecord¶

reason: str | None¶

force: bool¶

managed_roots: tuple[Path, ...]¶

class ogcat.operation_runner.OperationRunner[source]¶

Bases: ABC

Explicit command interface for internal operation runners.

abstractmethod run()[source]¶

Run the operation and return the persisted or staged record.

Return type:: CatalogRecord | None

class ogcat.operation_runner.AddOperationRunner(dependencies, request)[source]¶

Bases: OperationRunner

Template-method coordinator for one internal add-operation lifecycle.

dependencies: OperationServices¶

request: AddOperationRequest¶

run()[source]¶

Run the add operation and return the persisted or staged record.

Return type:: CatalogRecord

class ogcat.operation_runner.RecordLifecycleOperationRunner(dependencies, request)[source]¶

Bases: OperationRunner

Coordinator for delete, restore, and purge record lifecycle operations.

dependencies: OperationServices¶

request: RecordLifecycleOperationRequest¶

run()[source]¶

Run the requested record lifecycle operation.

Return type:: CatalogRecord | None

Materialisation and storage planning¶

Storage planning answers where the primary artifact belongs. Materialisation answers how data reaches that target, or whether the operation is record-only.

Internal artifact materialization planning.

class ogcat.materialization.MaterializationIntent(writer, target_kind, write_mode, ogcat_owned)[source]¶

Bases: object

Operation intent for artifact materialization.

Parameters:

writer (ArtifactWriter | None) – Optional writer used to materialize artifact data.
target_kind (Literal['file', 'directory']) – Whether the materialized target is file-like or directory-like.
write_mode (Literal['copy', 'move', 'write', 'reference']) – How the target is materialized, or "reference" when no write should occur.
ogcat_owned (bool) – Whether ogcat should treat the materialized target as owned for storage-plan metadata.

writer: ArtifactWriter | None¶

target_kind: Literal['file', 'directory']¶

write_mode: Literal['copy', 'move', 'write', 'reference']¶

ogcat_owned: bool¶

class ogcat.materialization.MaterializationPlan(primary_target, intent)[source]¶

Bases: object

Resolved primary target paired with materialization intent.

primary_target: MaterializationTarget¶

intent: MaterializationIntent¶

to_storage_plan(*, checksum='none', profile=None, time_added=None)[source]¶

Build the concrete storage plan for the primary target.

Return type:: StoragePlan

class ogcat.materialization.MaterializationTarget(locator, target_kind, adapter=None, storage_relative_path=None, resolved_directory=None, resolved_filename=None, artifact_uuid=None, primary_location=None)[source]¶

Bases: object

Resolved primary target for artifact materialization.

Parameters:

locator (ArtifactLocator) – Canonical artifact locator to store on the record.
target_kind (Literal['file', 'directory']) – Whether the target is file-like or directory-like.
adapter (str | None) – Optional storage adapter identifier.
storage_relative_path (str | None) – Optional target path relative to the relevant storage root.
resolved_directory (str | None) – Optional rendered directory metadata.
resolved_filename (str | None) – Optional rendered final path component.
artifact_uuid (str | None) – Optional UUID-style primary storage identifier.
primary_location (Optional[Literal['uuid', 'template', 'user_provided']]) – Optional primary placement policy used for the plan.

locator: ArtifactLocator¶

target_kind: Literal['file', 'directory']¶

adapter: str | None¶

storage_relative_path: str | None¶

resolved_directory: str | None¶

resolved_filename: str | None¶

artifact_uuid: str | None¶

primary_location: Literal['uuid', 'template', 'user_provided'] | None¶

ogcat.materialization.materialization_plan_from_locator(locator, *, intent)[source]¶

Build a materialization plan directly from a canonical locator.

Return type:: MaterializationPlan

ogcat.materialization.reference_intent()[source]¶

Return the materialization intent for record-only references.

Return type:: MaterializationIntent

ogcat.materialization.storage_plan_intent(plan, *, writer=None)[source]¶

Return materialization intent with an explicit storage plan as authority.

Return type:: MaterializationIntent

ogcat.materialization.target_from_locator(locator, *, target_kind)[source]¶

Build a materialization target directly from a canonical locator.

Return type:: MaterializationTarget

ogcat.materialization.target_kind_from_writer(writer)[source]¶

Infer a storage target kind from a writer when it declares one.

Return type:: Literal['file', 'directory']

ogcat.materialization.validate_writer_matches_storage_plan(writer, plan)[source]¶

Raise when a writer declares target semantics that conflict with a plan.

Return type:: None

ogcat.materialization.write_mode_from_writer(writer)[source]¶

Infer a storage write mode from a writer when it declares one.

Return type:: Literal['copy', 'move', 'write', 'reference']

ogcat.materialization.writer_intent(writer)[source]¶

Return the materialization intent declared by a writer.

Return type:: MaterializationIntent

Storage location planning helpers.

This module contains path and URL selection policy for catalog-managed artifacts. It is intentionally separate from storage adapters, which perform side effects against already-planned locators.

class ogcat.storage_planning.PlannedLocator(locator, storage_relative_path, resolved_directory, resolved_filename)[source]¶

Bases: object

Rendered locator and path metadata for a planned storage target.

Parameters:

locator (ArtifactLocator) – Target artifact locator.
storage_relative_path (str | None) – Path relative to the relevant storage root.
resolved_directory (str | None) – Rendered storage directory, when available.
resolved_filename (str | None) – Rendered final path component, when available.

locator: ArtifactLocator¶

storage_relative_path: str | None¶

resolved_directory: str | None¶

resolved_filename: str | None¶

class ogcat.storage_planning.PrimaryStoragePlanningContext(catalog_root, files_root, objects_root, operation_id, metadata, directory_template, filename_template, source_path, storage_root, date_added, primary_location, locator=None)[source]¶

Bases: object

Inputs required to plan a primary artifact storage location.

Parameters:

catalog_root (Path) – Catalog root used for catalog-local relative paths.
files_root (Path) – Catalog template-managed files root.
objects_root (Path) – Catalog UUID-managed object root.
operation_id (str) – Operation id used as the planned artifact UUID.
metadata (Mapping[str, object]) – Normalized metadata available to naming templates.
directory_template (str) – Directory naming template from the record schema.
filename_template (str) – Filename naming template from the record schema.
source_path (Path | None) – Optional source path used for naming context.
storage_root (str | Path | None) – Optional external local root or fsspec URL root.
date_added (str) – ISO date string used by date-based templates.
primary_location (Literal['uuid', 'template', 'user_provided']) – Primary placement policy to render.
locator (ArtifactLocator | None) – User-provided locator when primary_location is "user_provided".

catalog_root: Path¶

files_root: Path¶

objects_root: Path¶

operation_id: str¶

metadata: Mapping[str, object]¶

directory_template: str¶

filename_template: str¶

source_path: Path | None¶

storage_root: str | Path | None¶

date_added: str¶

primary_location: Literal['uuid', 'template', 'user_provided']¶

locator: ArtifactLocator | None¶

class ogcat.storage_planning.PrimaryStoragePlanResult(locator, storage_relative_path, resolved_directory, resolved_filename, artifact_uuid, primary_location, storage_root=None)[source]¶

Bases: object

Planned primary storage locator and derived metadata.

Parameters:

locator (ArtifactLocator) – Target artifact locator.
storage_relative_path (str | None) – Path relative to the relevant storage root.
resolved_directory (str | None) – Rendered storage directory, when available.
resolved_filename (str | None) – Rendered final path component, when available.
artifact_uuid (str | None) – UUID-style artifact storage identifier, when generated.
primary_location (Literal['uuid', 'template', 'user_provided']) – Primary placement policy used for the plan.
storage_root (Path | None) – Local storage root used to recalculate path metadata when hooks replace the planned locator.

locator: ArtifactLocator¶

storage_relative_path: str | None¶

resolved_directory: str | None¶

resolved_filename: str | None¶

artifact_uuid: str | None¶

primary_location: Literal['uuid', 'template', 'user_provided']¶

storage_root: Path | None¶

to_storage_plan(*, locator=None, target_kind='file', write_mode='reference', checksum='none', ogcat_owned=False, profile=None, adapter=None, time_added=None, artifact_uuid=None)[source]¶

Build a concrete StoragePlan.

Parameters:

locator (ArtifactLocator | None) – Optional hook-resolved canonical locator.
target_kind (Literal['file', 'directory']) – Whether the target is file-like or directory-like.
write_mode (Literal['copy', 'move', 'write', 'reference']) – Intended materialisation mode.
checksum (Literal['none']) – Checksum policy requested for the write.
ogcat_owned (bool) – Whether ogcat should treat the target as managed.
profile (str | None) – Optional storage profile name or hint.
adapter (str | None) – Optional adapter identifier. When omitted, it is inferred from the planned locator.
time_added (str | None) – Optional timestamp used for the storage plan.
artifact_uuid (str | None) – Optional artifact UUID override for compatibility with operations that record the operation id separately from primary storage identity.

Return type:

StoragePlan

Returns:

Storage plan carrying the primary storage planning metadata.

to_materialization_target(*, locator=None, target_kind='file', adapter=None, artifact_uuid=None)[source]¶

Build the resolved primary target for a materialization plan.

Parameters:

locator (ArtifactLocator | None) – Optional hook-resolved canonical locator.
target_kind (Literal['file', 'directory']) – Whether the target is file-like or directory-like.
adapter (str | None) – Optional adapter identifier. When omitted, it is inferred from the planned locator.
artifact_uuid (str | None) – Optional artifact UUID override for compatibility with operations that record the operation id separately from primary storage identity.

Return type:

MaterializationTarget

Returns:

Materialization target carrying primary storage planning metadata.

naming_metadata()[source]¶

Build record naming metadata from the planned primary storage.

Return type:: dict[str, object]

class ogcat.storage_planning.UuidStoragePath(target, catalog_relative_path, storage_relative_path)[source]¶

Bases: object

Local UUID storage path and relative path metadata.

Parameters:

target (Path) – Local UUID target path.
catalog_relative_path (str) – Target path relative to the catalog root.
storage_relative_path (str) – Target path relative to the objects root.

target: Path¶

catalog_relative_path: str¶

storage_relative_path: str¶

ogcat.storage_planning.join_urlpath(root_url, relative_path)[source]¶

Join an fsspec URL root and relative path without local path coercion.

Return type:: str

ogcat.storage_planning.plan_primary_storage(context)[source]¶

Plan the primary storage location for a catalog artifact.

Parameters:: context (PrimaryStoragePlanningContext) – Storage planning inputs including roots, naming templates, metadata, and primary placement policy.
Return type:: PrimaryStoragePlanResult
Returns:: Planned primary storage locator and metadata suitable for constructing a storage plan or record naming metadata.
Raises:: ValueError – If primary_location is "user_provided" without a locator, or an unsupported primary placement policy is supplied.

ogcat.storage_planning.render_planned_locator(*, catalog_root, files_root, objects_root, operation_id, metadata, directory_template, filename_template, source_path, storage_root, date_added, primary_location)[source]¶

Render schema naming templates into a local or fsspec target locator.

Parameters:

catalog_root (Path) – Catalog root used for catalog-local relative paths.
files_root (Path) – Catalog template-managed files root.
objects_root (Path) – Catalog UUID-managed object root.
operation_id (str) – Operation id used as the planned artifact UUID.
metadata (Mapping[str, object]) – Normalized metadata available to naming templates.
directory_template (str) – Directory naming template from the record schema.
filename_template (str) – Filename naming template from the record schema.
source_path (Path | None) – Optional source path used for naming context.
storage_root (str | Path | None) – Optional external local root or fsspec URL root.
date_added (str) – ISO date string used by date-based templates.
primary_location (Literal['uuid', 'template']) – Primary placement policy to render.

Return type:

PlannedLocator

Returns:

Rendered locator and path metadata.

ogcat.storage_planning.render_uuid_planned_locator(*, catalog_root, objects_root, storage_root, artifact_uuid, original_path)[source]¶

Render a UUID primary locator for local or fsspec storage roots.

Parameters:

catalog_root (Path) – Catalog root used for catalog-local relative paths.
objects_root (Path) – Catalog object-storage root used when no explicit storage root is supplied.
storage_root (str | Path | None) – Optional external local root or fsspec URL root.
artifact_uuid (str) – Stable artifact identifier used in the target filename.
original_path (Path) – Original source path used only to preserve naming suffixes.

Return type:

PlannedLocator

Returns:

Rendered locator and path metadata.

ogcat.storage_planning.storage_relative_path_for_locator(locator, *, storage_root)[source]¶

Return a storage-root-relative path for a locator when available.

Parameters:

locator (ArtifactLocator) – Locator to inspect.
storage_root (Path) – Local storage root used for path-backed locators.

Return type:

str | None

Returns:

Storage-root-relative path, existing locator relative path, or None when no relative storage metadata is available.

ogcat.storage_planning.urlpath_exists_if_supported(urlpath)[source]¶

Return whether a URL path exists when fsspec is installed.

Parameters:: urlpath (str) – URL path to check.
Return type:: bool
Returns:: True when the fsspec target exists. Returns False when fsspec or a protocol-specific dependency is not installed so planning can remain a dry-run without optional storage dependencies.

ogcat.storage_planning.uuid_storage_path(*, catalog_root, objects_root, artifact_uuid, original_path)[source]¶

Return the local UUID primary path and relative path metadata.

Parameters:

catalog_root (Path) – Catalog root used to build catalog-relative locator metadata.
objects_root (Path) – Root directory for UUID-managed object storage.
artifact_uuid (str) – Stable artifact identifier used in the target filename.
original_path (Path) – Original source path used only to preserve naming suffixes.

Return type:

UuidStoragePath

Returns:

Local UUID storage path and relative path metadata.

Secondary artifacts¶

Secondary artifacts are ordered follow-up operations, such as the required template-link symlink created after a UUID-primary file record is staged.

Secondary artifact operations coordinated by add-operation runners.

class ogcat.secondary_artifacts.SecondaryArtifactOperation(*args, **kwargs)[source]¶

Bases: Protocol

Secondary artifact operation run after primary record staging.

property role: Literal['template_link']¶: Semantic role of the secondary artifact.

run(transaction, context, record)[source]¶

Materialize the secondary artifact for the staged record.

Return type:: SecondaryArtifactResult | None

class ogcat.secondary_artifacts.SecondaryArtifactResult(role, mode, message, event_type='secondary_artifact', naming_metadata_updates=<factory>, artifacts=(), audit_details=<factory>)[source]¶

Bases: object

Result from a materialized secondary artifact operation.

Parameters:

role (SecondaryArtifactRole) – Semantic role of the secondary artifact.
mode (ReplicaMode) – Materialization mode used for the secondary artifact.
message (str) – Audit message to emit after successful materialization.
event_type (str) – Audit event type to emit after successful materialization.
naming_metadata_updates (MetadataDict) – Metadata fields to merge onto the staged catalog record.
artifacts (tuple[ArtifactDescriptor, …]) – Artifact descriptors to append to the staged record.
audit_details (Mapping[str, object]) – Structured audit details for the operation.

role: SecondaryArtifactRole¶

mode: ReplicaMode¶

message: str¶

event_type: str¶

naming_metadata_updates: MetadataDict¶

artifacts: tuple[ArtifactDescriptor, ...]¶

audit_details: Mapping[str, object]¶

class ogcat.secondary_artifacts.TemplateLinkSecondaryArtifact(catalog_root, files_root, directory_template, filename_template, role='template_link', mode='symlink')[source]¶

Bases: object

Template-link symlink secondary for UUID primary artifacts.

catalog_root: Path¶

files_root: Path¶

directory_template: str¶

filename_template: str¶

role: Literal['template_link']¶

mode: Literal['symlink']¶

run(transaction, context, record)[source]¶

Create the template-link replica and return record metadata updates.

Return type:: SecondaryArtifactResult | None

Default template-link replica materialization.

class ogcat.template_replicas.TemplateLinkReplicaMaterialization(primary_path, target_path, catalog_relative_path, storage_relative_path, resolved_directory, resolved_filename, naming_metadata)[source]¶

Bases: object

Materialized default template replica details.

Parameters:

primary_path (Path) – Local primary path the symlink points at.
target_path (Path) – Local template replica symlink path.
catalog_relative_path (str) – Replica path relative to the catalog root.
storage_relative_path (str) – Replica path relative to the readable files root.
resolved_directory (str) – Rendered template replica directory.
resolved_filename (str) – Rendered template replica filename.
naming_metadata (MetadataDict) – Naming metadata to merge onto the catalog record.

primary_path: Path¶

target_path: Path¶

catalog_relative_path: str¶

storage_relative_path: str¶

resolved_directory: str¶

resolved_filename: str¶

naming_metadata: MetadataDict¶

ogcat.template_replicas.materialize_template_link_replica(*, catalog_root, files_root, record, directory_template, filename_template, register_rollback=None)[source]¶

Create the default template symlink replica for a path-backed record.

Parameters:

catalog_root (Path) – Catalog root used for catalog-relative path metadata.
files_root (Path) – Human-readable template replica root.
record (CatalogRecord) – Record whose primary path should be linked.
directory_template (str) – Directory template to render.
filename_template (str) – Filename template to render.
register_rollback (Callable[[Callable[[], None], str], object] | None) – Optional rollback registration callback accepting an action and a human-readable description.

Return type:

TemplateLinkReplicaMaterialization | None

Returns:

Materialized replica details, or None if the record is not path-backed.

Template context helpers shared by replica planners and materializers.

ogcat.replica_context.replica_template_context(record)[source]¶

Build a template context from record metadata and locator fields.

Return type:: dict[str, object]

Shared local symlink helpers for replica materialization.

ogcat.replica_links.relative_symlink_target(source_path, *, link_path)[source]¶

Return a relative symlink target when the platform can represent one.

Return type:: str | Path

ogcat.replica_links.symlink_points_to(link_path, source_path)[source]¶

Return whether a symlink points at a source path.

Return type:: bool

Repository boundary¶

Repository implementations own persistence. Catalog and operation services depend on the protocol rather than a concrete backend.

Repository abstractions.

class ogcat.repository.CatalogRepository(*args, **kwargs)[source]¶

Bases: Protocol

Abstract storage for catalog records.

insert(record)[source]¶

Insert a new record and return it with its repository-assigned id.

Return type:: CatalogRecord

insert_many(records)[source]¶

Insert multiple records and return them with repository-assigned ids.

Return type:: list[CatalogRecord]

get(record_id)[source]¶

Get a record by id.

Return type:: CatalogRecord | None

update(record)[source]¶

Update an existing record.

Return type:: None

delete(record_id)[source]¶

Delete an existing record.

Return type:: None

search(*, query=None, where=None, contains=None, regex=None, match=None, exists=None, missing=None, ignore_case=False, resolution_order=None)[source]¶

Search records.

Return type:: list[CatalogRecord]

all()[source]¶

Return all records.

Return type:: list[CatalogRecord]