Skip to content

Registry and specs

A DatasetSpec is pure data — everything connecto knows about a dataset, with no code. Registering one makes it available by name everywhere. See Adding a dataset.

Registry

registry

The dataset registry.

Datasets are data, so they live in a dict. Anyone can add one - including you, at runtime, without editing connecto:

spec = co.CAVE("zheng_ca3", fields={"type": ("cell_type",)}).spec
co.register(spec)
co.get_dataset("zheng_ca3")

register

register(spec: DatasetSpec, *, overwrite: bool = True) -> DatasetSpec

Add a dataset to the registry.

Source code in connecto/core/registry.py
def register(spec: DatasetSpec, *, overwrite: bool = True) -> DatasetSpec:
    """Add a dataset to the registry."""
    if spec.name in REGISTRY and not overwrite:
        raise ValueError(f"Dataset {spec.name!r} is already registered.")
    REGISTRY[spec.name] = spec
    return spec

get_dataset

get_dataset(name: str, **kwargs)

Build a dataset by name, e.g. get_dataset("flywire", version=783).

Source code in connecto/core/registry.py
def get_dataset(name: str, **kwargs):
    """Build a dataset by name, e.g. ``get_dataset("flywire", version=783)``."""
    from ..backends import build

    return build(get_spec(name), **kwargs)

get_spec

get_spec(name: str) -> DatasetSpec
Source code in connecto/core/registry.py
def get_spec(name: str) -> DatasetSpec:
    try:
        return REGISTRY[name]
    except KeyError:
        known = ", ".join(sorted(REGISTRY))
        raise NoSuchDatasetError(
            f"No dataset {name!r}. Registered: {known}."
        ) from None

list_datasets

list_datasets() -> DataFrame

Every registered dataset, which backends serve it, and whether you can read it.

The first backend listed is the default - the one get_dataset(name) gives you. public=False is not a secret; it means a fresh token is not enough, and get_spec(name).access says what is.

Source code in connecto/core/registry.py
def list_datasets() -> pd.DataFrame:
    """Every registered dataset, which backends serve it, and whether you can read it.

    The first backend listed is the default - the one ``get_dataset(name)`` gives
    you. ``public=False`` is not a secret; it means a fresh token is not enough, and
    ``get_spec(name).access`` says what is.
    """
    rows = [
        {
            "name": s.name,
            "label": s.label,
            "species": s.species,
            "backends": ", ".join(s.backend_kinds),
            "annotations": ", ".join(a.name for a in s.annotation_sources),
            "public": s.public,
        }
        for s in sorted(REGISTRY.values(), key=lambda s: s.name)
    ]
    return pd.DataFrame(rows)

capability_matrix

capability_matrix() -> DataFrame

Datasets x capabilities - one row per (dataset, backend).

The executable form of the "no silent degradation" promise: if a cell is False, the corresponding call raises rather than quietly returning something subtly wrong.

There is a row per door, not per dataset, because that is where a capability actually lives. BANC and FlyWire are each served by both backends and the doors are different widths - neuPrint adds ROIs and takes away the chunkedgraph - so a single row per dataset could only be true by being vague. The default backend (what you get from get_dataset(name)) comes first.

Source code in connecto/core/registry.py
def capability_matrix() -> pd.DataFrame:
    """Datasets x capabilities - one row per (dataset, backend).

    The executable form of the "no silent degradation" promise: if a cell is
    False, the corresponding call raises rather than quietly returning something
    subtly wrong.

    There is a row per *door*, not per dataset, because that is where a capability
    actually lives. BANC and FlyWire are each served by both backends and the doors
    are different widths - neuPrint adds ROIs and takes away the chunkedgraph - so a
    single row per dataset could only be true by being vague. The default backend
    (what you get from ``get_dataset(name)``) comes first.
    """
    caps = list(Cap)
    rows, index = [], []
    for s in sorted(REGISTRY.values(), key=lambda s: s.name):
        for b in s.backends:
            have = s.capabilities_for(b.kind)
            rows.append({"backend": b.kind} | {str(c): c in have for c in caps})
            index.append(s.name)
    return pd.DataFrame(rows, index=index)

Specs

DatasetSpec dataclass

DatasetSpec(name: str, backends: tuple[BackendSpec, ...], label: str = '', species: str = '', description: str = '', publications: tuple[Publication, ...] = (), links: Mapping[str, str] = dict(), public: bool = True, access: str = '', annotation_sources: tuple[AnnotationSource, ...] = (), fields: Mapping[str, tuple[str, ...]] = dict(), derive: Mapping[str, tuple[str, str]] = dict(), side_map: Mapping[str, str] = dict(), voxel_size: tuple[float, float, float] | None = None, template_space: str | None = None, segmentation_source: str | None = None, skeleton_source: str | None = None, sparsevol_source: SparseVolSource | None = None, viewer: str | None = None, viewer_dialect: str = 'modern', capabilities: frozenset[Cap] = frozenset(), example_ids: tuple[int, ...] = ())

Everything connecto knows about a dataset.

backend

backend(kind: str | None = None) -> BackendSpec

Get the backend spec of the given kind, or the default (the first).

Source code in connecto/core/spec.py
def backend(self, kind: str | None = None) -> BackendSpec:
    """Get the backend spec of the given kind, or the default (the first)."""
    if kind is None:
        return self.backends[0]
    for b in self.backends:
        if b.kind == kind:
            return b
    available = ", ".join(b.kind for b in self.backends)
    raise ValueError(
        f"{self.label} is not served by the {kind!r} backend. Available: {available}."
    )

capabilities_for

capabilities_for(kind: str | None = None) -> frozenset[Cap]

What this dataset can do through this backend.

self.capabilities is what the data has. This is what you can actually reach, and it is the only one anyone should ask: a dataset object is always bound to one backend, so the unqualified set is a claim nobody can use.

Source code in connecto/core/spec.py
def capabilities_for(self, kind: str | None = None) -> frozenset[Cap]:
    """What this dataset can do *through this backend*.

    `self.capabilities` is what the data has. This is what you can actually
    reach, and it is the only one anyone should ask: a dataset object is always
    bound to one backend, so the unqualified set is a claim nobody can use.
    """
    b = self.backend(kind)
    return (
        (self.capabilities | b.extra_capabilities)
        - BACKEND_LIMITS[b.kind]
        - b.missing_capabilities
    )

backends_with

backends_with(cap: Cap) -> tuple[str, ...]

Which of this dataset's backends can serve cap. Possibly none.

Exists so a CapabilityError can end with "the cave backend does" instead of a flat no - the difference between a dead end and a next step.

Source code in connecto/core/spec.py
def backends_with(self, cap: Cap) -> tuple[str, ...]:
    """Which of this dataset's backends can serve `cap`. Possibly none.

    Exists so a CapabilityError can end with "the cave backend does" instead of
    a flat no - the difference between a dead end and a next step.
    """
    return tuple(
        b.kind for b in self.backends if cap in self.capabilities_for(b.kind)
    )

annotation_source

annotation_source(name: str | None = 'auto') -> AnnotationSource | None

Pick an annotation source by name.

"auto" picks the first public source; None disables annotations.

Source code in connecto/core/spec.py
def annotation_source(self, name: str | None = "auto") -> AnnotationSource | None:
    """Pick an annotation source by name.

    ``"auto"`` picks the first *public* source; ``None`` disables annotations.
    """
    if name is None or not self.annotation_sources:
        return None
    if name == "auto":
        for s in self.annotation_sources:
            if s.public:
                return s
        # No public source. That is a gated dataset whose only annotations are
        # lab-internal (aedes: FlyTable). Returning None would make
        # `.annotations.get()` claim "no source configured" - false, there is
        # one; it just needs credentials. Fall back to the first source and let
        # the fetch raise a clear missing-token error if the caller can't read it.
        return self.annotation_sources[0]
    for s in self.annotation_sources:
        if s.name == name:
            return s
    available = ", ".join(s.name for s in self.annotation_sources)
    raise ValueError(
        f"{self.label} has no annotation source {name!r}. Available: {available}."
    )

evolve

evolve(**changes) -> DatasetSpec

Return a copy with the given fields replaced.

Source code in connecto/core/spec.py
def evolve(self, **changes) -> DatasetSpec:
    """Return a copy with the given fields replaced."""
    return replace(self, **changes)

BackendSpec dataclass

BackendSpec(kind: str, source: str, default_version: object = 'latest', extra_capabilities: frozenset[Cap] = frozenset(), position_units: str | None = None, skeleton_version: object = None, missing_capabilities: frozenset[Cap] = frozenset(), synapse_table: str | None = None, edge_view: str | None = None, nucleus_table: str | None = None, proofreading_table: str | None = None, score_column: str = 'cleft_score')

How one backend serves one dataset.

A dataset may have several of these - see the module docstring.

AnnotationSource dataclass

AnnotationSource(name: str, kind: str, location: str = '', id_column: str = 'root_id', public: bool = True, instance: str = 'flytable', chunked: bool = False, pivot: tuple[str, str] | None = None)

Where a dataset's annotations come from.

Orthogonal to the query backend: maleCNS is a neuPrint backend with either neuPrint or clio annotations; FlyWire is a CAVE backend with either a GitHub TSV or SeaTable annotations. Modelling this as a cross-product, rather than as subclasses, is what lets annotations="clio" be a parameter.

Provenance

Who made the dataset, and whether you may have it. See Who made it, and may you have it?.

Publication dataclass

Publication(authors: str, year: int, title: str, journal: str = '', doi: str = '')

A paper to cite when you use a dataset.

Datasets are other people's years of work. connecto is in the unusual position of knowing exactly which dataset you just queried, so it can also tell you who to credit for it - and ds.cite() is a great deal harder to forget than a citation buried in a README.

Capabilities

A capability belongs to a (dataset, backend) pair, not to a dataset — DatasetSpec.capabilities_for is the only one to ask. BACKEND_LIMITS says what a backend can never do; BackendSpec.missing_capabilities says what one server happens not to host. See Capabilities.

Cap

Bases: str, Enum

A thing a dataset can do.

Capabilities are declared per dataset and enforced in two places: whole namespaces (.segmentation simply does not exist without SEGMENTATION) and individual arguments (min_score= raises without SYNAPSE_SCORES rather than being silently ignored).