Skip to content

orcalib.curate#

RunId module-attribute #

1
RunId = int

The id of a model run.

Curator #

1
Curator(target, model_id=None, model_version=None)

The Curator class provides an interface to record and query model feedback and input/output data.

Attributes:

  • model_id –

    The model id to filter runs by.

  • model_version –

    The model version to filter runs by.

  • runs (RunsHandle) –

    The runs table handle.

Examples:

Given a model with a lookup layer: 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
class MyModel(OrcaModel):
    def __init__(self):
        super().__init__(
            db,
            model_id="my_model_id"
            model_version="1"
        )
        self.lookup = OrcaLookupLayer(
            memory_index_name="text_index",
            lookup_column_names=["label, "$embedding"],
            num_memories=10
        )

    def forward(self, x):
        res = self.lookup(x)
        # do something with the memories

model = MyModel()

Create a curator for the model:

>>> curator = Curator(model)
>>> curator.model_id
'my_model_id'
>>> curator.model_version
'1'

Record feedback and input/output data for a model run:

>>> curator.record_model_feedback(
...     run_ids=1,
...     feedback=0.5,
...     name="default",
...     kind="CONTINUOUS"
... )
>>> curator.record_model_input_output(
...     run_ids=1,
...     inputs="test input",
...     outputs="test output"
... )

Query the runs of a model:

>>> runs_handle = curator.runs
>>> runs_handle.select(
...     "inputs", "outputs", "default_feedback"
... ).where(runs_handle.id == 1).fetch()
[{'inputs': 'test input', 'outputs': 'test output', 'default_feedback': 0.5}]

Query the memories of a model:

>>> my_memories_handle = curator.get_memories_handle("my_index")
>>> my_memories_handle.select("label", "text").aggregate_runs(
...     fn.avg(runs_handle.default_feedback).alias("avg_feedback"),
...     fn.count(runs_handle.id).alias("num_runs")
... ).fetch(1)
[{'label': 'my_label', 'text': 'my_text', 'avg_feedback': 0.5, 'num_runs': 1}]

Parameters:

  • target (Union[OrcaModule, OrcaDatabase, str]) –

    The target model or database to curate.

  • model_id (str | None, default: None ) –

    The model id to filter the results by.

  • model_version (str | None, default: None ) –

    The model version to filter the results by.

runs instance-attribute #

1
runs = RunsHandle(database_name, model_id, model_version)

A handle to query model runs.

get_memories_handle #

1
get_memories_handle(index_name)

Get a handle to query the memories table associated with the given index or table name.

Parameters:

  • index_name (str) –

    The index name associated with the memories.

Returns:

Examples:

>>> my_memories = curator.get_memories_handle("my_index")
>>> my_memories.select("label", "text").

record_model_feedback #

1
2
3
4
5
6
record_model_feedback(
    run_ids,
    val,
    name="default",
    kind=FeedbackKind.CONTINUOUS,
)

Records feedback for the given model runs.

Parameters:

  • run_ids (list[RunId] | RunId) –

    The run ids for which the feedback is recorded.

  • val (list[float] | float | int | list[int]) –

    The feedback to be recorded.

  • name (str, default: 'default' ) –

    The name of the feedback.

  • kind (FeedbackKind, default: CONTINUOUS ) –

    The kind of feedback

record_model_input_output #

1
record_model_input_output(run_ids, inputs, outputs)

Records the inputs and outputs of the given model runs.

Parameters:

  • run_ids (list[RunId] | RunId) –

    The run ids for which the inputs and outputs are recorded.

  • inputs (list[Any] | Any | None) –

    The inputs to be recorded.

  • outputs (list[Any] | Any | None) –

    The outputs to be recorded.

FeedbackKind #

Bases: str, Enum

The kind of feedback that can be recorded.

Attributes:

  • CONTINUOUS –

    any float values between -1.0 and 1.0 are allowed

  • BINARY –

    only the float or integer -1 or 1 is allowed

  • FLAG –

    only the float or integer 1 is allowed