orcalib.orca_chat

OrcaChat

OrcaChat(
    num_memories=20,
    model="facebook/opt-350m",
    mode="chat",
    use_rag=False,
    use_ground=False,
    max_context_length=4096,
    tokenizer=None,
    database=None,
    model_id="orca_public_chatbot_v0",
    model_version=None,
    metadata=None,
    curate_enabled=False,
    tags=None,
    memory_index_name=None,
    lookup_column_names=None,
    propagate_lookup_settings=True,
    **kwargs
)

Bases: OrcaModel

A basic wrapper for a chatbot model that uses Orca for memory retrieval, and a base model with a generate method for response generation.

Parameters:

• database (OrcaDatabase | str | None, default: None ) –

  The database to use for memory retrieval

• memory_index_name (str | None, default: None ) –

  The name of the index to use for memory retrieval

• lookup_column_names (list[str] | None, default: None ) –

  The names of the columns to retrieve for each memory

• num_memories (int, default: 20 ) –

  The number of memories to retrieve

• model (str | PreTrainedModel | Module, default: 'facebook/opt-350m' ) –

  The model to use for response generation

• mode (str, default: 'chat' ) –

  The mode of the chatbot

• use_rag (bool, default: False ) –

  Whether to use RAG for response generation

• use_ground (bool, default: False ) –

  Whether to use grounding for response generation

• max_context_length (int, default: 4096 ) –

  The maximum context length for response generation

• tokenizer (Optional[PreTrainedTokenizerBase], default: None ) –

  The tokenizer to use for response generation

• model_id (str, default: 'orca_public_chatbot_v0' ) –

  The name for the model for curate tracking of model runs

• model_version (str | None, default: None ) –

  The version of the model for curate tracking of model runs

• metadata (OrcaMetadataDict | None, default: None ) –

  The metadata for the model for curate tracking of model runs

• curate_enabled (bool, default: False ) –

  Whether to enable curate tracking of model runs

• tags (set[str] | None, default: None ) –

  The tags for the model for curate tracking of model runs

Other Parameters:

• **kwargs (Any) –

  Additional keyword arguments to pass to the model

lookup_result_transforms property writable

lookup_result_transforms

A list of transforms to apply to the lookup result. NOTE: This will be applied even when lookup_result_override is set.

extra_lookup_column_names property writable

extra_lookup_column_names

While set, all lookups will include these additional columns. They may inclue columns on the indexed table as well as index-specific columns, e.g., $score, $embedding.

lookup_query_override property writable

lookup_query_override

The query to use instead of performing a lookup. NOTE: This will be ignored if lookup_result_override is also set.

lookup_result_override property writable

lookup_result_override

The lookup result to use instead of performing a lookup.

lookup_database property writable

lookup_database

The name of the database to use for looking up memories.

memory_index_name property writable

memory_index_name

The name of the index to use for looking up memories.

lookup_column_names property writable

lookup_column_names

The names of the columns to retrieve for each memory.

num_memories property writable

num_memories

The number of memories to look up.

drop_exact_match property writable

drop_exact_match

Whether to drop exact matches from the results.

exact_match_threshold property writable

exact_match_threshold

The similarity threshold for exact matches.

shuffle_memories property writable

shuffle_memories

Whether to shuffle the looked up memories.

curate_database property writable

curate_database

The name of the database to use for saving curate tracking data.

curate_next_run_settings property writable

curate_next_run_settings

The settings for the next curate model run.

curate_model_id property writable

curate_model_id

The model id to associate with curated model runs.

curate_model_version property writable

curate_model_version

The model version to associate with curated model runs.

curate_metadata property writable

curate_metadata

The metadata to attach to curated model runs.

curate_tags property writable

curate_tags

The tags to attach to the curated model runs.

curate_seq_id property writable

curate_seq_id

The sequence id to associate with curated model runs.

curate_batch_size property writable

curate_batch_size

The batch size of the model run to track curate data for, usually inferred automatically.

last_curate_run_ids property writable

last_curate_run_ids

The run ids of the last model run for which curate tracking data was collected.

last_curate_run_settings property writable

last_curate_run_settings

The settings of the last model run for which curate tracking data was collected.

get_effective_lookup_settings

get_effective_lookup_settings()

Returns the effective lookup settings for this module, with any inherited settings applied.

Returns:

• LookupSettings –

  The effective lookup settings for this module. Practically, this be the lookup settings

• LookupSettings –

  set on this module. For any settings that are not set on this module, the inherited settings

• LookupSettings –

  will be used instead.

get_lookup_database_instance

get_lookup_database_instance()

Returns the OrcaDatabase instance to use for looking up memories.

get_orca_modules_recursively

get_orca_modules_recursively(
    max_depth=None, include_self=True, filter_type=None
)

Recursively yields all children of this module that are instances of the specified filter type.

• All parent nodes will be processed before their children
• This will search through all children —even those that are not a subclass of the filter_type— but it only returns children that are a subclass of filter_type.

Parameters:

• max_depth (int | None, default: None ) –

  The maximum depth to search.

  • Setting this to 0 will only include this module.
  • Setting this to 1 will include only this module and its children.
  • Setting it to None (the default) will search through all modules.
  • Modules that are not of filter_type or OrcaModule do not increment the depth.
• include_self (bool, default: True ) –

  Whether to include the current OrcaModule in the results.

• filter_type (type[ORCA_MODULE_TYPE] | None, default: None ) –

  The subtype of OrcaModule to filter for. If None, any subtypes of OrcaModule will be returned.

Yields:

• ORCA_MODULE_TYPE –

  modules of type filter_type that are used in the children of this module.

enable_curate

enable_curate(recursive=True)

Enable Curate tracking for the model and (if recursive is True) for all its descendants.

Parameters:

• recursive (bool, default: True ) –

  Whether to enable Curate tracking recursively.

disable_curate

disable_curate(recursive=True)

Disable Curate tracking for this module and (if recursive is True) for all its descendants.

Parameters:

• recursive (bool, default: True ) –

  Whether to disable Curate tracking recursively.

update_curate_settings

update_curate_settings(
    model_id=None,
    model_version=None,
    tags=None,
    extra_tags=None,
    metadata=None,
    extra_metadata=None,
    batch_size=None,
    seq_id=None,
    enabled=None,
    enable_recursive=True,
)

Update curate tracking settings for the module and all its children.

Parameters:

• model_id (str | None, default: None ) –

  The ID of the model.

• model_version (str | None, default: None ) –

  The version of the model.

• tags (Iterable[str] | None, default: None ) –

  The new tags to be added to the model.

• extra_tags (Iterable[str] | None, default: None ) –

  The extra tags to be added to the model.

• metadata (OrcaMetadataDict | None, default: None ) –

  The new metadata to be added to the model.

• extra_metadata (OrcaMetadataDict | None, default: None ) –

  The extra metadata to be added to the model.

• batch_size (int | None, default: None ) –

  The batch size to be used for the model.

• seq_id (UUID | None, default: None ) –

  The sequence ID to be used for the model.

record_next_model_memory_lookups

record_next_model_memory_lookups(
    tags=None, metadata=None, batch_size=None, seq_id=None
)

Sets up curate tracking for the memory lookups during the next forward pass only.

Parameters:

• tags (Iterable[str] | None, default: None ) –

  Additional tags to be recorded on the next model run.

• metadata (OrcaMetadataDict | None, default: None ) –

  Additional metadata to be recorded on the next model run.

• batch_size (int | None, default: None ) –

  The batch size to be used for the next model run.

• seq_id (UUID | None, default: None ) –

  The sequence ID to be used for the next model run.

record_model_feedback

record_model_feedback(
    val, name="default", kind=FeedbackKind.CONTINUOUS
)

Records feedback for the last model runs for which memory lookups were recorded by curate.

Parameters:

• val (list[float] | float | int | list[int]) –

  The feedback to be recorded.

• name (str, default: 'default' ) –

  The name of the feedback.

• kind (FeedbackKind, default: CONTINUOUS ) –

  The kind of feedback.

record_model_input_output

record_model_input_output(inputs, outputs)

Records the inputs and outputs of the last model runs for which memory lookups were recorded by curate.

Parameters:

• inputs (list[Any] | Any) –

  The inputs to be recorded.

• outputs (list[Any] | Any) –

  The outputs to be recorded.

get_lookup_setting_summary

get_lookup_setting_summary()

Returns a summary of the lookup settings for each OrcaLookupLayer in this module and its descendants.

enable_memory

enable_memory()

Enables memory access for the model and all its children.

disable_memory

disable_memory()

Disables memory access for the model and all its children.

prep_next_gen

prep_next_gen(last_user_input)

Prepares the next generation by updating the chat history and last user input.

Parameters:

• last_user_input (str) –

  The last user input

Returns:

• input ( str ) –

  The last user input

• ui_history ( list[list[str]] ) –

  The list of user input and system responses

• last_accessed_memories ( list[list[Any]] | None ) –

  The last accessed memories

retrieve

retrieve(query)

Retrieve memories from the database based on the query.

Parameters:

• query (str) –

  The query to retrieve memories for

Returns:

• list[str] –

  list of retrieved memories

gen_response

gen_response()

Generate a response based on the last user input and the current context.

clear_chat

clear_chat()

Reset the ui_history, chat_history, and lookup_history attributes.

format_context

format_context(memories)

String format memories for prompt injection to model

Parameters:

• memories (list[str]) –

  The memories to format

Returns:

• str –

  The formatted context

update_grounding_params

update_grounding_params(sim_weight, bag_weight)

Update the grounding parameters for the OrcaGroundingProcessor.

Parameters:

• sim_weight (float) –

  The similarity weight

• bag_weight (float) –

  The bag weight

switch_config

switch_config(
    database_name,
    index_name,
    lookup_column_names,
    num_memories,
)

Switch the OrcaLookupConfig for the OrcaChat instance, clearing the chat history.

Note

The settings will be propagated to the OrcaLookupLayer, if it exists.

Parameters:

• database_name (str | None) –

  The name of the database to use for memory retrieval

• index_name (str | None) –

  The name of the index to use for memory retrieval

• lookup_column_names (list[str] | None) –

  The names of the columns to retrieve for each memory

• num_memories (int | None) –

  The number of memories to retrieve

print_chat_log

print_chat_log()

Print the chat log