jina.types.document.multimodal

class jina.types.document.multimodal.MultimodalDocument(document=None, chunks=None, modality_content_map=None, copy=False, **kwargs)[source]

Bases: jina.types.document.Document

MultimodalDocument is a data type created based on Jina primitive data type Document.

It shares the same methods and properties with Document, while it focus on modality at chunk level.

Warning

  • It assumes that every chunk of a document belongs to a different modality.

  • It assumes that every MultimodalDocument have at least two chunks.

  • Build MultimodalDocument from modality_content_mapping expects you assign Document.content as the value of the dictionary.

Parameters
  • document (Optional[~DocumentSourceType]) – the document to construct from. If bytes is given then deserialize a DocumentProto; dict is given then parse a DocumentProto from it; str is given, then consider it as a JSON string and parse a DocumentProto from it; finally, one can also give DocumentProto directly, then depending on the copy, it builds a view or a copy from it.

  • chunks (Optional[Sequence[Document]]) – the chunks of the multimodal document to initialize with. Expected to received a list of Document, with different modalities.

  • copy (bool) – when document is given as a DocumentProto object, build a view (i.e. weak reference) from it or a deep copy from it.

  • kwargs – further key value arguments

  • document – the document to construct from. If bytes is given then deserialize a DocumentProto; dict is given then parse a DocumentProto from it; str is given, then consider it as a JSON string and parse a DocumentProto from it; finally, one can also give DocumentProto directly, then depending on the copy, it builds a view or a copy from it.

  • copy – when document is given as a DocumentProto object, build a view (i.e. weak reference) from it or a deep copy from it.

  • field_resolver – a map from field names defined in document (JSON, dict) to the field names defined in Protobuf. This is only used when the given document is a JSON string or a Python dict.

  • kwargs – other parameters to be set _after_ the document is constructed

Param

modality_content_mapping: A Python dict, the keys are the modalities and the values are the content of the Document

Note

When document is a JSON string or Python dictionary object, the constructor will only map the values from known fields defined in Protobuf, all unknown fields are mapped to document.tags. For example,

d = Document({'id': '123', 'hello': 'world', 'tags': {'good': 'bye'}})

assert d.id == '123'  # true
assert d.tags['hello'] == 'world'  # true
assert d.tags['good'] == 'bye'  # true
property is_valid

A valid MultimodalDocument should meet the following requirements:

  • Document should consist at least 2 chunks.

  • Length of modality is not identical to length of chunks.

Return type

bool

Returns

true if the document is valid

property modality_content_map

Get the mapping of modality and content, the mapping is represented as a dict, the keys are the modalities of the chunks, the values are the corresponded content of the chunks.

Return type

Dict

Returns

the mapping of modality and content extracted from chunks.

property modalities

Get all modalities of the MultimodalDocument.

Return type

List[str]

Returns

List of modalities extracted from chunks of the document.

update_content_hash(exclude_fields=('id', 'matches', 'content_hash'), include_fields=None)[source]

Update content hash of the document by including chunks when computing the hash

param exclude_fields

a tuple of field names that excluded when computing content hash

Parameters

include_fields (Optional[Tuple[str]]) – a tuple of field names that included when computing content hash

Return type

None