Protocol Documentation

Table of Contents

• jina.proto

  • DenseNdArrayProto

  • DocumentProto

  • EnvelopeProto

  • EnvelopeProto.CompressConfigProto

  • EnvelopeProto.VersionProto

  • MessageProto

  • NamedScoreProto

  • NdArrayProto

  • QueryLangProto

  • RequestProto

  • RequestProto.ControlRequestProto

  • RequestProto.DeleteRequestProto

  • RequestProto.IndexRequestProto

  • RequestProto.SearchRequestProto

  • RequestProto.TrainRequestProto

  • RequestProto.UpdateRequestProto

  • RouteProto

  • SparseNdArrayProto

  • StatusProto

  • StatusProto.ExceptionProto

  • DenseNdArrayProto.QuantizationMode

  • RequestProto.ControlRequestProto.Command

  • StatusProto.StatusCode

  • JinaRPC

• Scalar Value Types

Top

jina.proto

DenseNdArrayProto

Represents a (quantized) dense n-dim array

Field	Type	Label	Description
buffer	bytes		the actual array data, in bytes
shape	uint32	repeated	the shape (dimensions) of the array
dtype	string		the data type of the array
quantization	DenseNdArrayProto.QuantizationMode		quantization mode
max_val	float		the max value of the ndarray
min_val	float		the min value of the ndarray
scale	float		the scale of the ndarray
original_dtype	string		the original dtype of the array

DocumentProto

Represents a Document

Field	Type	Label	Description
id	string		A hexdigest that represents a unique document ID
content_hash	string		A hexdigest that represents the hash of the content of the document
granularity	uint32		the depth of the recursive chunk structure
adjacency	uint32		the width of the recursive match structure
level_name	string		the name of the level
parent_id	string		the parent id from the previous level
buffer	bytes		the raw binary content of this document, which often represents the original document when comes into jina
blob	NdArrayProto		the ndarray of the image/audio/video document
text	string		a text document
chunks	DocumentProto	repeated	list of the chunks of this document (recursive structure)
weight	float		the weight of this document
length	uint32		total number of siblings of this document (docs that are in the same granularity and parent_id)
matches	DocumentProto	repeated	the top-k matched Docs on the same level (recursive structure)
mime_type	string		mime type of this document, for buffer content, this is required; for other contents, this can be guessed
uri	string		a uri of the document could be: a local file path, a remote url starts with http or https or data URI scheme
tags	google.protobuf.Struct		a structured data value, consisting of field which map to dynamically typed values.
location	uint32	repeated	the position of the doc, could be start and end index of a string; could be x,y (top, left) coordinate of an image crop; could be timestamp of an audio clip
offset	uint32		the offset of this doc in the previous level document
embedding	NdArrayProto		the embedding array of this doc
score	NamedScoreProto		TODO: List of matching scores performed on the document, each element corresponds to a metric
modality	string		modality, an identifier to the modality this document belongs to. In the scope of multi/cross modal search
evaluations	NamedScoreProto	repeated	List of evaluations performed on the document, each element corresponds to a metric

EnvelopeProto

Represents a Envelope, a part of the Message.

Field	Type	Label	Description
sender_id	string		unique id of the sender of the message
receiver_id	string		unique id of the receiver of the message, only used in router-dealer pattern
request_id	string		unique id of the request
timeout	uint32		timeout in second until this message is dropped
routes	RouteProto	repeated	a list of routes this message goes through
version	EnvelopeProto.VersionProto		version info
status	StatusProto		status info, when present, it is the first exception that routes carry
request_type	string		type of the request: TrainRequest, IndexRequest, SearchRequest, ControlRequest
check_version	bool		check local Protobuf version on every Pod that this message flows to
compression	EnvelopeProto.CompressConfigProto		compress configuration used for request

EnvelopeProto.CompressConfigProto

Represents a config for the compression algorithm

Field	Type	Label	Description
algorithm	string		compress algorithm used for request
high_watermark	uint64		the high watermark that triggers the message compression. message bigger than this HWM (in bytes) will be compressed by the algorithm.
low_watermark	float		the low watermark that enables the sending of a compressed message. compression rate (after_size/before_size) lower than this LWM will be considered as successeful compression, and will be sent. Otherwise, it will send the original message without compression
parameters	google.protobuf.Struct		other parameters that can be accepted by the algorithm

EnvelopeProto.VersionProto

Represents a the version information

Field	Type	Label	Description
jina	string		jina's version
proto	string		protobuf's version
vcs	string		vcs's version

MessageProto

Represents a Message

Field	Type	Label	Description
envelope	EnvelopeProto		the envelope of the message, used internally in jina, dropped when returning to client
request	RequestProto		the request body

NamedScoreProto

Represents the relevance model to ref_id

Field	Type	Label	Description
value	float		value
op_name	string		the name of the operator/score function
description	string		text description of the score
operands	NamedScoreProto	repeated	the score can be nested
ref_id	string		the score is computed between doc id and ref_id

NdArrayProto

Represents a general n-dim array, can be either dense or sparse

Field	Type	Label	Description
dense	DenseNdArrayProto		dense representation of the ndarray
sparse	SparseNdArrayProto		sparse representation of the ndarray

QueryLangProto

Field	Type	Label	Description
name	string		the name of the query lang, map to the corresponding driver
parameters	google.protobuf.Struct		the parameter of the query lang
disabled	bool		when set, then is query lang is disabled
priority	int32		when query lang is set on both driver and request levels, higher priority query lang is applied, lower priority is dropped

RequestProto

Represents a Request

Field	Type	Label	Description
request_id	string		the unique ID of this request. Multiple requests with the same ID will be gathered
train	RequestProto.TrainRequestProto		a train request
index	RequestProto.IndexRequestProto		an index request
search	RequestProto.SearchRequestProto		a search request
control	RequestProto.ControlRequestProto		a control request
delete	RequestProto.DeleteRequestProto		a delete request
update	RequestProto.UpdateRequestProto		an update request
status	StatusProto		status info
routes	RouteProto	repeated	status info on every routes
queryset	QueryLangProto	repeated	the query set

RequestProto.ControlRequestProto

Represents a control request used to control the BasePod

Field	Type	Label	Description
command	RequestProto.ControlRequestProto.Command		the control command
args	google.protobuf.Struct		the pea arguments, useful in STATUS

RequestProto.DeleteRequestProto

Represents a delete request

Field	Type	Label	Description
docs	DocumentProto	repeated	a list of Documents to delete
groundtruths	DocumentProto	repeated	the groundtruth document you want to evaluate it with

RequestProto.IndexRequestProto

Represents an index request

Field	Type	Label	Description
docs	DocumentProto	repeated	a list of Documents to index
groundtruths	DocumentProto	repeated	the groundtruth document you want to evaluate it with

RequestProto.SearchRequestProto

Represents a search request

Field	Type	Label	Description
docs	DocumentProto	repeated	a list of Documents to query
groundtruths	DocumentProto	repeated	the groundtruth document you want to evaluate it with

RequestProto.TrainRequestProto

Represents a train request

Field	Type	Label	Description
docs	DocumentProto	repeated	a list of Documents to train
groundtruths	DocumentProto	repeated	the groundtruth document you want to evaluate it with
flush	bool		if True then do actual training, otherwise only collect all documents but not do training.

RequestProto.UpdateRequestProto

Represents an update request

Field	Type	Label	Description
docs	DocumentProto	repeated	a list of Documents to update
groundtruths	DocumentProto	repeated	the groundtruth document you want to evaluate it with

RouteProto

Represents a the route paths of this message

Field	Type	Label	Description
pod	string		the name of the BasePod
pod_id	string		the id of the BasePod
start_time	google.protobuf.Timestamp		receiving time
end_time	google.protobuf.Timestamp		sending (out) time
status	StatusProto		the status of the execution

SparseNdArrayProto

Represents a sparse ndarray

Field	Type	Label	Description
indices	DenseNdArrayProto		A 2-D int64 tensor of shape [N, ndims], which specifies the indices of the elements in the sparse tensor that contain nonzero values (elements are zero-indexed)
values	DenseNdArrayProto		A 1-D tensor of any type and shape [N], which supplies the values for each element in indices.
dense_shape	int64	repeated	A 1-D int64 tensor of shape [ndims], which specifies the dense_shape of the sparse tensor.

StatusProto

Field	Type	Label	Description
code	StatusProto.StatusCode		status code
description	string		error description of the very first exception
exception	StatusProto.ExceptionProto		the details of the error

StatusProto.ExceptionProto

Field	Type	Label	Description
name	string		the class name of the exception
args	string	repeated	the list of arguments given to the exception constructor.
stacks	string	repeated	the exception traceback stacks
executor	string		the name of the executor bind to that peapod (if applicable)
driver	string		the name of the driver bind to that peapod (if applicable)

DenseNdArrayProto.QuantizationMode

Name	Number	Description
NONE	0	no quantization is performed, stored in the original dtype
FP16	1	2x smaller if dtype is set to FP32
UINT8	2	4x smaller but lossy when dtype is FP32

RequestProto.ControlRequestProto.Command

Name	Number	Description
TERMINATE	0	shutdown the BasePod
STATUS	1	check the status of the BasePod
IDLE	3	used in ROUTER-DEALER pattern, tells the router that the dealer is idle

StatusProto.StatusCode

Name	Number	Description
SUCCESS	0	success
PENDING	1	there are pending messages, more messages are followed
READY	2	ready to use
ERROR	3	error
ERROR_DUPLICATE	4	already a existing pod running
ERROR_NOTALLOWED	5	not allowed to open pod remotely
ERROR_CHAINED	6	chained from the previous error

JinaRPC

jina gRPC service.

Method Name	Request Type	Response Type	Description
Call	RequestProto stream	RequestProto stream	Pass in a Request and a filled Request with matches will be returned.

Scalar Value Types

.proto Type	Notes	C++	Java	Python	Go	C#	PHP	Ruby
double		double	double	float	float64	double	float	Float
float		float	float	float	float32	float	float	Float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
int64	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.	int64	long	int/long	int64	long	integer/string	Bignum
uint32	Uses variable-length encoding.	uint32	int	int/long	uint32	uint	integer	Bignum or Fixnum (as required)
uint64	Uses variable-length encoding.	uint64	long	int/long	uint64	ulong	integer/string	Bignum or Fixnum (as required)
sint32	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sint64	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.	int64	long	int/long	int64	long	integer/string	Bignum
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	int	uint32	uint	integer	Bignum or Fixnum (as required)
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	int/long	uint64	ulong	integer/string	Bignum
sfixed32	Always four bytes.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sfixed64	Always eight bytes.	int64	long	int/long	int64	long	integer/string	Bignum
bool		bool	boolean	boolean	bool	bool	boolean	TrueClass/FalseClass
string	A string must always contain UTF-8 encoded or 7-bit ASCII text.	string	String	str/unicode	string	string	string	String (UTF-8)
bytes	May contain any arbitrary sequence of bytes.	string	ByteString	str	[]byte	ByteString	string	String (ASCII-8BIT)