jina.jaml package#

Subpackages#

Submodules#

Module contents#

class jina.jaml.JAML[source]#

Bases: object

A Jina YAML parser supports loading and dumping and substituting variables.

To use it:

from jina.jaml import JAML

JAML.load(...)
JAML.dump(...)


class DummyClass:
    pass


JAML.register(DummyClass)

You can use expressions to programmatically set variables in YAML files and access contexts. An expression can be any combination of literal values, references to a context, or functions. You can combine literals, context references, and functions using operators.

You need to use specific syntax to tell Jina to evaluate an expression rather than treat it as a string, which is based on GitHub actions syntax, and looks like this:

${{ <expression> }}

This expression can be evaluated directly (i.e. substituted by the real value) when being loaded, by using load(substitute=True)()

JAML supports three different kinds of variables to be used as expressions: Environment variables (coming form the environment itself), context variables (being passed as a dict), and internal references (included in the .yaml file itself).

An environment variable var is accessed through the following syntax:

${{ env.var }}

Note the mandatory spaces before and after the variable denotation.

Context variables can be accessed using the following syntax:

${{ context_var }}

Or, if you want to be explicit:

${{ context.context_var }}

These context variables are passed as a dict:

obj = JAML.load(
    fp, substitute=True, context={'context_var': 3.14, 'context_var2': 'hello-world'}
)

Internal references point to other variables in the yaml file itself, and can be accessed using the following syntax:

${{root.path.to.var}}

Note omission of spaces in this syntax.

Note

BaseFlow, BaseExecutor, BaseGateway and all their subclasses have already implemented JAML interfaces, to load YAML config into objects, please use Flow.load_config(), BaseExecutor.load_config(), etc.

static load(stream, substitute=False, context=None, runtime_args=None)[source]#

Parse the first YAML document in a stream and produce the corresponding Python object.

Note

BaseFlow, BaseExecutor, BaseGateway and all their subclasses have already implemented JAML interfaces, to load YAML config into objects, please use Flow.load_config(), BaseExecutor.load_config(), etc.

Parameters:

  • stream – the stream to load

  • substitute (bool) – substitute environment, internal reference and context variables.

  • context (Optional[Dict[str, Any]]) – context replacement variables in a dict, the value of the dict is the replacement.

  • runtime_args (Optional[Dict[str, Any]]) – Optional runtime_args to be directly passed without being parsed into a yaml config

Returns:

the Python object

static escape(value, include_unknown_tags=True)[source]#

Escape the YAML content by replacing all customized tags ! to ``jtype: ``.

Parameters:

  • value (str) – the original YAML content

  • include_unknown_tags (bool) – if to include unknown tags during escaping

Return type:

str

Returns:

escaped YAML

static unescape(value, include_unknown_tags=True, jtype_whitelist=None)[source]#

Unescape the YAML content by replacing all ``jtype: `` to tags.

Parameters:

  • value (str) – the escaped YAML content

  • include_unknown_tags (bool) – if to include unknown tags during unescaping

  • jtype_whitelist (Optional[Tuple[str, ...]]) – the list of jtype to be unescaped

Return type:

str

Returns:

unescaped YAML

static registered_tags()[source]#

Return a list of JAMLCompatible classes that have been registered.

Return type:

List[str]

Returns:

tags

static registered_classes()[source]#

Return a dict of tags and JAMLCompatible classes that have been registered.

Return type:

Dict

Returns:

tags and classes

static cls_from_tag(tag)[source]#

Fetch class from yaml tag

Parameters:

tag (str) – yaml tag

Return type:

Optional[JAMLCompatible]

Returns:

class object from tag

static load_no_tags(stream, **kwargs)[source]#

Load yaml object but ignore all customized tags, e.g. !Executor, !Driver, !Flow.

Parameters:

  • stream – the output stream

  • kwargs – other kwargs

Returns:

the Python object

static expand_dict(d, context=None, resolve_cycle_ref=True, resolve_passes=3)[source]#

Expand variables from YAML file.

Parameters:

  • d (Dict) – yaml file loaded as python dict

  • context (Union[Dict, SimpleNamespace, None]) – context replacement variables in a dict, the value of the dict is the replacement.

  • resolve_cycle_ref – resolve internal reference if True.

  • resolve_passes (int) – number of rounds to resolve internal reference.

Return type:

Dict[str, Any]

Returns:

expanded dict.

static dump(data, stream=None, **kwargs)[source]#

Serialize a Python object into a YAML stream.

If stream is None, return the produced string instead.

Parameters:

  • data – the data to serialize

  • stream – the output stream

  • kwargs – other kwargs

Returns:

the yaml output

static register(cls)[source]#

Register a class for dumping loading.

  • if it has attribute yaml_tag use that to register, else use class name

  • if it has methods to_yaml/from_yaml use those to dump/load else dump attributes as mapping

Parameters:

cls – the class to register

Returns:

the registered class

class jina.jaml.JAMLCompatible[source]#

Bases: object

JAMLCompatible is a mixin class designed to be used with multiple inheritance.

It will add to_yaml() and from_yaml() to the target class, making that class JAML-friendly.

Warning

For the sake of cooperative multiple inheritance, do NOT implement __init__() for this class

save_config(filename=None)[source]#

Save the object’s config into a YAML file.

Parameters:

filename (Optional[str]) – file path of the yaml file, if not given then config_abspath is used

classmethod load_config(source, *, allow_py_modules=True, substitute=True, context=None, uses_with=None, uses_metas=None, uses_requests=None, extra_search_paths=None, py_modules=None, runtime_args=None, uses_dynamic_batching=None, needs=None, include_gateway=True, noblock_on_start=False, **kwargs)[source]#

A high-level interface for loading configuration with features of loading extra py_modules, substitute env & context variables. Any class that implements JAMLCompatible mixin can enjoy this feature, e.g. BaseFlow, BaseExecutor, BaseGateway and all their subclasses.

Support substitutions in YAML:

  • Environment variables: ${{ ENV.VAR }} (recommended), $VAR (deprecated).

  • Context dict (context): ${{ CONTEXT.VAR }}``(recommended), ``${{ VAR }}.

  • Internal reference via this and root: ${{this.same_level_key}}, ${{root.root_level_key}}

Substitutions are carried in the order and multiple passes to resolve variables with best effort.

!BaseEncoder
metas:
    name: ${{VAR_A}}  # env or context variables
    workspace: my-${{this.name}}  # internal reference

# load Executor from yaml file
BaseExecutor.load_config('a.yml')

# load Executor from yaml file and substitute environment variables
os.environ['VAR_A'] = 'hello-world'
b = BaseExecutor.load_config('a.yml')
assert b.name == 'hello-world'

# load Executor from yaml file and substitute variables from a dict
b = BaseExecutor.load_config('a.yml', context={'VAR_A': 'hello-world'})
assert b.name == 'hello-world'

# disable substitute
b = BaseExecutor.load_config('a.yml', substitute=False)
Parameters:

  • source (Union[str, TextIO, Dict]) – the multi-kind source of the configs.

  • allow_py_modules (bool) – allow importing plugins specified by py_modules in YAML at any levels

  • substitute (bool) – substitute environment, internal reference and context variables.

  • context (Optional[Dict[str, Any]]) – context replacement variables in a dict, the value of the dict is the replacement.

  • uses_with (Optional[Dict]) – dictionary of parameters to overwrite from the default config’s with field

  • uses_metas (Optional[Dict]) – dictionary of parameters to overwrite from the default config’s metas field

  • uses_requests (Optional[Dict]) – dictionary of parameters to overwrite from the default config’s requests field

  • extra_search_paths (Optional[List[str]]) – extra paths used when looking for executor yaml files

  • py_modules (Optional[str]) – Optional py_module from which the object need to be loaded

  • runtime_args (Optional[Dict[str, Any]]) – Optional dictionary of parameters runtime_args to be directly passed without being parsed into a yaml config

  • uses_dynamic_batching (Optional[Dict]) – dictionary of parameters to overwrite from the default config’s dynamic_batching field

  • needs (Optional[Set[str]]) – the name of the Deployment(s) that this Deployment receives data from. One can also use “gateway” to indicate the connection with the gateway.

  • include_gateway (bool) – Defines if the gateway deployment should be included, defaults to True

  • noblock_on_start (bool) – If set, starting a Pod/Deployment does not block the thread/process. It then relies on ‘ ‘wait_start_success at outer function for the postpone check.

  • kwargs – kwargs for parse_config_source

Return type:

JAMLCompatible

Returns:

JAMLCompatible object

static is_valid_jaml(obj)[source]#

Verifies the yaml syntax of a given object by first serializing it and attempting to deserialize and catch parser errors :type obj: Dict :param obj: yaml object :rtype: bool :return: whether the syntax is valid or not