jina.jaml package#
Subpackages#
Submodules#
- jina.jaml.helper module
JinaConstructor
JinaConstructor.get_hashable_key()
JinaConstructor.construct_mapping()
JinaConstructor.add_constructor()
JinaConstructor.add_multi_constructor()
JinaConstructor.bool_values
JinaConstructor.check_data()
JinaConstructor.check_state_key()
JinaConstructor.construct_document()
JinaConstructor.construct_object()
JinaConstructor.construct_pairs()
JinaConstructor.construct_python_bytes()
JinaConstructor.construct_python_complex()
JinaConstructor.construct_python_long()
JinaConstructor.construct_python_module()
JinaConstructor.construct_python_name()
JinaConstructor.construct_python_object()
JinaConstructor.construct_python_object_apply()
JinaConstructor.construct_python_object_new()
JinaConstructor.construct_python_str()
JinaConstructor.construct_python_tuple()
JinaConstructor.construct_python_unicode()
JinaConstructor.construct_scalar()
JinaConstructor.construct_sequence()
JinaConstructor.construct_undefined()
JinaConstructor.construct_yaml_binary()
JinaConstructor.construct_yaml_bool()
JinaConstructor.construct_yaml_float()
JinaConstructor.construct_yaml_int()
JinaConstructor.construct_yaml_map()
JinaConstructor.construct_yaml_null()
JinaConstructor.construct_yaml_object()
JinaConstructor.construct_yaml_omap()
JinaConstructor.construct_yaml_pairs()
JinaConstructor.construct_yaml_seq()
JinaConstructor.construct_yaml_set()
JinaConstructor.construct_yaml_str()
JinaConstructor.construct_yaml_timestamp()
JinaConstructor.find_python_module()
JinaConstructor.find_python_name()
JinaConstructor.flatten_mapping()
JinaConstructor.get_data()
JinaConstructor.get_single_data()
JinaConstructor.get_state_keys_blacklist()
JinaConstructor.get_state_keys_blacklist_regexp()
JinaConstructor.inf_value
JinaConstructor.make_python_instance()
JinaConstructor.nan_value
JinaConstructor.set_python_instance_state()
JinaConstructor.timestamp_regexp
JinaConstructor.yaml_constructors
JinaConstructor.yaml_multi_constructors
JinaResolver
JinaResolver.DEFAULT_MAPPING_TAG
JinaResolver.DEFAULT_SCALAR_TAG
JinaResolver.DEFAULT_SEQUENCE_TAG
JinaResolver.add_implicit_resolver()
JinaResolver.add_path_resolver()
JinaResolver.ascend_resolver()
JinaResolver.check_resolver_prefix()
JinaResolver.descend_resolver()
JinaResolver.resolve()
JinaResolver.yaml_implicit_resolvers
JinaResolver.yaml_path_resolvers
JinaLoader
JinaLoader.DEFAULT_MAPPING_TAG
JinaLoader.DEFAULT_SCALAR_TAG
JinaLoader.DEFAULT_SEQUENCE_TAG
JinaLoader.DEFAULT_TAGS
JinaLoader.ESCAPE_CODES
JinaLoader.ESCAPE_REPLACEMENTS
JinaLoader.NON_PRINTABLE
JinaLoader.add_constructor()
JinaLoader.add_implicit_resolver()
JinaLoader.add_indent()
JinaLoader.add_multi_constructor()
JinaLoader.add_path_resolver()
JinaLoader.ascend_resolver()
JinaLoader.bool_values
JinaLoader.check_block_entry()
JinaLoader.check_data()
JinaLoader.check_directive()
JinaLoader.check_document_end()
JinaLoader.check_document_start()
JinaLoader.check_event()
JinaLoader.check_key()
JinaLoader.check_node()
JinaLoader.check_plain()
JinaLoader.check_printable()
JinaLoader.check_resolver_prefix()
JinaLoader.check_state_key()
JinaLoader.check_token()
JinaLoader.check_value()
JinaLoader.compose_document()
JinaLoader.compose_mapping_node()
JinaLoader.compose_node()
JinaLoader.compose_scalar_node()
JinaLoader.compose_sequence_node()
JinaLoader.construct_document()
JinaLoader.construct_mapping()
JinaLoader.construct_object()
JinaLoader.construct_pairs()
JinaLoader.construct_python_bytes()
JinaLoader.construct_python_complex()
JinaLoader.construct_python_long()
JinaLoader.construct_python_module()
JinaLoader.construct_python_name()
JinaLoader.construct_python_object()
JinaLoader.construct_python_object_apply()
JinaLoader.construct_python_object_new()
JinaLoader.construct_python_str()
JinaLoader.construct_python_tuple()
JinaLoader.construct_python_unicode()
JinaLoader.construct_scalar()
JinaLoader.construct_sequence()
JinaLoader.construct_undefined()
JinaLoader.construct_yaml_binary()
JinaLoader.construct_yaml_bool()
JinaLoader.construct_yaml_float()
JinaLoader.construct_yaml_int()
JinaLoader.construct_yaml_map()
JinaLoader.construct_yaml_null()
JinaLoader.construct_yaml_object()
JinaLoader.construct_yaml_omap()
JinaLoader.construct_yaml_pairs()
JinaLoader.construct_yaml_seq()
JinaLoader.construct_yaml_set()
JinaLoader.construct_yaml_str()
JinaLoader.construct_yaml_timestamp()
JinaLoader.descend_resolver()
JinaLoader.determine_encoding()
JinaLoader.dispose()
JinaLoader.fetch_alias()
JinaLoader.fetch_anchor()
JinaLoader.fetch_block_entry()
JinaLoader.fetch_block_scalar()
JinaLoader.fetch_directive()
JinaLoader.fetch_document_end()
JinaLoader.fetch_document_indicator()
JinaLoader.fetch_document_start()
JinaLoader.fetch_double()
JinaLoader.fetch_flow_collection_end()
JinaLoader.fetch_flow_collection_start()
JinaLoader.fetch_flow_entry()
JinaLoader.fetch_flow_mapping_end()
JinaLoader.fetch_flow_mapping_start()
JinaLoader.fetch_flow_scalar()
JinaLoader.fetch_flow_sequence_end()
JinaLoader.fetch_flow_sequence_start()
JinaLoader.fetch_folded()
JinaLoader.fetch_key()
JinaLoader.fetch_literal()
JinaLoader.fetch_more_tokens()
JinaLoader.fetch_plain()
JinaLoader.fetch_single()
JinaLoader.fetch_stream_end()
JinaLoader.fetch_stream_start()
JinaLoader.fetch_tag()
JinaLoader.fetch_value()
JinaLoader.find_python_module()
JinaLoader.find_python_name()
JinaLoader.flatten_mapping()
JinaLoader.forward()
JinaLoader.get_data()
JinaLoader.get_event()
JinaLoader.get_hashable_key()
JinaLoader.get_mark()
JinaLoader.get_node()
JinaLoader.get_single_data()
JinaLoader.get_single_node()
JinaLoader.get_state_keys_blacklist()
JinaLoader.get_state_keys_blacklist_regexp()
JinaLoader.get_token()
JinaLoader.inf_value
JinaLoader.make_python_instance()
JinaLoader.nan_value
JinaLoader.need_more_tokens()
JinaLoader.next_possible_simple_key()
JinaLoader.parse_block_mapping_first_key()
JinaLoader.parse_block_mapping_key()
JinaLoader.parse_block_mapping_value()
JinaLoader.parse_block_node()
JinaLoader.parse_block_node_or_indentless_sequence()
JinaLoader.parse_block_sequence_entry()
JinaLoader.parse_block_sequence_first_entry()
JinaLoader.parse_document_content()
JinaLoader.parse_document_end()
JinaLoader.parse_document_start()
JinaLoader.parse_flow_mapping_empty_value()
JinaLoader.parse_flow_mapping_first_key()
JinaLoader.parse_flow_mapping_key()
JinaLoader.parse_flow_mapping_value()
JinaLoader.parse_flow_node()
JinaLoader.parse_flow_sequence_entry()
JinaLoader.parse_flow_sequence_entry_mapping_end()
JinaLoader.parse_flow_sequence_entry_mapping_key()
JinaLoader.parse_flow_sequence_entry_mapping_value()
JinaLoader.parse_flow_sequence_first_entry()
JinaLoader.parse_implicit_document_start()
JinaLoader.parse_indentless_sequence_entry()
JinaLoader.parse_node()
JinaLoader.parse_stream_start()
JinaLoader.peek()
JinaLoader.peek_event()
JinaLoader.peek_token()
JinaLoader.prefix()
JinaLoader.process_directives()
JinaLoader.process_empty_scalar()
JinaLoader.remove_possible_simple_key()
JinaLoader.resolve()
JinaLoader.save_possible_simple_key()
JinaLoader.scan_anchor()
JinaLoader.scan_block_scalar()
JinaLoader.scan_block_scalar_breaks()
JinaLoader.scan_block_scalar_ignored_line()
JinaLoader.scan_block_scalar_indentation()
JinaLoader.scan_block_scalar_indicators()
JinaLoader.scan_directive()
JinaLoader.scan_directive_ignored_line()
JinaLoader.scan_directive_name()
JinaLoader.scan_flow_scalar()
JinaLoader.scan_flow_scalar_breaks()
JinaLoader.scan_flow_scalar_non_spaces()
JinaLoader.scan_flow_scalar_spaces()
JinaLoader.scan_line_break()
JinaLoader.scan_plain()
JinaLoader.scan_plain_spaces()
JinaLoader.scan_tag()
JinaLoader.scan_tag_directive_handle()
JinaLoader.scan_tag_directive_prefix()
JinaLoader.scan_tag_directive_value()
JinaLoader.scan_tag_handle()
JinaLoader.scan_tag_uri()
JinaLoader.scan_to_next_token()
JinaLoader.scan_uri_escapes()
JinaLoader.scan_yaml_directive_number()
JinaLoader.scan_yaml_directive_value()
JinaLoader.set_python_instance_state()
JinaLoader.stale_possible_simple_keys()
JinaLoader.timestamp_regexp
JinaLoader.unwind_indent()
JinaLoader.update()
JinaLoader.update_raw()
JinaLoader.yaml_constructors
JinaLoader.yaml_implicit_resolvers
JinaLoader.yaml_multi_constructors
JinaLoader.yaml_path_resolvers
get_jina_loader_with_runtime()
parse_config_source()
complete_path()
load_py_modules()
Module contents#
- class jina.jaml.JAML[source]#
Bases:
object
A Jina YAML parser supports loading and dumping and substituting variables.
To use it:
from jina.jaml import JAML JAML.load(...) JAML.dump(...) class DummyClass: pass JAML.register(DummyClass)
You can use expressions to programmatically set variables in YAML files and access contexts. An expression can be any combination of literal values, references to a context, or functions. You can combine literals, context references, and functions using operators.
You need to use specific syntax to tell Jina to evaluate an expression rather than treat it as a string, which is based on GitHub actions syntax, and looks like this:
${{ <expression> }}
This expression can be evaluated directly (i.e. substituted by the real value) when being loaded, by using
load(substitute=True)()
JAML supports three different kinds of variables to be used as expressions: Environment variables (coming form the environment itself), context variables (being passed as a dict), and internal references (included in the .yaml file itself).
An environment variable var is accessed through the following syntax:
${{ env.var }}
Note the mandatory spaces before and after the variable denotation.
Context variables can be accessed using the following syntax:
${{ context_var }}
Or, if you want to be explicit:
${{ context.context_var }}
These context variables are passed as a dict:
obj = JAML.load( fp, substitute=True, context={'context_var': 3.14, 'context_var2': 'hello-world'} )
Internal references point to other variables in the yaml file itself, and can be accessed using the following syntax:
${{root.path.to.var}}
Note omission of spaces in this syntax.
Note
BaseFlow
,BaseExecutor
,BaseGateway
and all their subclasses have already implemented JAML interfaces, to load YAML config into objects, please useFlow.load_config()
,BaseExecutor.load_config()
, etc.- static load(stream, substitute=False, context=None, runtime_args=None)[source]#
Parse the first YAML document in a stream and produce the corresponding Python object.
Note
BaseFlow
,BaseExecutor
,BaseGateway
and all their subclasses have already implemented JAML interfaces, to load YAML config into objects, please useFlow.load_config()
,BaseExecutor.load_config()
, etc.- Parameters:
stream – the stream to load
substitute (
bool
) – substitute environment, internal reference and context variables.context (
Optional
[Dict
[str
,Any
]]) – context replacement variables in a dict, the value of the dict is the replacement.runtime_args (
Optional
[Dict
[str
,Any
]]) – Optional runtime_args to be directly passed without being parsed into a yaml config
- Returns:
the Python object
- static escape(value, include_unknown_tags=True)[source]#
Escape the YAML content by replacing all customized tags
!
to ``jtype: ``.- Parameters:
value (
str
) – the original YAML contentinclude_unknown_tags (
bool
) – if to include unknown tags during escaping
- Return type:
str
- Returns:
escaped YAML
- static unescape(value, include_unknown_tags=True, jtype_whitelist=None)[source]#
Unescape the YAML content by replacing all ``jtype: `` to tags.
- Parameters:
value (
str
) – the escaped YAML contentinclude_unknown_tags (
bool
) – if to include unknown tags during unescapingjtype_whitelist (
Optional
[Tuple
[str
,...
]]) – the list of jtype to be unescaped
- Return type:
str
- Returns:
unescaped YAML
- static registered_tags()[source]#
Return a list of
JAMLCompatible
classes that have been registered.- Return type:
List
[str
]- Returns:
tags
- static registered_classes()[source]#
Return a dict of tags and
JAMLCompatible
classes that have been registered.- Return type:
Dict
- Returns:
tags and classes
- static cls_from_tag(tag)[source]#
Fetch class from yaml tag
- Parameters:
tag (
str
) – yaml tag- Return type:
Optional
[JAMLCompatible
]- Returns:
class object from tag
- static load_no_tags(stream, **kwargs)[source]#
Load yaml object but ignore all customized tags, e.g. !Executor, !Driver, !Flow.
- Parameters:
stream – the output stream
kwargs – other kwargs
- Returns:
the Python object
- static expand_dict(d, context=None, resolve_cycle_ref=True, resolve_passes=3)[source]#
Expand variables from YAML file.
- Parameters:
d (
Dict
) – yaml file loaded as python dictcontext (
Union
[Dict
,SimpleNamespace
,None
]) – context replacement variables in a dict, the value of the dict is the replacement.resolve_cycle_ref – resolve internal reference if True.
resolve_passes (
int
) – number of rounds to resolve internal reference.
- Return type:
Dict
[str
,Any
]- Returns:
expanded dict.
- class jina.jaml.JAMLCompatible[source]#
Bases:
object
JAMLCompatible
is a mixin class designed to be used with multiple inheritance.It will add
to_yaml()
andfrom_yaml()
to the target class, making that class JAML-friendly.Warning
For the sake of cooperative multiple inheritance, do NOT implement
__init__()
for this class- save_config(filename=None)[source]#
Save the object’s config into a YAML file.
- Parameters:
filename (
Optional
[str
]) – file path of the yaml file, if not given thenconfig_abspath
is used
- classmethod load_config(source, *, allow_py_modules=True, substitute=True, context=None, uses_with=None, uses_metas=None, uses_requests=None, extra_search_paths=None, py_modules=None, runtime_args=None, uses_dynamic_batching=None, needs=None, include_gateway=True, noblock_on_start=False, **kwargs)[source]#
A high-level interface for loading configuration with features of loading extra py_modules, substitute env & context variables. Any class that implements
JAMLCompatible
mixin can enjoy this feature, e.g.BaseFlow
,BaseExecutor
,BaseGateway
and all their subclasses.- Support substitutions in YAML:
Environment variables:
${{ ENV.VAR }}
(recommended),$VAR
(deprecated).Context dict (
context
):${{ CONTEXT.VAR }}``(recommended), ``${{ VAR }}
.Internal reference via
this
androot
:${{this.same_level_key}}
,${{root.root_level_key}}
Substitutions are carried in the order and multiple passes to resolve variables with best effort.
!BaseEncoder metas: name: ${{VAR_A}} # env or context variables workspace: my-${{this.name}} # internal reference
# load Executor from yaml file BaseExecutor.load_config('a.yml') # load Executor from yaml file and substitute environment variables os.environ['VAR_A'] = 'hello-world' b = BaseExecutor.load_config('a.yml') assert b.name == 'hello-world' # load Executor from yaml file and substitute variables from a dict b = BaseExecutor.load_config('a.yml', context={'VAR_A': 'hello-world'}) assert b.name == 'hello-world' # disable substitute b = BaseExecutor.load_config('a.yml', substitute=False)
- Parameters:
source (
Union
[str
,TextIO
,Dict
]) – the multi-kind source of the configs.allow_py_modules (
bool
) – allow importing plugins specified bypy_modules
in YAML at any levelssubstitute (
bool
) – substitute environment, internal reference and context variables.context (
Optional
[Dict
[str
,Any
]]) – context replacement variables in a dict, the value of the dict is the replacement.uses_with (
Optional
[Dict
]) – dictionary of parameters to overwrite from the default config’s with fielduses_metas (
Optional
[Dict
]) – dictionary of parameters to overwrite from the default config’s metas fielduses_requests (
Optional
[Dict
]) – dictionary of parameters to overwrite from the default config’s requests fieldextra_search_paths (
Optional
[List
[str
]]) – extra paths used when looking for executor yaml filespy_modules (
Optional
[str
]) – Optional py_module from which the object need to be loadedruntime_args (
Optional
[Dict
[str
,Any
]]) – Optional dictionary of parameters runtime_args to be directly passed without being parsed into a yaml configuses_dynamic_batching (
Optional
[Dict
]) – dictionary of parameters to overwrite from the default config’s dynamic_batching fieldneeds (
Optional
[Set
[str
]]) – the name of the Deployment(s) that this Deployment receives data from. One can also use “gateway” to indicate the connection with the gateway.include_gateway (
bool
) – Defines if the gateway deployment should be included, defaults to Truenoblock_on_start (
bool
) – If set, starting a Pod/Deployment does not block the thread/process. It then relies on ‘ ‘wait_start_success at outer function for the postpone check.kwargs – kwargs for parse_config_source
- Return type:
- Returns:
JAMLCompatible
object