Gateway#

Every Flow has a Gateway component that receives requests over the network, allowing clients to send data to the Flow for processing.

The Gateway is the first destination of a client request and its final destination, meaning that all incoming requests are routed to the Gateway and the Gateway is responsible for handling and responding to those requests. The Gateway supports multiple protocols and endpoints, such as gRPC, HTTP, WebSocket, and GraphQL, allowing clients to communicate with the Flow using the protocol of their choice.

In most cases, the Gateway is automatically configured when you initialize a Flow object, so you do not need to configure it yourself.

However, you can always explicitly configure the Gateway in Python using the config_gateway() method, or in YAML. The full YAML specification for configuring the Gateway can be found here.

Set protocol in Python#

You can use three different protocols to serve the Flow: gRPC, HTTP and WebSocket.

gRPC

from jina import Client, Executor, Flow, requests
from docarray import DocList
from docarray.documents import TextDoc

class FooExecutor(Executor):
    @requests
    def foo(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
        for doc in docs:
            doc.text = 'foo was called'


f = Flow().config_gateway(protocol='grpc', port=12345).add(uses=FooExecutor)
with f:
    client = Client(port=12345)
    docs = client.post(on='/', inputs=TextDoc(), return_type=DocList[TextDoc])
    print(docs.text)

['foo was called']

HTTP

from jina import Client, Executor, Flow, requests
from docarray import DocList
from docarray.documents import TextDoc

class FooExecutor(Executor):
    @requests
    def foo(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
        for doc in docs:
            doc.text = 'foo was called'


f = Flow().config_gateway(protocol='http', port=12345).add(uses=FooExecutor)
with f:
    client = Client(port=12345, protocol='http')
    docs = client.post(on='/', inputs=TextDoc(), return_type=DocList[TextDoc])
    print(docs.text)

['foo was called']

WebSocket

from jina import Client, Executor, Flow, requests
from docarray import DocList
from docarray.documents import TextDoc

class FooExecutor(Executor):
    @requests
    def foo(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
        for doc in docs:
            doc.text = 'foo was called'


f = Flow().config_gateway(protocol='websocket', port=12345).add(uses=FooExecutor)
with f:
    client = Client(port=12345, protocol='websocket')
    docs = client.post(on='/', inputs=TextDoc(), return_type=DocList[TextDoc])
    print(docs.text)

['foo was called']

Set protocol in YAML#

To configure the protocol in a YAML file:

gRPC

Note that gRPC is the default protocol, so you can just omit it.

jtype: Flow
gateway:
  protocol: 'grpc'

HTTP

jtype: Flow
gateway:
  protocol: 'http'

WebSocket

jtype: Flow
gateway:
  protocol: 'websocket'

Enable multiple protocols#

You can enable multiple protocols on the Gateway. This allows polyglot clients connect to your Flow with different protocols.

Python

from jina import Flow
flow = Flow().config_gateway(protocol=['grpc', 'http', 'websocket'])
with flow:
    flow.block()

YAML

jtype: Flow
gateway:
  protocol:
    - 'grpc'
    - 'http'
    - 'websocket'

../../../_images/multi-protocol-flow.png

Important

In case you want to serve a Flow using multiple protocols, make sure to specify as much ports as protocols used.

Enable TLS for client traffics#

You can enable TLS encryption between your Gateway and Clients, for any of the protocols supported by Jina (HTTP, gRPC, and WebSocket).

Caution

Enabling TLS will encrypt the data that is transferred between the Flow and the Client. Data that is passed between the microservices configured by the Flow, such as Executors, will not be encrypted.

To enable TLS encryption, you need to pass a valid keyfile and certfile to the Flow, using the ssl_keyfile ssl_certfile parameters:

from jina import Flow

Flow().config_gateway(
    port=12345,
    ssl_certfile='path/to/certfile.crt',
    ssl_keyfile='path/to/keyfile.crt',
)

If both of these are provided, the Flow will automatically configure itself to use TLS encryption for its communication with any Client.

Enable in-Flow compression#

The communication between Executors inside a Flow is done via gRPC. To optimize the performance and the bandwidth of these connections, you can enable compression by specifying compression argument to the Gateway.

The supported methods are: none, gzip and deflate.

from jina import Flow

f = Flow().config_gateway(compression='gzip').add(...)

Note that this setting is only effective the internal communication of the Flow. One can also specify the compression between client and gateway as described here.

Get environment information#

Gateway provides an endpoint that exposes environment information where it runs.

It is a dict-like structure with the following keys:

jina: A dictionary containing information about the system and the versions of several packages including jina package itself
envs: A dictionary containing all the values if set of the environment variables used in Jina

Use gRPC#

To see how this works, first instantiate a Flow with an Executor exposed to a specific port and block it for serving:

from jina import Flow

with Flow().config_gateway(protocol=['grpc'], port=12345) as f:
    f.block()

Then, you can use grpcurl sending status check request to the Gateway.

docker pull fullstorydev/grpcurl:latest
docker run --network='host' fullstorydev/grpcurl -plaintext 127.0.0.1:12345 jina.JinaInfoRPC/_status

The error-free output below signifies a correctly running Gateway:

{
  "jina": {
    "architecture": "######",
    "ci-vendor": "######",
    "docarray": "######",
    "grpcio": "######",
    "jina": "######",
    "jina-proto": "######",
    "jina-vcs-tag": "######",
    "platform": "######",
    "platform-release": "######",
    "platform-version": "######",
    "processor": "######",
    "proto-backend": "######",
    "protobuf": "######",
    "python": "######",
    "pyyaml": "######",
    "session-id": "######",
    "uid": "######",
    "uptime": "######"
  },
  "envs": {
    "JINA_AUTH_TOKEN": "(unset)",
    "JINA_DEFAULT_HOST": "(unset)",
    "JINA_DEFAULT_TIMEOUT_CTRL": "(unset)",
    "JINA_DEPLOYMENT_NAME": "(unset)",
    "JINA_DISABLE_HEALTHCHECK_LOGS": "(unset)",
    "JINA_DISABLE_UVLOOP": "(unset)",
    "JINA_EARLY_STOP": "(unset)",
    "JINA_FULL_CLI": "(unset)",
    "JINA_GATEWAY_IMAGE": "(unset)",
    "JINA_GRPC_RECV_BYTES": "(unset)",
    "JINA_GRPC_SEND_BYTES": "(unset)",
    "JINA_HUBBLE_REGISTRY": "(unset)",
    "JINA_HUB_NO_IMAGE_REBUILD": "(unset)",
    "JINA_LOCKS_ROOT": "(unset)",
    "JINA_LOG_CONFIG": "(unset)",
    "JINA_LOG_LEVEL": "(unset)",
    "JINA_LOG_NO_COLOR": "(unset)",
    "JINA_MP_START_METHOD": "(unset)",
    "JINA_RANDOM_PORT_MAX": "(unset)",
    "JINA_RANDOM_PORT_MIN": "(unset)"
  }
}

Tip

You can also use it to check Executor status, as Executor’s communication protocol is gRPC.

Configure Gateway gRPC options#

The Gateway supports the grpc_server_options parameter which allows more customization of the gRPC server. The grpc_server_options parameter accepts a dictionary of gRPC configuration options which will be used to overwrite the default options. The gRPC channel used for server to server communication can also be customized using the grpc_channel_options parameter.

The default gRPC options are:

('grpc.max_send_message_length', -1),
('grpc.max_receive_message_length', -1),
('grpc.keepalive_time_ms', 9999),
# send keepalive ping every 9 second, default is 2 hours.
('grpc.keepalive_timeout_ms', 4999),
# keepalive ping time out after 4 seconds, default is 20 seconds
('grpc.keepalive_permit_without_calls', True),
# allow keepalive pings when there's no gRPC calls
('grpc.http1.max_pings_without_data', 0),
# allow unlimited amount of keepalive pings without data
('grpc.http1.min_time_between_pings_ms', 10000),
# allow grpc pings from client every 9 seconds
('grpc.http1.min_ping_interval_without_data_ms', 5000),
# allow grpc pings from client without data every 4 seconds

Refer to the channel_arguments section for the full list of available gRPC options.

Hint

Refer to the Configure gRPC Client options section for configuring the Client gRPC channel options. Refer to the Configure Executor gRPC options section for configuring the Executor gRPC options.

Use HTTP/WebSocket#

When using HTTP or WebSocket as the Gateway protocol, you can use curl to target the /status endpoint and get the Jina info.

curl http://localhost:12345/status

{
  "jina": {
    "jina": "######",
    "docarray": "######",
    "jina-proto": "######",
    "jina-vcs-tag": "(unset)",
    "protobuf": "######",
    "proto-backend": "######",
    "grpcio": "######",
    "pyyaml": "######",
    "python": "######",
    "platform": "######",
    "platform-release": "######",
    "platform-version": "######",
    "architecture": "######",
    "processor": "######",
    "uid": "######",
    "session-id": "######",
    "uptime": "######",
    "ci-vendor": "(unset)"
  },
  "envs": {
    "JINA_AUTH_TOKEN": "(unset)",
    "JINA_DEFAULT_HOST": "(unset)",
    "JINA_DEFAULT_TIMEOUT_CTRL": "(unset)",
    "JINA_DEPLOYMENT_NAME": "(unset)",
    "JINA_DISABLE_UVLOOP": "(unset)",
    "JINA_EARLY_STOP": "(unset)",
    "JINA_FULL_CLI": "(unset)",
    "JINA_GATEWAY_IMAGE": "(unset)",
    "JINA_GRPC_RECV_BYTES": "(unset)",
    "JINA_GRPC_SEND_BYTES": "(unset)",
    "JINA_HUBBLE_REGISTRY": "(unset)",
    "JINA_HUB_NO_IMAGE_REBUILD": "(unset)",
    "JINA_LOG_CONFIG": "(unset)",
    "JINA_LOG_LEVEL": "(unset)",
    "JINA_LOG_NO_COLOR": "(unset)",
    "JINA_MP_START_METHOD": "(unset)",
    "JINA_RANDOM_PORT_MAX": "(unset)",
    "JINA_RANDOM_PORT_MIN": "(unset)",
    "JINA_DISABLE_HEALTHCHECK_LOGS": "(unset)",
    "JINA_LOCKS_ROOT": "(unset)"
  }
}

Custom logging configuration#

The Custom logging configuration section describes customizing the logging configuration for all entities of the Flow. The Gateway logging can also be individually configured using a custom logging.json.yml file as in the below example. The custom logging file logging.json.yml is described in more detail in the Custom logging configuration section.

Python

from jina import Flow

f = Flow().config_gateway(log_config='./logging.json.yml')

YAML

jtype: Flow
gateway:
  log_config: './logging.json.yml'

Gateway#

Set protocol in Python#

Set protocol in YAML#

Enable multiple protocols#

Enable TLS for client traffics#

Enable in-Flow compression#

Get environment information#

Use gRPC#

Configure Gateway gRPC options#

Use HTTP/WebSocket#

Custom logging configuration#

See also#