Serve#

Executors can be served and accessed over gRPC, allowing you to use them to create a gRPC-based service for various tasks such as model inference, data processing, generative AI, and search services.

There are different options for deploying and running a standalone Executor:

See also

Executors can also be combined to form a pipeline of microservices. We will see in a later step how to achieve this with the Flow

Served vs. shared Executor

In Jina there are two ways of running standalone Executors: Served Executors and shared Executors.

  • A served Executor is launched by one of the following methods: Deployment, to_kubernetes_yaml(), or to_docker_compose_yaml(). It resides behind a Gateway and can thus be directly accessed by a Client. It can also be used as part of a Flow.

  • A shared Executor is launched using the Jina CLI and does not sit behind a Gateway. It is intended to be used in one or more Flows. Because a shared Executor does not reside behind a Gataway, it cannot be directly accessed by a Client, but it requires fewer networking hops when used inside of a Flow.

Both served and shared Executors can be used as part of a Flow, by adding them as an external Executor.

Serve directly#

An Executor can be served using the Deployment class.

The Deployment class aims to separate the deployment configuration from the serving logic. In other words:

  • the Executor cares about defining the logic to serve, which endpoints to define and what data to accept.

  • the Deployment layer cares about how to orchestrate this service, how many replicas or shards, etc.

This separation also aims to enhance the reusability of Executors: the same implementation of an Executor can be served in multiple ways/configurations using Deployment.

from docarray import DocumentArray, Document
from jina import Executor, requests, Deployment


class MyExec(Executor):
    @requests
    def foo(self, docs: DocumentArray, **kwargs):
        docs[0].text = 'executed MyExec'  # custom logic goes here


with Deployment(uses=MyExec, port=12345, replicas=2) as dep:
    docs = dep.post(on='/foo', inputs=DocumentArray.empty(1))
    print(docs.texts)

executor.yaml:

jtype: MyExec
py_modules:
    - executor.py

from jina import Deployment

with Deployment(uses='executor.yaml', port=12345, replicas=2) as dep:
    docs = dep.post(on='/foo', inputs=DocumentArray.empty(1))
    print(docs.texts)

from jina import Deployment

with Deployment(uses='jinaai://my-username/MyExec/', port=12345, replicas=2) as dep:
    docs = dep.post(on='/foo', inputs=DocumentArray.empty(1))
    print(docs.texts)

from jina import Deployment

with Deployment(uses='docker://my-executor-image', port=12345, replicas=2) as dep:
    docs = dep.post(on='/foo', inputs=DocumentArray.empty(1))
    print(docs.texts)

─────────────────────── 🎉 Deployment is ready to serve! ───────────────────────
╭────────────── 🔗 Endpoint ────────────────╮
│  ⛓     Protocol                    GRPC  │
│  🏠       Local           0.0.0.0:12345   │
│  🔒     Private     192.168.3.147:12345   │
│  🌍      Public    87.191.159.105:12345   │
╰───────────────────────────────────────────╯
['executed MyExec']

Hint

You can use dep.block() to serve forever:

with Deployment(uses=MyExec, port=12345, replicas=2) as dep:
    dep.block()

The Deployment class accepts configuration options similar to Executor configuration with Flows.

See Also

For more details on these arguments and the workings of a Flow, see the Flow section.

Serve via Kubernetes#

You can generate Kubernetes configuration files for your containerized Executor by using the static Executor.to_kubernetes_yaml() method. This works like deploying a Flow in Kubernetes, because your Executor is wrapped automatically in a Flow and uses the very same deployment techniques.

from jina import Executor

Executor.to_kubernetes_yaml(
    output_base_path='/tmp/config_out_folder',
    port_expose=8080,
    uses='jinaai+docker://jina-ai/DummyHubExecutor',
    executor_type=Executor.StandaloneExecutorType.EXTERNAL,
)

kubectl apply -R -f /tmp/config_out_folder

The above example deploys the DummyHubExecutor from Executor Hub into your Kubernetes cluster.

Hint

The Executor you use needs to be already containerized and stored in a registry accessible from your Kubernetes cluster. We recommend Executor Hub for this.

External and shared Executors#

This type of standalone Executor can be either external or shared. By default, it is external.

  • An external Executor is deployed alongside a Gateway.

  • A shared Executor has no Gateway.

Both types of Executor can be used directly in any Flow. Having a Gateway may be useful if you want to access your Executor with the Client without an additional Flow. If the Executor is only used inside other Flows, you should define a shared Executor to save the costs of running the Gateway in Kubernetes.

Serve via Docker Compose#

You can generate a Docker Compose service file for your containerized Executor by using the static to_docker_compose_yaml() method. This works like running a Flow with Docker Compose, because your Executor is wrapped automatically in a Flow and uses the very same deployment techniques.

from jina import Executor

Executor.to_docker_compose_yaml(
    output_path='/tmp/docker-compose.yml',
    port_expose=8080,
    uses='jinaai+docker://jina-ai/DummyHubExecutor',
)

docker-compose -f /tmp/docker-compose.yml up

The above example runs the DummyHubExecutor from Executor Hub locally on your computer using Docker Compose.

Hint

The Executor you use needs to be already containerized and stored in an accessible registry. We recommend Executor Hub for this.