Transcoding gRPC to HTTP/JSON using Envoy

When building a service in gRPC you define the message and service definition in a .proto file. gRPC generates client, server and DTO implementations automatically for you in multiple languages. At the end of this post you will understand how to make your gRPC API also accessible via HTTP JSON by using Envoy as a transcoding proxy. You can test it out yourself by running the Java code in the attached github repo. For a quick introduction on gRPC itself, please read gRPC as an alternative to REST.

Why transcode a gRPC service?

Once you have a working gRPC service, you can expose a gRPC service as an HTTP JSON API by simply adding some extra annotations to your service definition. Then you need a proxy that translates your HTTP JSON calls and passes them to your gRPC service. We call this process transcoding. Your service is then accessible via gRPC and via HTTP/JSON. I would prefer using gRPC most of the time because it’s more convenient and safer to work with type-safe generated code that follows the ‘contract’, but sometimes transcoding can come in handy:

Your webapp can talk to your gRPC service using HTTP/JSON calls. https://github.com/grpc/grpc-web is a JavaScript gRPC implementation that can be used from within the browser. This project is promising but is not yet mature.
Because gRPC uses a binary format on the wire, it can be hard to see what is actually being sent and received. Exposing it as an HTTP/JSON API makes it easier to inspect a service by using for example cURL or postman.
If you are using a language for which no gRPC compiler exists, you can access it via HTTP/JSON.
It paves the way for a smoother adoption of gRPC in your projects, allowing other teams to gradually transition.

Creating a gRPC service: ReservationService

Let’s create a simple gRPC service to use as an example. In gRPC, you define types and services containing remote procedure calls (rpc). You are free to design your services however you want, but Google recommends to use a resource oriented design (source: https://cloud.google.com/apis/design/resources) because it’s easier for users to understand your API without having to know what each method does. If you create a lot of rpc’s in loose format, your users have to understand what each method does making your API harder to learn. A resource oriented design also translates better to a HTTP/JSON API. In this example we will create a service that allows us to make reservations for meetings. This service is called ReservationService and consists of 4 operations to create, get, list and delete reservations. This is the service definition:

Listing 1. reservation_service.proto

syntax = "proto3";

package reservations.v1;
option java_multiple_files = true;
option java_outer_classname = "ReservationServiceProto";
option java_package = "nl.toefel.reservations.v1";

import "google/protobuf/empty.proto";

service ReservationService {

    rpc CreateReservation(CreateReservationRequest) returns (Reservation) {  }
    rpc GetReservation(GetReservationRequest) returns (Reservation) {  }
    rpc ListReservations(ListReservationsRequest) returns (stream Reservation) {  }
    rpc DeleteReservation(DeleteReservationRequest) returns (google.protobuf.Empty) {  }

}

message Reservation {
    string id = 1;
    string title = 2;
    string venue = 3;
    string room = 4;
    string timestamp = 5;
    repeated Person attendees = 6;
}

message Person {
    string ssn = 1;
    string firstName = 2;
    string lastName = 3;
}

message CreateReservationRequest {
    Reservation reservation = 2;
}

message CreateReservationResponse {
    Reservation reservation = 1;
}

message GetReservationRequest {
    string id = 1;
}

message ListReservationsRequest {
    string venue = 1;
    string timestamp = 2;
    string room = 3;

    Attendees attendees = 4;

    message Attendees {
        repeated string lastName = 1;
    }
}

message DeleteReservationRequest {
    string id = 1;
}

It is common practice to wrap the input for the operations inside a request object. This makes adding extra fields or options to your operation in the future easier. The ListReservations operation returns a stream of Reservations. In Java that means you will get an iterator of Reservation objects. The client can start processing the responses before the server is even finished sending them, pretty awesome :D. If you would like to see how this gRPC service can be used in Java, see(ServerMain.java and ClientMain.java for implementation.

Annotating the service with HTTP options for transcoding

Inside the curly braces of each rpc operation you can add options. Google defined an javaoption that allows you to specify how to transcode your operation to an HTTP endpoint. The option becomes available after importing google/api/annotations.proto inside reservation_service.proto. This import is not available by default, but you can make it available by adding the following compile dependency to build.gradle:

Listing 2. build.gradle

compile "com.google.api.grpc:proto-google-common-protos:1.13.0-pre2"

This dependency will be unpacked by the protobuf task and put several .proto files inside the build directory. You can now import google/api/annotations.proto inside your .proto file and start specifying how to transcode your API.

Transcoding the GetReservation operation as GET

Let’s start with the GetReservation operation, I’ve also added GetReservationRequest to the code sample for clarity:

message GetReservationRequest {
    string id = 1;
}

rpc GetReservation(GetReservationRequest) returns (Reservation) {
    option (google.api.http) = {
        get: "/v1/reservations/{id}"
    };
}

Inside the option definition there is one field named get set to /v1/reservations/{id}. The field name corresponds to the HTTP request method that should be used by the HTTP clients. The value of get corresponds to the request URL. Inside the URL we see a path variable called id. This path variable is automatically mapped to a field with the same name in the input operation. In this example that will be GetReservationRequest.id. Sending GET /v1/reservations/1234 to the proxy will transcode to the following pseudocode:

var request = GetReservationRequest.builder().setId(“1234”).build()
var reservation = reservationServiceClient.GetReservation(request)
return toJson(reservation)

The HTTP response body will be the JSON representation of all non-empty fields inside the reservation.

Remember, transcoding is not done by the gRPC service itself. Running this example on its own will not expose it as HTTP JSON API. A proxy in the front takes care of transcoding. We will configure that later.

Transcoding the CreateReservation operation as POST

Let’s now consider the CreateReservation operation.

message CreateReservationRequest {
   Reservation reservation = 2;
}

rpc CreateReservation(CreateReservationRequest) returns (Reservation) {
   option(google.api.http) = {
      post: "/v1/reservations"
      body: "reservation"
   };
}

This operation is transcoded to a POST on /v1/reservations. The field named body inside the option tells the transcoder to marshall the request body into the reservation field of the CreateReservationRequest message. This means we could use the following curl call:

curl -X POST
    http://localhost:51051/v1/reservations
    -H 'Content-Type: application/json'
    -d '{
    "title": "Lunchmeeting",
    "venue": "JDriven Coltbaan 3",
    "room": "atrium",
    "timestamp": "2018-10-10T11:12:13",
    "attendees": [
       {
           "ssn": "1234567890",
           "firstName": "Jimmy",
           "lastName": "Jones"
       },
       {
           "ssn": "9999999999",
           "firstName": "Dennis",
           "lastName": "Richie"
       }
    ]
}'

The response contains the same object, but with an extra generated ‘id’ field.

Transcoding ListReservations with query parameter filters

A common way of querying a collection resource is by providing query parameters as filter. This functionality corresponds to the ListReservations in our gRPC service. ListReservations receives a ListReservationRequest that contains optional fields to filter the reservation collection with.

message ListReservationsRequest {
    string venue = 1;
    string timestamp = 2;
    string room = 3;

    Attendees attendees = 4;

    message Attendees {
        repeated string lastName = 1;
    }
}

rpc ListReservations(ListReservationsRequest) returns (stream Reservation) {
   option (google.api.http) = {
       get: "/v1/reservations"
   };
}

Here, the transcoder will automatically create a ListReservationsRequest and map query parameters onto fields inside the ListReservationRequest for you. All the fields you do not specify will contain their default value, for strings this is “”. For example:

curl http://localhost:51051/v1/reservations?room=atrium

Will be mapped to a ListReservationRequest with the field room set to atrium, and the rest to their default values. It’s also possible to provide fields of sub-messages as follows:

curl "http://localhost:51051/v1/reservations?attendees.lastName=Richie"

And since attendees.lastName is a repeated field, it can be specified multiple times:

curl  "http://localhost:51051/v1/reservations?attendees.lastName=Richie&attendees.lastName=Kruger"

The gRPC service will see ListReservationRequest.attendees.lastName as a list with two items, Richie and Kruger. Supernice.

Running the transcoder

Now it’s time to actually get this thing working. The Google cloud supports transcoding, even when running in Kubernetes (incl GKE) or Compute Engine, for more information see https://cloud.google.com/endpoints/docs/grpc/tutorials. If you are not running inside the Google cloud or when you’re running locally, then you can use Envoy. Envoy is a very flexible proxy initially created by Lyft. It’s a major component in https://istio.io/ as well. We will use Envoy for this example. In order to to start transcoding we need to:

Have a project with a gRPC service, including transcoding options in the .proto files.
Generate .pd file from our .proto file that contains a gRPC service descriptor.
Configure envoy to proxy HTTP requests to our gRPC service using that definition.
Run envoy using docker.

Step 1

I’ve created the application described above and published it in github. You can clone it here: https://github.com/toefel18/transcoding-grpc-to-http-json. Then build it using

# Script will download gradle if it’s not installed, no need to install it :)
./gradlew.sh clean build    # windows: ./gradlew.bat clean build

I’ve created a script that automates steps 2 to 4. It is located in the root of https://github.com/toefel18/transcoding-grpc-to-http-json. This saves you a lot of time while developing. Steps 2 to 4 explain in greater detail what happens and how it works.

./start-envoy.sh

Step 2

Then we need to create .pb file. To do this, you need to download the precompiled protoc executable here https://github.com/protocolbuffers/protobuf/releases/latest (choose the correct version for your platform, i.e. protoc-3.6.1-osx-x86_64.zip for mac) and extract it somewhere in your path, that’s it, simple. Then running the following command inside the transcoding-grpc-to-http-json directory will result in a reservation_service_definition.pb file that envoy understands. (Don’t forget to build the project first to actually retrieve the required .proto files imported by reservation_service.proto)

protoc -I. -Ibuild/extracted-include-protos/main --include_imports \
               --include_source_info \
               --descriptor_set_out=reservation_service_definition.pb \
               src/main/proto/reservation_service.proto

This command might look complex but in reality it’s quite simple. The -I stands for include, these are directories where protoc will look for .proto files. --descriptor_set_out signifies the output file containing the definition and the last argument is the proto file we want to process.

Step 3

We are almost there, the last thing we need before running Envoy is to create its configuration file. Envoy’s configuration is described in yaml. There are many things you can do with Envoy, however, let’s now just focus on the minimum required to transcode our service. I took a basic config example from their website modified it a bit and marked the interesting parts with # markers.

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 } (1)

static_resources:
  listeners:
  - name: main-listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 51051 } (2)
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/", grpc: {} } (3)
                route: { cluster: grpc-backend-services, timeout: { seconds: 60 } }  (3)
          http_filters:
          - name: envoy.grpc_json_transcoder
            config:
              proto_descriptor: "/data/reservation_service_definition.pb" (4)
              services: ["reservations.v1.ReservationService"] (5)
              print_options:
                add_whitespace: true
                always_print_primitive_fields: true
                always_print_enums_as_ints: false
                preserve_proto_field_names: false (6)
          - name: envoy.router

  clusters:
  - name: grpc-backend-services (7)
    connect_timeout: 1.25s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: 127.0.0.1 (8)
        port_value: 53000

1	The address of the admin interface. You can also get prometheus metrics here to see how the service performs!!
2	The address at which the HTTP API will be available
3	by including an empty `grpc: {}` object, you tell the transcoder to only forward calls to the gRPC backend. Paths unknown to the gRPC service return 404. The name of the backend services to route requests to. Step defines this name.
4	The path to the .pb descriptor file we generated before
5	The services to transcode
6	Protobuf field names usually contain underscores. Setting this field to false will translate field names to camelcase.
7	A cluster defines upstream services (services that envoy can proxy to in step )
8	The address and port at which the backend services are reachable. I’ve used (127.0.0.1/localhost).

Step 4

We are now ready to run envoy. The easiest way to run envoy is by running the docker image. This requires that docker is installed. If you haven’t, please install docker first. There are two resources that Envoy needs, the config file, and .pb descriptor file. We can map these files inside the container so that envoy finds them when it starts. Run this command within the github repo root directory:

sudo docker run -it --rm --name envoy --network="host" \
  -v "$(pwd)/reservation_service_definition.pb:/data/reservation_service_definition.pb:ro" \
  -v "$(pwd)/envoy-config.yml:/etc/envoy/envoy.yaml:ro" \
  envoyproxy/envoy

If envoy started successfully you will see a log line at the end :

[2018-11-10 14:55:02.058][000009][info][main] [source/server/server.cc:454] starting main dispatch loop

Note that I set the --network to “host” in the docker run command. This means that the running container is accessible on localhost without additional network configuration. This works on Linux, but might not work on Windows and Mac. The docker pages suggest you should change the IP address in step of the envoy config to host.docker.internal or gateway.docker.internal according to: docs.docker.com/docker-for-mac/networking/.

Using your service via HTTP

If all goes well, you can now cURL your service using HTTP. On Linux, you can connect to localhost, but on windows or mac you might have to connect to the IP address of the VM or docker container. The examples use localhost as there are many ways you can configure docker.

Create a reservation via HTTP

curl -X POST http://localhost:51051/v1/reservations \
          -H 'Content-Type: application/json' \
          -d '{
            "title": "Lunchmeeting2",
            "venue": "JDriven Coltbaan 3",
            "room": "atrium",
            "timestamp": "2018-10-10T11:12:13",
            "attendees": [
                {
                    "ssn": "1234567890",
                    "firstName": "Jimmy",
                    "lastName": "Jones"
                },
                {
                    "ssn": "9999999999",
                    "firstName": "Dennis",
                    "lastName": "Richie"
                }
            ]
        }'

Example output:

 {
        "id": "2cec91a7-d2d6-4600-8cc3-4ebf5417ac4b",
        "title": "Lunchmeeting2",
        "venue": "JDriven Coltbaan 3",
...

Retrieve a reservation via HTTP

Use the ID that the POST created!

curl http://localhost:51051/v1/reservations/ENTER-ID-HERE!

The output should be the same as the result of the create

List reservationS via HTTP

For this example it might be nice to run CreateReservation multiple times with different fields to see the filter in action.

curl "http://localhost:51051/v1/reservations"

curl "http://localhost:51051/v1/reservations?room=atrium"

curl "http://localhost:51051/v1/reservations?room=atrium&attendees.lastName=Jones"

The response will be an array of Reservations.

DELETE a reservation

curl -X DELETE http://localhost:51051/v1/reservations/ENTER-ID-HERE!

Returned headers

gRPC returns several HTTP headers. Some might might help you with debugging:

grpc-status: the value is the ordinal value of io.grpc.Status.Code. It can come in handy to see what status gRPC returns.
grpc-message: the error in case something went wrong.

Visit for more info: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md

Imperfections

1. Weird responses if path does not exist.

Envoy does a good job but it sometimes returns, in my opinion, an incorrect status code. For example, when I GET a valid reservation:

curl http://localhost:51051/v1/reservations/ENTER-ID-HERE!

It returns 200, which is good. But then if I do this

curl http://localhost:51051/v1/reservations/ENTER-ID-HERE!/blabla

Envoy returns:

415 Unsupported Media Type

Content-Type is missing from the request

I expect 404 here, and the body doesn’t really explain the error well. I filed an issue here: https://github.com/envoyproxy/envoy/issues/5010 RESOLVED: Envoy was routing all requests to the gRPC service, if the path did not exist in the gRPC service, the gRPC service itself responded with that error. The solution is to make Envoy only forward requests that have an implementation in the gRPC service by adding grpc: {} to the configuration of envoy:

name: local_route
virtual_hosts:
- name: local_service
    domains: ["*"]
    routes:
    - match: { prefix: "/" , grpc: {}}  # <--- this fixes it
    route: { cluster: grpc-backend-services, timeout: { seconds: 60 } }

2. Sometimes when querying a collection the resource returns '[ ]' even when the server responds with an error.

I filed an issue with the envoy developers https://github.com/envoyproxy/envoy/issues/5011 PARTIAL SOLUTION: Part of this is a known limitation of transcoding as the status and headers are sent first. In a response. The transcoder starts by sending a 200 followed by transcoding the stream.

Upcoming features

In the future it will also be possible to provide subfields of the response message that you want to return in the response body, in case you do not want to return the complete response body. This can be done via a "response_body" field inside the HTTP option. This could be nice if you want to cut out a wrapper object in your HTTP API.

Final words

I hope this gives a good overview on transcoding a gRPC API to HTTP/JSON.