gRPC and Protobuffer API Documentation with proto2asciidoc and code2asciidoc

Introduction

In the past few days we have released two projects on Github, proto2asciidoc and code2asciidoc.

These projects came to fruition from a need to properly document new API’s being written internally with gRPC and Protobuffers.

However, we figured these tools might be useful for others so we have decided to release them under the MIT license to the public.

Due to the specific need of these tools, we also thought it would be nice to explain how they’re being used and how you yourself can use them in your next gRPC/Protobuffer project.

This article is a basic tutorial how to use gRPC/Protobuffer in conjunction with the tooling mentioned before to generate documentation along side of it. We will create a small API for a ToDo list.

If you however already know how to use gRPC/Protobuffer and have an existing project you can skip ahead to Generating Documentation

What is gRPC and/or Protobuffers

If you have never heard of gRPC and/or Protobuffers (Protocol Buffers) before here are two quotes from their respective project sites.

gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services.
— Why gRPC?, From the https://gRPC.io website

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
— What are protocol buffers?, From the https://developers.google.com/protocol-buffers website

Prerequisites

To follow along in this article you’ll need a few things, we won’t dive into how to setup those tools but will quickly sum up what is needed/recommended.

Go runtime: https://golang.org
Make and other build tools
- Debian: apt-get install build-essential
AsciiDoctor
- gem install asciidoctor
- (Optional) gem install asciidoctor-rouge for syntax highlighting
jq: https://stedolan.github.io/jq/
Editor with Go support
- I personally use VSCode: https://code.visualstudio.com

Getting Started

Create a directory on your machine that will be your workspace, from here on out this will be referred to as workspace.

Protobuffer

We will first start with a Protobuffer declaration, to find out more about it checkout the official website: https://developers.google.com/protocol-buffers

Inside your workspace create a directory called proto/ and there we create a file api.proto.

Header

The follow excerpts are needed as a base, we’ll go over the messages later.

api.proto header

1
2
3
4
5
6
7
8
9
syntax = "proto3"; // Protobuffer 3 syntax
package proto;     // Package name it compiles to, in our case `proto`

import "google/api/annotations.proto"; // Required for the REST JSON Gateway
import "google/protobuf/timestamp.proto"; /* A well known Protobuffer type from
                                             Google for exchanging timestamps */
import "github.com/gogo/protobuf@v1.3.1/gogoproto/gogo.proto"; /* We will
            be using the https://github.com/gogo/protobuf protobuffer extensions
            and the respective compiler. */

Messages

Every task needs to be somehow identified by the system when we later want to mark it as complete. We’ll use an UUID (Universally unique identifier) for this.

api.proto UUID

1
2
3
4
5
6
// tag::UUID[]
message UUID {
  string data = 1; /* We choose a string to store the UUID so when using the
                    REST JSON Gateway the UUID is normally readable. */
}
// end::UUID[]

For the Todo we want to be able to sort by priority so let’s create an Enum for that.

api.proto Priority

1
2
3
4
5
6
7
8
9
// tag::Priority[]
enum Priority {
  PRIORITY_UNSPECIFIED = 0;
  LOW = 1;
  MEDIUM = 2;
  HIGH = 3;
  URGENT = 4;
}
// end::Priority[]

We will also need an Empty message for listing our Todo’s later

api.proto Empty

1
2
3
// tag::Empty[]
message Empty{}
// end::Empty[]

And finally of course we need a Task to store inside a Todo List

api.proto Task

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
// tag::Task[]
message Task {
  UUID uuid = 1; /* Include the UUID message */
  string title = 2;
  string description = 3;
  Priority priority = 4; // Include the Priority enum
  bool completed = 5; // A boolean will suffice for marking as completed
  google.protobuf.Timestamp created_at = 6 [(gogoproto.stdtime) = true]; /*
    Timestamp for creation, `[(gogoproto.stdtime) = true]` allows us to use it
    as a normal Go Time value.
  */
  google.protobuf.Timestamp completed_at = 7 [(gogoproto.stdtime) = true]; /*
    Timestamp for when the task was marked as completed
  */
}
// end::Task[]

Service

Now we have the Messages and Enums sorted, let’s declare the API we’ll use for the Todo List API.

api.proto Todo Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// tag::Todo[]
service Todo {
  // tag::Create[]
  // Declare a new RPC endpoint for our method Create
  rpc Create(Task) returns (UUID) {
    option (google.api.http) = { /* Declares we also want a REST JSON Gateway */
      post: "/v1/todo/create",  // The URL Endpoint for this RPC call
      body: "*" /* The body we expect when receiving over the REST JSON Gateway,
                   `*` denotes the full message */
    };
  }
  // end::Create[]

  // tag::Complete[]
  // Declare a new RPC endpoint for our method Complete
  rpc Complete(UUID) returns (UUID) {
    option (google.api.http) = {
      get: "/v1/todo/complete/{data}" // which accepts the UUID in the URL
    };
  }
  // end::Complete[]

  // tag::List[]
  // Declare a new RPC endpoint for our method List
  rpc List(Empty) returns (stream Task) {
    option (google.api.http) = {
      get: "/v1/todo/list"
    };
  }
  // end::List[]
// end::Todo[]

Now save your newly created proto/api.proto file and let’s move to compiling the Protobuffer to something we can use inside Go!

Compiling

Because this is something you might be doing more often in the future, it’s recommended to create a Makefile. Create one in the root of your workspace.

I won’t delve too deep into what it does, as that surpasses the goal of this tutorial quite far. Feel free to read up the docs of https://github.com/gogo/protobuf for more information, or even the official Protoc compiler: https://developers.google.com/protocol-buffers/docs/downloads

To make it easier to get going, I’ve included a few other steps.

Caution:

Be sure to uncomment the correct PROTOC_ZIP version for your OS!

Makefile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
PROTOC_VERSION := 3.7.1
# uncomment the one for your OS
#PROTOC_ZIP := protoc-${PROTOC_VERSION}-linux-x86_64.zip
#PROTOC_ZIP := protoc-${PROTOC_VERSION}-osx-x86_64.zip
#PROTOC_ZIP := protoc-${PROTOC_VERSION}-win64.zip

API_DIR := proto
API_OUT := api.pb.go
API_REST_OUT := api.pb.gw.go

MODPATH := ${GOPATH}/pkg/mod

GOGOPROTO = $(shell go list -m -json github.com/gogo/protobuf | jq -r .Dir)
GOGOPROTOVERSION = $(shell go list -m -json github.com/gogo/protobuf | jq -r .Version | sed 's/\./\\./g')
GRPC-GW = $(shell go list -m -json github.com/grpc-ecosystem/grpc-gateway | jq -r .Dir)

${API_DIR}/api.pb.go: ${API_DIR}/api.proto
# we need to set the correct version
	@sed -i'' 's/gogo\/protobuf\//gogo\/protobuf\@${GOGOPROTOVERSION}\//' ${API_DIR}/api.proto
# the tabs are very important, it's a SINGLE line, do NOT tab in the M's
# also the last COLON : is to signify the output target
	@protoc/bin/protoc -I=${API_DIR}/ \
	-I=${MODPATH} \
	-I=${GOGOPROTO}/gogoproto \
	-I=${GOGOPROTO}/protobuf \
	-I=${GRPC-GW}/third_party/googleapis \
	--gogofaster_out=plugins=grpc,\
	Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,\
	Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,\
	Mgoogle/protobuf/struct.proto=github.com/gogo/protobuf/types,\
	Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,\
	Mgoogle/protobuf/wrappers.proto=github.com/gogo/protobuf/types:${API_DIR} \
	${API_DIR}/api.proto

${API_DIR}/api.pb.gw.go: ${API_DIR}/api.proto
	@protoc/bin/protoc -I=${API_DIR}/ \
		-I=${MODPATH} \
		-I=${GOGOPROTO}/protobuf \
		-I${GRPC-GW}/third_party/googleapis \
		--grpc-gateway_out=logtostderr=true:${API_DIR} \
		${API_DIR}/api.proto

${API_DIR}/gen: ${API_DIR}/api.pb.go ${API_DIR}/api.pb.gw.go ## Auto-generate grpc go sources
	@touch ${API_DIR}/gen

dep: ## Get the dependencies
	@go get github.com/productsupcom/proto2asciidoc
	@go get github.com/productsupcom/code2asciidoc
	@go get github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
	@go get github.com/golang/protobuf/protoc-gen-go
	@go get github.com/yoheimuta/protolint/cmd/protolint
# needed to allow better Go structs from Protobufs
	@go get github.com/gogo/protobuf/proto
	@go get github.com/gogo/protobuf/protoc-gen-gogofaster
	@go get github.com/gogo/protobuf/gogoproto


protoc: ## get protobuffer
	@curl -OL https://github.com/google/protobuf/releases/download/v${PROTOC_VERSION}/${PROTOC_ZIP}
	@unzip -o ${PROTOC_ZIP} -d protoc/ bin/protoc
	@unzip -o ${PROTOC_ZIP} -d protoc/ include/*
	@rm -f ${PROTOC_ZIP}

After this has been copied we can start with the first step, getting the protoc command, we can now do:

Getting protoc

1
$ make protoc

Inside your workspace there is now a new directory called protoc/ that contains the compiler we need for the later steps.

Go Code

To fully demonstrate how the ToDo works we’ll create a small gRPC server. And to show the strengths of proto2asciidoc and code2asciidoc we’ll now dive in.

However, it will not explain specifically why we do certain things as that’s beyond the scope of this article.

Let’s create the basis for our server, we need a few directories. Switch to the root of your workspace and execute the following commands.

I personally prefer to create a file on disk before editing them inside an editor, but feel free to create the directories/files any way you see fit.

Creating the structure

1
2
3
$ mkdir -p cmd/server/internal/{config,server}
$ touch cmd/server/main.go
$ touch cmd/server/internal/server/server.go

We will also use Go Modules, it’s important to initialise this first. For the sake of this project we will use a simple namespace, e.g. acme/todo

Go mod init

1
$ go mod init acme/todo

Caution:

Ensure your Go path and Go bin path is correctly set! If not you can do the following:

Setting GOPATH, GOBIN and PATH

1
2
3
4
5
6
mkdir ~/go
# make sure your path is set correctly, Mac uses /Users/youruser
echo "export GOPATH=\"/home/youruser/go\"" >> ~/.profile
echo "export GOBIN=\"$GOPATH/bin\"" >> ~/.profile
echo "export PATH=\"$PATH:$GOBIN\"" >> ~/.profile
source ~/.profile

But first, to ensure you have all the required dependencies, run the following

Getting the dependencies

1
$ make dep

Now we can finally compile our Protobuf file into Go code:

Compiling the Protobuffer

1
$ make proto/gen

Two new files are now created, proto/api.pb.go and proto/api.pb.gw.go. Both files should never be modified by hand. For larger projects you can either choose to commit them to your repository or choose to ignore them. Either is accepted. I personally prefer to commit them for easily switching back to a version in case maybe dependencies changed.

Now we start with cmd/server/main.go, as this is the first entry into any Go program. It will not compile yet as things are still missing.

Go cmd/server/main.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
package main

import (
	"log"

	// note the namespace here starts with the previously named go mod init
	// we must import package server as we will initialise it here
	"acme/todo/cmd/server/internal/server"
)

func main() {
	// create a config so we know where we listen on
	// package server exposes struct Config
	conf := server.Config{
		ListenAddressForgRPC: "127.0.0.1:7777",
		ListenAddressForREST: "127.0.0.1:7778",
	}

	// we create a NewServer from package server
	s := server.NewServer(conf)

	// due to the nature of gRPC it's a blocking call, so we must run it inside
	// a Go routine
	go func(s *server.Server) {
		err := s.StartGRPCServer()
		if err != nil {
			log.Fatalf("Failed to start gRPC server: %s", err)
		}
	}(s)

	// and we also want a REST JSON Gateway
	go func(s *server.Server) {
		err := s.StartRESTServer()
		if err != nil {
			log.Fatalf("Failed to start REST Gateway for gRPC server: %s", err)
		}
	}(s)

	// the gRPC server is up and running
	if <-s.Up {
		log.Println("We are up and running")

		// enter an infinite loop as else the program would exit since our only
		// blocking call is inside a Go routine
		select {}
	}
}

Once that file has been created, we can create the server package. Let’s create /cmd/server/internal/server/server.go. Note the internal directory enforces the Go compiler that only your namespace is allowed to import and interact with it.

Go cmd/server/internal/server/server.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
package server

import (
	"context"
	"fmt"
	"log"
	"net"
	"net/http"


	"acme/todo/proto"

	"github.com/gogo/gateway"
	"github.com/grpc-ecosystem/grpc-gateway/runtime"

	"google.golang.org/grpc"
)

type Config struct {
	ListenAddressForgRPC string
	ListenAddressForREST string
}

type Server struct {
	grpcServer *grpc.Server
	conf       Config
	todos      map[string]*proto.Task
	Up         chan bool
}

func NewServer(conf Config) *Server {
	s := Server{
		conf: conf,
	}

	s.todos = make(map[string]*proto.Task)
	s.Up = make(chan bool)
	return &s
}

func (s *Server) StartGRPCServer() error {
	lis, err := net.Listen("tcp", s.conf.ListenAddressForgRPC)
	if err != nil {
		return err
	}

	// create a gRPC server object
	s.grpcServer = grpc.NewServer()

	proto.RegisterTodoServer(s.grpcServer, s)

	s.Up <- true
	log.Printf("starting HTTP/2 gRPC server, listening on: %s", lis.Addr().String())
	if err := s.grpcServer.Serve(lis); err != nil {
		return err
	}

	return nil
}

func (s *Server) StartRESTServer() error {
	ctx := context.Background()
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	// needed for the gogo extensions for time
	// https://github.com/gogo/protobuf/issues/212
	m := &gateway.JSONPb{}
	mux := runtime.NewServeMux(
		runtime.WithMarshalerOption(runtime.MIMEWildcard, m),
		runtime.WithProtoErrorHandler(runtime.DefaultHTTPProtoErrorHandler),
	)

	opts := []grpc.DialOption{grpc.WithInsecure()}

	err := proto.RegisterTodoHandlerFromEndpoint(ctx,
		mux,
		s.conf.ListenAddressForgRPC,
		opts,
	)
	if err != nil {
		return fmt.Errorf("could not register service Todo: %w", err)
	}

	lis, err := net.Listen("tcp", s.conf.ListenAddressForREST)
	if err != nil {
		return err
	}

	log.Println("starting HTTP/1.1 REST server")

	http.Serve(lis, mux)

	return nil
}

Now both files are there, it should be able to compile right? Let’s give it a try.

Compiling

1
2
3
4
5
$ go build -o server cmd/server/main.go
# acme/todo/cmd/server/internal/server
cmd/server/internal/server/server.go:48:28: cannot use s (type *Server)
        as type "acme/todo/proto".TodoServer in argument to "acme/todo/proto".RegisterTodoServer:
        *Server does not implement "acme/todo/proto".TodoServer (missing Complete method)

The bootstrap code I provided doesn’t yet include our methods we declared in the Protobuf package. That’s one of the nice features of gRPC/Protobuffer that it enforces these checks. You cannot accidentally ship an API with a key endpoint missing!

Let’s add the bare minimum needed so it compiles, again open up cmd/server/internal/server/server.go and add the following.

Go cmd/server/internal/server/server.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
func (s *Server) Create(ctx context.Context, in *proto.Task) (*proto.UUID, error) {
	log.Printf("Create received: \n%v\n", in)
	return &proto.UUID{}, nil
}


func (s *Server) Complete(ctx context.Context, in *proto.UUID) (*proto.UUID, error) {
	log.Printf("Complete received: \n%v\n", in)
	return &proto.UUID{}, nil
}


func (s *Server) List(in *proto.Empty, stream proto.Todo_ListServer) error {
	return nil
}

Let’s compile and now it should work without any problems. Let’s run it right after and see what it does.

Compiling

1
2
3
4
5
$ go build -o server cmd/server/main.go
$ ./server
2020/05/13 17:05:42 starting HTTP/1.1 REST server
2020/05/13 17:05:42 starting HTTP/2 gRPC server, listening on: 127.0.0.1:7777
2020/05/13 17:05:42 We are up and running

Awesome! Pat yourself on the back, you have a gRPC/Protobuffer server running.

Now open a second Terminal and try the following

First request

1
2
3
$ curl -d '{"title":"hello world", "description":"I am the first todo!"}' \
   localhost:7778/v1/todo/create
{}

That isn’t very exciting but in our first Terminal there should be an extra line

Terminal 1 ./server

1
2
2020/05/13 17:05:47 Create received:
title:"hello world" description:"I am the first todo!"

This is great, we now have a working API. Let’s add some basic functionality so we can interact with it.

Kill the ./server with Ctrl+C and let’s continue.

We know we have a UUID Message, so let’s make a few convenience methods for it.

Creating custom.go

1
$ touch proto/custom.go

Open up proto/custom.go and add the following:

Go proto/custom.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
// it has to be part of the proto package
package proto

import "github.com/google/uuid"

func (u *UUID) New() {
	u.Data = uuid.New().String()
}

func (u *UUID) FromString(data string) error {
	uuid, err := uuid.Parse(data)
	if err != nil {
		return err
	}
	u.Data = uuid.String()
	return nil
}

Open up cmd/server/internal/server/server.go again and change the following for Create:

Go Create cmd/server/internal/server/server.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
func (s *Server) Create(ctx context.Context, in *proto.Task) (*proto.UUID, error) {
	log.Printf("Create received: \n%v\n", in)

	// a new task doesn't have a UUID, so let's check
	if in.Uuid != nil {
		return nil, fmt.Errorf("UUID set, cannot create")
	}

	var newUUID proto.UUID // create a new UUID for a task
	newUUID.New()          // initialise it
	in.Uuid = &newUUID     // assign it to the task, which expects a pointer

	now := time.Now()   // create a Now timestamp
	in.CreatedAt = &now // attach it to the task, which expects a pointer

	// UUID contains a field called data which is the string value
	// we can use this to store it in a map
	s.todos[in.Uuid.Data] = in

	return in.Uuid, nil
}

Assuming your editor is smart you won’t have the do the following, but make sure that in cmd/server/internal/server/server.go the import was updated to include "time".

Go import cmd/server/internal/server/server.go

1
2
3
import (
  "time"
)

Awesome, now we can store the incoming Todo’s. This however is only persisted as long as the server is running. Storing it persistently is outside the scope of this article.

Let’s implement a way to see the current list. Open up cmd/server/internal/server/server.go again and change the following for List.

Go List cmd/server/internal/server/server.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
func (s *Server) List(in *proto.Empty, stream proto.Todo_ListServer) error {
	for _, task := range s.todos { // range over the todos
		if !task.Completed { // only send back if task is marked as incomplete
			err := stream.Send(task) // send back the task
			if err != nil {
				return err
			}
		}
	}

	return nil
}

Let’s now compile again and try it out, keep your second terminal ready.

Compile and Run

1
$ go build -o server cmd/server/main.go && ./server

In our second terminal:

Trying out the API

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ curl -s -d '{"title":"hello world", "description":"I am the first todo!"}' \
   localhost:7778/v1/todo/create | jq .
{
  "data": "9dad7872-33e3-462b-8616-010c6d938575"
}
$ curl -s localhost:7778/v1/todo/list | jq .
{
  "result": {
    "uuid": {
      "data": "9dad7872-33e3-462b-8616-010c6d938575"
    },
    "title": "hello world",
    "description": "I am the first todo!",
    "createdAt": "2020-05-13T15:47:11.383561502Z"
  }
}

Feel free to add more tasks and see the list grow :)

Kill the ./server with Ctrl+C and let’s continue. Now all there is to remain is setting a task to complete.

Open up cmd/server/internal/server/server.go again and change the following for Complete.

Go Complete cmd/server/internal/server/server.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
func (s *Server) Complete(ctx context.Context, in *proto.UUID) (*proto.UUID, error) {
	log.Printf("Complete received: \n%v\n", in)

	// a task to complete must have a UUID set
	if in == nil {
		return nil, fmt.Errorf("UUID unset, cannot complete")
	}

	if task, ok := s.todos[in.Data]; ok { // check if the UUID appears in the todo list
		completed := time.Now()       // timestamp when the task was completed
		task.CompletedAt = &completed // attach it to the task
		task.Completed = true         // mark the task as complete

		s.todos[in.Data] = task // overwrite the Task in the map
		return in, nil
	}

	return nil, fmt.Errorf("Task was not found in the Todo list")
}

Let’s try it out in our Terminal again, have your two ready.

Compile and Run

1
$ go build -o server cmd/server/main.go && ./server

In our second terminal:

Trying out the API

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
$ curl -s -d '{"title":"hello world", "description":"I am the first todo!"}' \
    localhost:7778/v1/todo/create | jq .
{
  "data": "ac8d5880-acff-4caa-a812-8d18687b9282"
}
$ curl -s localhost:7778/v1/todo/list | jq .
{
  "result": {
    "uuid": {
      "data": "ac8d5880-acff-4caa-a812-8d18687b9282"
    },
    "title": "hello world",
    "description": "I am the first todo!",
    "createdAt": "2020-05-13T16:01:50.653300473Z"
  }
}
$ curl -s localhost:7778/v1/todo/complete/ac8d5880-acff-4caa-a812-8d18687b9282 | jq .
{
  "data": "ac8d5880-acff-4caa-a812-8d18687b9282"
}
$ curl -s localhost:7778/v1/todo/list | jq .

Great! The API works. Now we can get to the documentation part why we’re all here!

Kill the ./server with Ctrl+C and let’s continue.

Generating Documentation

Installation

Now we can grab the tools and generate some API documentation, because we would like other people to use our awesome API too.

Getting proto2asciidoc

1
2
$ go get github.com/productsupcom/proto2asciidoc/cmd/proto2asciidoc
$ go get github.com/productsupcom/code2asciidoc/cmd/code2asciidoc

proto2asciidoc

Now to produce API docs, we have to create a base directory for it, inside your workspace. Afterwards we can run the generator right away.

proto2asciidoc expects the source to be an absolute path. It works without but the includes inside the generated AsciiDoc would be incorrect. Pass the --overwrite (or -f) flag to overwrite it.

Running proto2asciidoc

1
2
$ mkdir -p docs/generated
$ proto2asciidoc --source $(pwd)/proto/api.proto --out docs/generated/api.adoc --api-docs

You can now view the generated documentation as an AsciiDoc, however let’s generate it into a HTML page.

Let’s add some things to our Makefile

Makefile modifications

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
PROJECT_NAME := todo
PROJECT_REPO := acme/$(PROJECT_NAME)
PROJECT_URI := https://$(PROJECT_REPO)
PROJECT_AUTHOR := "Productsup GmbH"

proto2asciidoc := $(shell go list -m -json github.com/productsupcom/proto2asciidoc | jq -r .Dir)

ASCIIDOC_EXT := -r $(proto2asciidoc)/asciidoctor/extensions/proto2asciidoc-inline-macro.rb
ASCIIDOC_ATTRIBUTES := ${ASCIIDOC_EXT} \
	-a project-name=${PROJECT_NAME} \
	-a project-author=${PROJECT_AUTHOR} \
	-a project-repo=${PROJECT_URI} \
	-a version=${GIT_VERSION_NAME}
ASCIIDOC_STYLING := -a rouge-style=igorpro -a source-highlighter=rouge -a source-linenums-option

generatedocs:
	@proto2asciidoc --source ${CURDIR}/proto/api.proto --out docs/generated/api.adoc -f --api-docs

html: generatedocs
	@rm -fr html
	@mkdir html
	@asciidoctor ${ASCIIDOC_ATTRIBUTES} ${ASCIIDOC_STYLING} docs/generated/api.adoc -o html/api.html

Now we can do the following:

Generate HTML

1
make html

And now in your favorite browser open up html/api.html. As you’ll be able to see, all comments next to fields in the message are included. This API documentation is already quite complete as you know what to send and receive.

But we would like to include receive/response examples of the message in JSON format.

This is where code2asciidoc comes into play.

code2asciidoc

One of the things I personally like to have the most inside API documentation are samples that the API expects and can potentially receive. The problem is that usually during the lifespan of an API, things change and the docs can get out of sync to what the API becomes.

This is what code2asciidoc tries to solve. The samples are generated from live code that are run from a Go test suite. Meaning it’s syntax-checked and the result changes along with the API.

Let’s create a file for it, note it has to be postfixed with _test for Go to properly want to run them as a test file.

Go test for samples

1
2
touch proto/create_samples_test.go
touch proto/response_samples_test.go

Now again in your favorite editor, let’s add some code.

proto/create_samples_test.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
package proto

import (
	"testing"

	"github.com/productsupcom/code2asciidoc/sample"
)

func Test_TaskSample(t *testing.T) {
	// startapidocs Example Task
	// A simple example how to send a Todo to the API.
	//
	// startpostdocs Response
	// The response we get is a UUID, see <<UUID Response>>
	// endpostdocs
	// enddocs
	// tag::TaskSample[]
	task := Task{
		Title:       "Sample task",
		Description: "Some description for the task",
		Priority:    Priority_URGENT,
	}
	// end::TaskSample[]

	f, err := sample.Setup("create_samples.apisamples")
	if err != nil {
		t.Errorf("%v", err)
	}
	err = sample.CreateJsonSample(&task, "TaskSample", f)
	if err != nil {
		t.Errorf("%v", err)
	}
}

proto/response_samples_test.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
package proto

import (
	"testing"

	"github.com/productsupcom/code2asciidoc/sample"
)

func Test_UUIDResponse(t *testing.T) {
	// startapidocs UUID Response
	// A UUID format the API sends back after a Create
	// enddocs
	// tag::UUIDResponse[]
	var u UUID
	u.New()
	// end::UUIDResponse[]

	f, err := sample.Setup("response_samples.apisamples")
	if err != nil {
		t.Errorf("%v", err)
	}
	err = sample.CreateJsonSample(&u, "UUIDResponse", f)
	if err != nil {
		t.Errorf("%v", err)
	}
}

Now the code is in place, let’s modify the Makefile and add code2asciidoc

Makefile

1
2
3
4
5
6
7
8
generatedocs:
	@rm -f proto/*.apisamples
	@code2asciidoc --source ${CURDIR}/proto/create_samples_test.go \
	--out docs/generated/api/todo/create_samples.adoc --f --run --no-header
	@code2asciidoc --source ${CURDIR}/proto/response_samples_test.go \
	--out docs/generated/api/todo/response_samples.adoc --f --run --no-header
	@proto2asciidoc --source ${CURDIR}/proto/api.proto --out docs/generated/api.adoc -f --api-docs \
	--sample-files ${CURDIR}/docs/generated/api/todo/create_samples.adoc,${CURDIR}/docs/generated/api/todo/response_samples.adoc

Create a directory and run the make generatedocs

code2asciidoc

1
2
$ mkdir -p docs/generated/api/todo
$ make generatedocs

Now a new file appeared at docs/generated/api/todo/todo_samples.adoc. However we’re not done yet, that would be magical ;)

We have to create a file that proto2asciidoc can include where we shall include the generated file into. The reason for this is that you might want to generate more samples, and you don’t know which you want where.

For the one’s with a keen eye, we have added a --sample-files parameter to proto2asciidoc, this makes a variable inside the api.adoc file where in our case we can use :todo_samples: to refer to the filename.

Create a directory and a file that would contain information for your API, in our case we create the directory docs/api/todo and file docs/api/todo/create.adoc.

Create file and dir

1
2
$ mkdir -p docs/api/todo
$ touch docs/api/todo/create.adoc

Now in our favorite editor let’s add the following content to docs/api/todo/create.adoc.

docs/api/todo/create.adoc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
= Create

For the Todo API you can create a new Task by using this endpoint.

== Response
include::{response_samples}[tag=UUIDResponse]

== API Examples
// this file does not exist unless code2asciidoc has been run
// on proto/todo_samples_test.go
// run `make generatedocs`
include::{create_samples}[leveloffset=+1]

Now we can build our docs again and take a look at the output,

HTML building

1
$ make html

Your code now has documentation in the format your API expects.

Conclusion

This concludes this article, we hope it was useful and fun to follow. And we hope that in the future you’ll be more inclined to write verbose documentation using these tools.