Bash: Poor mans code generation

When it comes to generating code, I often find that the simplest code generation only needs variable replacement. There’s a cli utility that already facilitates this with environment variables, named envsubst, part of the gettext suite of tools.

In it’s most simple form, it takes a template file and replaces any variables contained within, and outputs the resulting text. It’s not super great if you have any dangling $ characters in your template, but for most purposes it’s going to work well.

Generating our service cmd/ folders

The first variable that we will need in our generated files, is the full import name that our project lives at. We can get this name from the go.mod file in the root of the project. Let’s start with a dynamic rule for make templates and go from there.

templates: $(shell ls -d rpc/* | sed -e 's/rpc\//templates./g')
	@echo OK.

templates.%: export SERVICE=$*
templates.%: export MODULE=$(shell grep ^module go.mod | sed -e 's/module //g')
templates.%:
	@echo templates: $(SERVICE) $(MODULE)

For each of our RPC services, a templates.{service} target is created, and for those, the SERVICE and MODULE environment variable are exported. When we’re going to be using envsubst, the environment variables will be available for replacement.

Let’s create our basic template for main.go files (save as templates/cmd_main.go.tpl):

package main

// file is autogenerated, do not modify here, see
// generator and template: templates/cmd_main.go.tpl

import (
	"log"
	"context"

	"net/http"

	_ "github.com/go-sql-driver/mysql"

	"${MODULE}/rpc/${SERVICE}"
	server "${MODULE}/server/${SERVICE}"
)

func main() {
	ctx := context.TODO()

	srv, err := server.New(ctx)
	if err != nil {
		log.Fatalf("Error in service.New(): %+v", err)
	}

	twirpHandler := ${SERVICE}.New${SERVICE_CAMEL}ServiceServer(srv, nil)

	http.ListenAndServe(":3000", twirpHandler)
}

As you can see, there is really only one issue with the replacements here, and that is the need to have a camel-cased name of the service we are writing. It is quite easy to provide it:

templates.%: export SERVICE_CAMEL=$(shell echo $(SERVICE) | sed -r 's/(^|_)([a-z])/\U\2/g')

The regular expression replacement is looking for two matches:

• either the beggining of the string (^) or an underscore (_)
• the character directly following the first match

And then replaces it with the uppercase version of this character. So stats becomes Stats, and hello_world would become HelloWorld, and so on. Let’s use this template to generate cmd/stats/main.go, and verify the output:

templates.%:
	mkdir -p cmd/$(SERVICE)
	envsubst < templates/cmd_main.go.tpl > cmd/$(SERVICE)/main.go

Looking at the main.go output produced, we can verify that everything works as expected:

	twirpHandler := stats.NewStatsServiceServer(srv, nil)

Generating the client for our service

Another simple template which we can generate is the Twirp client for our microservice.

package ${SERVICE}

// file is autogenerated, do not modify here, see
// generator and template: templates/client_client.go.tpl

import (
	"net/http"

	"${MODULE}/rpc/${SERVICE}"
)

func New() ${SERVICE}.${SERVICE_CAMEL}Service {
	return NewCustom("http://${SERVICE}.service:3000", http.Client{})
}

func NewCustom(addr string, client ${SERVICE}.HTTPClient) ${SERVICE}.${SERVICE_CAMEL}Service {
	return ${SERVICE}.New${SERVICE_CAMEL}ServiceJSONClient(addr, client)
}

No additional variables are required, so we can just add an additional envsubst line to the makefile:

templates.%:
	mkdir -p cmd/$(SERVICE) client/$(SERVICE)
	envsubst < templates/cmd_main.go.tpl > cmd/$(SERVICE)/main.go
	envsubst < templates/client_client.go.tpl > client/$(SERVICE)/client.go

Server implementation

From here on out we need to see what we can do to build the implementation of the server side of our service. The templating here will serve only as an initial stubbed version of a service implementation, as we will need to extend it with complexity of our own implementation.

Let’s start by adding an interface scaffolding utility to our build environment Dockerfile:

# Install interface scaffolder
RUN go get -u github.com/josharian/impl

Using josharian/impl, we can generate a scaffold of the interface which we need to satisfy. We can now create the server side template like this:

package ${SERVICE}

import (
	"context"

	"${MODULE}/rpc/${SERVICE}"
)

type Server struct {
}

func New(ctx context.Context) (*Server, error) {
	return &Server{}, nil
}

var _ ${SERVICE}.${SERVICE_CAMEL}Service = &Server{}

And then we can just append the output of impl after it, so the generated code immediately satisfies the service interface. We need to update the makefile target now, to invoke this template, as well as generate the function stubs for the StatsService interface.

templates.%:
	mkdir -p cmd/$(SERVICE) client/$(SERVICE) server/$(SERVICE)
	envsubst < templates/cmd_main.go.tpl > cmd/$(SERVICE)/main.go
	envsubst < templates/client_client.go.tpl > client/$(SERVICE)/client.go
	envsubst < templates/server_server.go.tpl > server/$(SERVICE)/server.go
	impl -dir rpc/$(SERVICE) 'svc *Server' $(SERVICE).$(SERVICE_CAMEL)Service >> server/$(SERVICE)/server.go

After checking that the stubs are generated, running make will successfully build our project. Looking at our service, we are left to implement the function defined in the server interface:

func (svc *Server) Push(_ context.Context, _ *stats.PushRequest) (*stats.PushResponse, error) {
	panic("not implemented") // TODO: Implement
}

We also want to generate our server.go file only if it doesn’t exist - we only want to scaffold it for the first time, and checking file existance in Makefile is difficult, so let’s move this into a bash script, and adjust the verbosity/output of the complete templates target:

templates.%:
	@mkdir -p cmd/$(SERVICE) client/$(SERVICE) server/$(SERVICE)
	@envsubst < templates/cmd_main.go.tpl > cmd/$(SERVICE)/main.go
	@echo "~ cmd/$(SERVICE)/main.go"
	@envsubst < templates/client_client.go.tpl > client/$(SERVICE)/client.go
	@echo "~ client/$(SERVICE)/client.go"
	@./templates/server_server.go.sh

The @ command suppresses the output of the command being run, and we output ~ file.go for each file being written. Our server_server.go.sh has checks for the existance of our server.go file:

#!/bin/bash
cd $(dirname $(dirname $(readlink -f $0)))

if [ -z "${SERVICE}" ]; then
	echo "Usage: SERVICE=[name] SERVICE_CAMEL=[Name] $0"
	exit 255
fi

OUTPUT="server/${SERVICE}/server.go"

# only generate server.go if it doesn't exist
if [ ! -f "$OUTPUT" ]; then
	envsubst < templates/server_server.go.tpl > $OUTPUT
	impl -dir rpc/${SERVICE} 'svc *Server' ${SERVICE}.${SERVICE_CAMEL}Service >> $OUTPUT
	echo "~ $OUTPUT"
fi

This should give us a good start for our scaffolding. The correct and most optimal way to plan your services would be to take your time planning the proto schema, and then let the scaffolding take you to a place where you only need to implement the defined RPCs.

With more fluid and changing requirements, you will not really take advantage of the code generation when you’ll be updating the proto file during development. Adding new RPC calls will mean you’ll have to define the new functions by hand, which isn’t really that bad with ~3 SLOC, but still isn’t ideal for anything other than a microservice with only a few function calls. Planning saves time.

This article is part of a Advent of Go Microservices book. I’ll be publishing one article each day leading up to christmas. Please consider buying the ebook to support my writing, and reach out to me with feedback which can make the articles more useful for you.

All the articles from the series are listed on the advent2019 tag.

While I have you here...

It would be great if you buy one of my books:

• Go with Databases
• Advent of Go Microservices
• API Foundations in Go
• 12 Factor Apps with Docker and Go

Feel free to send me an email if you want to book my time for consultancy/freelance services. I'm great at APIs, Go, Docker, VueJS and scaling services, among many other things.