Extending pflag with environment variables

Chances are that if you’re writing Go code for a while now, your needs for command line arguments have grown beyond just what the standard library flag package is able to provide.

A quick list

To quickly list a few packages:

• namsral/flag - add environment variable parsing,
• spf13/pflag - use posix/gnu-style --flags,
• urfave/cli - adds a default action and command actions, env aliases,…
• spf13/cobra - an interface for cli commands like git and go tools,
• spf13/viper - a larger configuration management package supporting key/value stores like etcd…

And beyond that, there are more and more packages that are quite popular to handle a variety of use cases beyond what the standard library flag can provide. If you check you the Awesome Go Command Line list of packages, currently there are 41 packages listed. Now, when you’re looking for a package to handle command line flags, evaluating 41 of them is going to take it’s own damn time.

Let’s assume that we can take a package like spf13/pflag and build what we need upon it, while looking at what we like about some of the packages, and what we don’t.

Flags and types

I personally love the standard library interface for flags. You have two ways to declare a flag, one which takes a pointer receiver, or one which returns a pointer to the flag value. For small apps, returning pointers may be good enough, but as you grow you will tend to create structures for components of your app - a database connection flags structure would be a good example.

This is the function signature we are looking at:

func StringVar

func StringVar(p *string, name string, value string, usage string)

StringVar defines a string flag with specified name, default value, and usage string. The argument p points to a string variable in which to store the value of the flag.

The flag package has functions for some common types (but not all), so we’re interested only in the *Var(...) functions, in order to define our various flags structures and just bind to them. What would that look like?

type (
	// Credentials contains DSN and Driver
	Credentials struct {
		DSN    string
		Driver string
	}

	// Options include database connection options
	Options struct {
		Credentials Credentials

		Retries        int
		RetryDelay     time.Duration
		ConnectTimeout time.Duration
	}
)

This would be a configuration options structure for a database package, e.g. db.Options. Ideally all you need to do from this point is to create an instance of Options{}, and bind individual values with the flag api.

func (options *Options) BindWithPrefix(prefix string) *Options {
	p := func(s string) string {
		if prefix != "" {
			return prefix + "-" + s
		}
		return s
	}
	flag.StringVar(&options.Credentials.Driver, p("db-driver"), "mysql", "Database driver")
	flag.StringVar(&options.Credentials.DSN, p("db-dsn"), "", "DSN for database connection")
	return options
}

As an example, as you might have a program which connects to multiple databases, I created a function which takes a prefix for the database credential flags. Let’s assume you have the services twitch and youtube, by binding the flags with a prefix, you can pass youtube-db-driver, youtube-db-dsn,…

The flag package lets you control your structures this way, but what about the others?

Taking a quick look at urfave/cli, from the v2 manual:

  var language string

  app := &cli.App{
    Flags: []cli.Flag {
      &cli.StringFlag{
        Name:        "lang",
        Value:       "english",
        Usage:       "language for the greeting",
        Destination: &language,
      },
    },
    Action: func(c *cli.Context) error {
      name := "someone"
      if c.NArg() > 0 {
        name = c.Args().Get(0)
      }
      if language == "spanish" {
        fmt.Println("Hola", name)
      } else {
        fmt.Println("Hello", name)
      }
      return nil
    },
  }

In this example:

1. an “app” is created, which defines 1 flag (“language”), and binds it with a reference (Destination: &language),
2. tight coupling over cli.App.Flags/cli.StringFlag - one package handling both flags and cli commands

Further more, the package also allows the declaration of the flags, without an explicit binding with a Destination field.

@@ -1,12 +1,9 @@
-  var language string
-
   app := &cli.App{
     Flags: []cli.Flag {
       &cli.StringFlag{
         Name:        "lang",
         Value:       "english",
         Usage:       "language for the greeting",
-        Destination: &language,
       },
     },
     Action: func(c *cli.Context) error {
@@ -14,7 +11,7 @@
       if c.NArg() > 0 {
         name = c.Args().Get(0)
       }
-      if language == "spanish" {
+      if c.String("lang") == "spanish" {
         fmt.Println("Hola", name)
       } else {
         fmt.Println("Hello", name)

This pattern has become what is generally known as “a bag of values”. Between it’s decleration and usage, the program isn’t sure that the types match, or that the name of the flag being read from matches the declared flag.

With explicit bindings, we know for sure that a field matches the name and the type being declared at compile time. What would happen if we made a typo to the c.String parameter? What would happen if we used c.Bool here? Would the program panic if the flag wasn’t declared as a Bool? Would we always get it’s uninitialized value?

If we can ensure compile-time safety, why would we want to move away from that?

Choose a package and build upon it

Let’s as an exercise consider parsing os.Environ() to populate our flags. We will take the spf13/pflag package to use posix/gnu-style flags and parse the environment into the command arguments.

Firstly, environment variables are usually uppercase, and delimited with an underscore, for example, DB_DSN. Posix flags in turn are lowercase and delimited with a single dash, so for this case, db-dsn would be the flag name. Let’s create a function which translates the environment name into the posix style flag name.

func flagNameFromEnvironmentName(s string) string {
	s = strings.ToLower(s)
	s = strings.Replace(s, "_", "-", -1)
	return s
}

Then - we need to figure out if a flag has already been passed to our program. If an environment variable is defined, but the command line flag is also defined - we want to ignore the environment variable in this case. We check for the prefix, as both --lang=en and --lang en are valid passed flags.

func containsFlag(haystack []string, needle string) bool {
	for _, v := range haystack {
		if strings.HasPrefix(v, needle) {
			return true
		}
	}
	return false
}

Now all that’s left is to parse the env variables and feed them into os.Args when appropriate.

func Parse() {
	for _, v := range os.Environ() {
		vals := strings.SplitN(v, "=", 2)
		flagName := flagNameFromEnvironmentName(vals[0])
		if fn := flag.CommandLine.Lookup(flagName); fn == nil {
			continue
		}
		flagOption := "--" + flagName
		if containsFlag(os.Args, flagOption) {
			continue
		}
		os.Args = append(os.Args, flagOption, vals[1])
	}
	flag.Parse()
}

We check the flag API (this works on flag as well as pflag) if a flag is defined with flag.CommandLine.Lookup; if it is not defined then we can ignore adding it onto flags (flag packages would throw an “unknown flag” error). After that we just check if the flag has been passed as a command line argument, and ignore environment values in this case.

Finally, we can check to see the environment vars being used for populating flags:

func main() {
        var lang string
        flag.StringVar(&lang, "lang", "", "Language")
        Parse()

        fmt.Println("Language:", lang)
}

And we can run our test a few times to see it works:

# go run main.go 
Language: en_US.UTF-8
# go run main.go --lang=slovenian
Language: slovenian
# go run main.go --lang SLO
Language: SLO
# LANG=SI go run main.go 
Language: SI

As the LANG environment is usually defined in your shell, that’s the first result when run with no arguments, but we explicitly check by passing LANG=SI as well. This also works with the stdlib flag package fully, as long as you don’t forget to replace the flagOption prefix from -- to -.

Using the package API

There are differences between the flag and spf13/pflag APIs. The pflag API actually allows us to optimize our code further. The Lookup function returns a pflag.Flag which we are using to find values based on environment field names. This value has additional fields which the standard library does not. We are interested in the following:

type Flag struct {
    Changed  bool    // If the user set the value (or if left to default)
    Value    Value   // value as set
    // omitted fields
}

The Changed field lets us know that a flag has been modified, which means that we can drop our check for passed flags in os.Args, including our contains function. The Value field already exists in the standard library, and is an interface which allows us to modify a flag value:

type Value interface {
    Set(string) error
    // omitted functions
}

So, we can now simply check if a flag has been modified, and update it’s value using the provided interface.

func Parse() error {
        for _, v := range os.Environ() {
                vals := strings.SplitN(v, "=", 2)
                flagName := flagNameFromEnvironmentName(vals[0])
                fn := flag.CommandLine.Lookup(flagName)
                if fn == nil || fn.Changed {
                        continue
                }
                if err := fn.Value.Set(vals[1]); err != nil {
                        return err
                }
        }
        flag.Parse()
        return nil
}

Thanks goes to /u/dave-sch which pointed out the extended API by pflag on reddit. Which begs the following question - why doesn’t flag.Parse() return an error? If you’re setting a complex value from a parameter, the Set() error signature suggests that parsing a flag value can fail. And if it does?

There’s an ErrorHandling value in the flag package which can be either ContinueOnError, ExitOnError or PanicOnError. The default is ExitOnError, which will invoke Usage func() defined on FlagSet (our flag.CommandLine). Just something to think about, if you want to build other things like commands on the flag package.

Wrapping up

All flag packages, in some way or another, basically take the command line arguments you provide and parse them to populate the flag values you have declared. This means, that you can provide your own config file reader or environment parsing on top of the standard library flag package, or spf13/pflag package.

Next time we’re going to look at adding the concept of “commands” into our very own cli package. We’ll look at some more examples in the wild, and figure out if we can take the good parts and nail down the API surface so it only provides what we need and doesn’t re-implement things that have already been solved well.

While I have you here...

It would be great if you buy one of my books:

• Go with Databases
• Advent of Go Microservices
• API Foundations in Go
• 12 Factor Apps with Docker and Go

I promise you'll learn a lot more if you buy one. Buying a copy supports me writing more about similar topics. Say thank you and buy my books.

Feel free to send me an email if you want to book my time for consultancy/freelance services. I'm great at APIs, Go, Docker, VueJS and scaling services, among many other things.