Smarter JSON Configs in Go

Introduction

Go‘s standard library support for JSONis an expedient method of implementing configuration files.An application can define a configuration struct typecontaining all the tunable elements, and load JSON from a file into aninstance of this struct, for example:

type Config struct {
ServerUrl   string
APIKey    string
MaxSessions int
}

func readConfig(filename string) (*Config, error) {
// initialize conf with default values.
conf := &Config{Url: "http://localhost:8080/", MaxSessions: 10}

b, err := ioutil.ReadFile("./conf.json")
if err != nil {
return nil, err
}
if err = json.Unmarshal(b, conf); err != nil {
return nil, err
}
return conf, nil
}


This is already an improvement over loading JSON into an untypeddictionary structure, as would happen in Javascript or Python, inthat json.Unmarshal provides some minimal validation. If configured with:

{ "MaxSessions": "quite a few" }

json.Unmarshal will return a error indicating that "quite a few" is nota valid integer, and the program can handle this error at startup ratherthan later in the runtime.

Customizing

The Go JSON library supports Unmarshaling into custom types throughan interface type json.Unmarshaler:

type Unmarshaler interface {
UnmarshalJSON([]byte) error
}


If our config struct contains a field of a type satisfying json.Unmarshaler,that field’s UnmarshalJSON method will be called to fill in the field.This has a number of uses which can make config handling more pleasant.

Loading Values From Elsewhere

One issue with the above example is the API key in the config. Configsget copied and pasted and checked in to version control, and over timethis runs the risk of leaking API keys. A simple solution is to havethe API key stored in a separate file, and read it in, and store thisfilename in the config instead:

type Config struct {
...
APIKeyFile string
}


but this requires a separate step to load the API key from the fileand store it elsewhere. If you have multiple config values needingthe same handling, this gets repetitive. A custom Unmarshalerstreamlines this process dramatically.

type FileString string

func (f *FileString) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
f, err := ioutil.ReadFile(s)
if err != nil {
return err
}
val := strings.TrimSpace(string(f))
*f = FileString(val)
return nil
}

type Config struct {
// ...
APIKey FileString
// ...
}


Now, the Config APIKey field is a string read from the file namedin the config.

Validation and Parsing

The above example adds some extra error conditions toloading the config, namely returning appropriate errorsif the API key file does not exist or can’t be read. Wecan take this effect a step further and validate thatthe ServerUrl parameter contains a valid URL usingthe net/url library.

type Url string

func (u *Url) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
*u = Url(s)
_, err = url.Parse(s)
return err
}

type Config struct {
ServerUrl  Url
// ...
}


With this, if the user supplies an invalid URL string for the URLparameter, it will be flagged as an error. The ServerURL fieldwill always be populated with a valid URL string

Note that we are calling the parser above, but throwing away its results.Those results are useful, we should keep them around!

type Url struct { *net.URL }

func (u *Url) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
u.URL, err = url.Parse(s)
return err
}


Now, the ServerURL field is a struct embedding a *net.URL,with all the fields and methods of that type.The string value is available with .ServerUrl.String().

Bringing it all together

The above two techniques can be combined to load more complicatedconfigurations. For example, the crypto/tls library providesa Config structure that many TLS-supporting libraries use. TheConfig structure contains parsed forms of certificates andCA certificates. With a custom Unmarshaler, we can have theuser supply file names for certificates and keys, and load theminto a tls.Config:

type certFiles struct {
KeyFile, CertFile string
}

type tlsConfigFiles struct {
CertFile   string
KeyFile    string
CACertFile FileString
}

type TLSConfig struct { *tls.Config }

func (t *TLSConfig) UnmarshalJSON(b []byte) error {
var cf tlsConfigFiles
err := json.Unmarshal(b, &cf)
if err != nil {
return err
}
pool := x509.NewCertPool()
if !pool.AppendCertFromPEM([]byte(cf.CACertFile)) {
return errors.New("invalid CA Cert")
}
cert, err := tls.LoadX509KeyPair(cf.CertFile, cf.KeyFile)
if err != nil {
return err
}
t.Config = &tls.Config{
RootCAs: pool,
Certificates: []Certificate{cert},
}
return nil
}

struct Config {
//...
TLS  TLSConfig
}


With this, any library call needing a tls config can use conf.TLS.Config.

Conclusion

The Unmarshaler interface in the encoding/json Golibrary is a powerful abstraction. It allows you tointercept the parsing of a loosely-structured JSONobject and validate or transform the underlying valuesinto the form the application needs them.

Although this post was JSON-centric, analogousUnmarshaler interfaces appear in the standard XMLlibrary and the leading third party YAML librariesso the above techniques would work for XML and YAMLconfigs.

Chris Mikkelson is a Senior Distributed Systems Engineer at Farsight Security, Inc.

‍