Introduction
Go‘s standard library support for JSON is an expedient method of implementing configuration files. An application can define a configuration struct type containing all the tunable elements, and load JSON from a file into an instance of this struct, for example:
type Config struct {
ServerUrl string
APIKey string
MaxSessions int
}
func readConfig(filename string) (*Config, error) {
// initialize conf with default values.
conf := &Config{Url: "http://localhost:8080/", MaxSessions: 10}
b, err := ioutil.ReadFile("./conf.json")
if err != nil {
return nil, err
}
if err = json.Unmarshal(b, conf); err != nil {
return nil, err
}
return conf, nil
}
This is already an improvement over loading JSON into an untyped
dictionary structure, as would happen in Javascript or Python, in
that json.Unmarshal provides some minimal validation. If configured with:
{ "MaxSessions": "quite a few" }
json.Unmarshal will return a error indicating that "quite a few" is not
a valid integer, and the program can handle this error at startup rather
than later in the runtime.
Customizing
The Go JSON library supports Unmarshaling into custom types through
an interface type json.Unmarshaler:
type Unmarshaler interface {
UnmarshalJSON([]byte) error
}
If our config struct contains a field of a type satisfying json.Unmarshaler,
that field’s UnmarshalJSON method will be called to fill in the field.
This has a number of uses which can make config handling more pleasant.
Loading Values From Elsewhere
One issue with the above example is the API key in the config. Configs get copied and pasted and checked in to version control, and over time this runs the risk of leaking API keys. A simple solution is to have the API key stored in a separate file, and read it in, and store this filename in the config instead:
type Config struct {
...
APIKeyFile string
}
but this requires a separate step to load the API key from the file
and store it elsewhere. If you have multiple config values needing
the same handling, this gets repetitive. A custom Unmarshaler
streamlines this process dramatically.
type FileString string
func (f *FileString) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
f, err := ioutil.ReadFile(s)
if err != nil {
return err
}
val := strings.TrimSpace(string(f))
*f = FileString(val)
return nil
}
type Config struct {
// ...
APIKey FileString
// ...
}
Now, the Config APIKey field is a string read from the file named
in the config.
Validation and Parsing
The above example adds some extra error conditions to
loading the config, namely returning appropriate errors
if the API key file does not exist or can’t be read. We
can take this effect a step further and validate that
the ServerUrl parameter contains a valid URL using
the net/url library.
type Url string
func (u *Url) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
*u = Url(s)
_, err = url.Parse(s)
return err
}
type Config struct {
ServerUrl Url
// ...
}
With this, if the user supplies an invalid URL string for the URL
parameter, it will be flagged as an error. The ServerURL field
will always be populated with a valid URL string
Note that we are calling the parser above, but throwing away its results. Those results are useful, we should keep them around!
type Url struct { *net.URL }
func (u *Url) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
u.URL, err = url.Parse(s)
return err
}
Now, the ServerURL field is a struct embedding a *net.URL,
with all the fields and methods of that type.
The string value is available with .ServerUrl.String().
Bringing it all together
The above two techniques can be combined to load more complicated
configurations. For example, the crypto/tls library provides
a Config structure that many TLS-supporting libraries use. The
Config structure contains parsed forms of certificates and
CA certificates. With a custom Unmarshaler, we can have the
user supply file names for certificates and keys, and load them
into a tls.Config:
type certFiles struct {
KeyFile, CertFile string
}
type tlsConfigFiles struct {
CertFile string
KeyFile string
CACertFile FileString
}
type TLSConfig struct { *tls.Config }
func (t *TLSConfig) UnmarshalJSON(b []byte) error {
var cf tlsConfigFiles
err := json.Unmarshal(b, &cf)
if err != nil {
return err
}
pool := x509.NewCertPool()
if !pool.AppendCertFromPEM([]byte(cf.CACertFile)) {
return errors.New("invalid CA Cert")
}
cert, err := tls.LoadX509KeyPair(cf.CertFile, cf.KeyFile)
if err != nil {
return err
}
t.Config = &tls.Config{
RootCAs: pool,
Certificates: []Certificate{cert},
}
return nil
}
struct Config {
//...
TLS TLSConfig
}
With this, any library call needing a tls config can use conf.TLS.Config.
Conclusion
The Unmarshaler interface in the encoding/json Go
library is a powerful abstraction. It allows you to
intercept the parsing of a loosely-structured JSON
object and validate or transform the underlying values
into the form the application needs them.
Although this post was JSON-centric, analogous
Unmarshaler interfaces appear in the standard XML
library and the leading third party YAML libraries
so the above techniques would work for XML and YAML
configs.
Chris Mikkelson is a Senior Distributed Systems Engineer at Farsight Security, Inc.
