Introduction
The goprogramming language’sprofiling systemsupports aninteresting feature: http access. If your application alreadyhas an HTTP interface, and it uses the
net/http
default multiplexer(i.e., it calls
http.Handle()
and
http.ListenAndServe()
), enabling httpprofiling is as simple as:
import _ "net/http/pprof"
and the profiling APIs will be visible under
/debug/pprof/
alongsideyour application’s URL paths, where you can access them with:
go tool pprof http://host:port/debug/pprof/profile
While this is a slick way of profiling a runningprogram, the fact that this capability is installed byan
import
makes turning it off and on cumbersome: theimport either has to be added and removed, or placedin a separate file withbuild tagsto turn compilationon and off. If left in shipped code, it has the side effectof publishing the profiling hooks (with no access control)on the application http listener. This is far from ideal,but there are a few good workarounds.
Access Control
A simple and effective option is to put the pprof http serveron a separate port on localhost, separate from the applicationhttp server. If the application does not use the http defaultmultiplexer, starting the profiling http server is as simple as:
go func() {
log.Fatal(http.ListenAndServe("localhost:8081"))
}()
Otherwise, you can run the following additional setup prior toany http.Handle() calls:
pprofMux := http.DefaultServeMux
http.DefaultServeMux = http.NewServeMux()
go func() {
log.Fatal(&http.Server{
Addr: "localhost:8081",
Handler: pprofMux,
}.ListenAndServe())
}()
Either of the above snippets can be conditionally run, so theprofiling server may be turned on or off by command line flagsor application configuration. However, since they are nowonly locally accessible, there is no downside to leaving themon.
Remote Control
Now that you have the profiling hooks safely exposed via alocalhost-only interface, you can invoke the following:
go tool pprof http://localhost:8081/debug/pprof/profile
and profile the program with the local go tools. However, sincego compiles to a static binary which can be installed withoutany go-related dependencies, chances are you don’t have go toolswhere your program is running. But you probably have SSH, so:
ssh -L 8081:localhost:8081 user@remote-host
or if you are already logged in:
~C
-L 8081:localhost:8081
will make the local
go tool pprof
on your local machine profilethe remote code over the ssh tunnel.
Conclusion
If you are running go services on servers with only trusted localusers, exposing the profiling http port through port forwardinglimits access to the trusted local users, making it practicalto leave profiling on as just another administrative featureof the system. This allows the developer to observe and/ortroubleshoot the program under a real-world, and real, load.
Chris Mikkelson is a Senior Distributed Systems Engineer at Farsight Security, Inc.
Heading
