Using CGO bindings under Alpine, CentOS and Ubuntu

Go is a programming language that interacts with C libraries pretty well. Nevertheless there are some pitfalls using proper bindings to C libraries within a Go program (referred to as CGO bindings), especially when it happens to us to cross-compile our Go programs for a variety of Linux distributions.

The very first decision to make is to either dynamically or statically link our final Go binary to the used C libraries.

Dynamic linking is done by the linker that includes the names of all used libraries within the final executable binary.During run-time these names are resolved against matching libraries already installed on the system.

On the contrary, when performing static linking the linker includes all used libraries within the executable binary as the final compilation step, thus the executable can be run on all platforms regardless of any pre-installed library.

Since Go is widely known and appreciated to produce a single static binary we forget about the dynamic linking in this blog and bring some light into static CGO cross-compilation

But wait, does Go really produce a static binary even without using CGO bindings at all?

Short answer: it depends.

If we make no use of the net and os/user packages Go will provide us with a pure static binary that runs on all platforms without any issues.

Otherwise we end up with a dynamically linked binary against libc (most likely glibc, the GNU Project's implementation of the standard C library), since these two packages use CGO internally!

Normally, this should not be a big problem since almost every Linux distribution is already equipped with a proper libc library.

Nevertheless, there are enough situations where this assumption does not hold, especially if we plan to use C libraries not included in libc.

Tip: if we plan to use net package but neither os/user nor any other CGO binding we can still produce a pure static binary by disabling CGO through the environment variable CGO_ENABLED=0 and using the -netgo flag, which instructs Go to use a pure Go net package implementation, i.e.

CGO_ENABLED=0 go build -netgo our-program.go

Everything gets easier having an example

Let's assume we are asked to write a Go program that is able to grab the properties from a given partition of the underlying system,

despite the fact that there are already such tools available, such as blkid or lsblk. But these binaries are dynamically linked and therefore must be installed together with all its dependencies, e.g. blkid requires the dynamically linked libblkid.so and libuuid.so C libraries.

Thus let`s assume further that our Go program should be self-contained, that is without any dynamical linked part, and therewith is a perfect candidate to run on every Linux distribution, especially within a Docker container that is built from scratch.

The most intended but fairly tough solution would be to implement this functionality in pure Go.

We do not go this way and rather use the static libblkid.a and libuuid.a C libraries from within our Go program, which we will include in our final binary.

This leads to a bunch of problems:

1. How to get proper libblkid.a and libuuid.a libraries?
2. How to burn them into the binary?
3. How to bind to them within our Go program?
4. How to call C functions using Go?
5. How to implement the overall functionality using those C functions?
6. How to build the binary from distinct Linux distributions?

Partition property grabber

Go enables us to call C functions of any C library by importing the package C annotated with corresponding #include and #cgo instructions.

For our partition property grabber example, which we are going to call xcgo (since I'm an employee at x-xellent and we are using CGO bindings :smirk:), this results in

/*
#cgo linux LDFLAGS: -lblkid -luuid
#include <blkid/blkid.h>
*/
import "C"

Here we refer to the blkid.h header through the #include instruction and provide the LD linker with the two libraries libblkid.a and libuuid.a through the #cgo linux LDFLAGS: instruction, which are all required by our implementation. Note that all C libraries are propagated to LD by replacing the lib prefix by a single l and omitting the file type suffix, e.g. libxyz.so.1.1.0 must be propagated by -lxyz.

If there is a corresponding dynamically linked libxyz.so as well as a statically linked libxyz.a available, the statically linked one has precedence and will be taken eventually.

Note that the annotations have to be commented out in order to not break the Go compilation.

We can now invoke all C functions declared within the blkid.h header by prefix them with C. within our Go program.

Tip: it is also feasible to implement the C functions directly within the annotation block, e.g. the following program will print Hello world to the console:

package main

/*
#include <stdio.h>
void hello_world() { printf("hello, world\n"); }
*/
import "C"

import (
    "fmt"
)

func main() {
    C.hello_world()
}

Our third and fourth questions should be answered now, except that we later need to ensure that the three artifacts libblkid.a, libuuid.a and blkid.h are available at the expected locations and also compatible with the used GCC.

Implementation

Now let's complete our implementation, which could be something like this (xcgo.go):

package main

/*
#cgo linux LDFLAGS: -lblkid -luuid
#include <blkid/blkid.h>
*/
import "C"

import (
    "fmt"
    "os"
    "strings"
    "unsafe"
)

func main() {
    v, err := lookup(os.Args[1], strings.ToUpper(os.Args[2]))
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(-1)
    }

    fmt.Println(v)
}

func lookup(device, label string) (string, error) {
    fmt.Printf("Lookup %q probe value of device %v\n", label, device)
    return lookupProbeValue(device, label)
}

func lookupProbeValue(devName, label string) (string, error) {
    blkidProbe := C.blkid_new_probe_from_filename(C.CString(devName))
    if blkidProbe == nil {
        return "", fmt.Errorf("failed to create new blkid probe")
    }
    defer C.blkid_free_probe(blkidProbe)

    if err := C.blkid_do_probe(blkidProbe); err != 0 {
        return "", fmt.Errorf("failed to probe")
    }

    var bufPtr *C.char
    err := C.blkid_probe_lookup_value(
        blkidProbe, C.CString(label),
        (**C.char)(unsafe.Pointer(&bufPtr)),
        nil,
    )
    if err != 0 {
        return "", fmt.Errorf("failed to lookup probe value")
    }
    return C.GoString(bufPtr), nil
}

This addresses our fifth question, which actually is not related to the goal of this blog at all.Nevertheless, let's have a short walk-through.

The main function expects two parameters to be passed by the user. The first one should be the partition device from which we want to see some property such as /dev/nvme0n1p1. The second parameter is the property itself, such as UUID, TYPE or LABEL.

The whole lookup is implemented by the lookupProbeValue method using the C library libblkid.a that in turn depends on libuuid.a. Thus, although we do not call any C method from libuuid.so we must provide it to LD in the C annotation block.

It more or less re-implements the corresponding function get_properties_by_blkid from util-linux in Go using CGO, we only have to take care of data type conversions from and to C.

For example, C does not have a dedicated string type such as Go, it uses rather an array of characters. Thus we need to call the built-in conversion function C.CString() to convert a Go string to a C character array. The opposite can be achieved by calling C.GoString().

Build the static binary

It's time to compile and test our little static xcgo program. This totally depends on our used Linux distribution.

We present various techniques for at least Alpine, CentOS and Ubuntu.

This is where Docker comes into play.

Alpine

Alpine does not use glibc, the standard C/C++ compiler used by most of the Linux distributions including CentOS and Ubuntu.Instead it uses musl-gcc, which is something like the lightweight little brother of glibc having correctness as its primary goal.

For us this means we have the least problems to compile xcgo. Here is a Dockerfile that answers our remaining questions 1, 2 and 6:

FROM alpine:3.8 as builder
RUN apk --update add \
    go \
    musl-dev
WORKDIR /app
RUN apk --update add \
    util-linux-dev
WORKDIR /app
COPY xcgo.go ./
RUN go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" xcgo.go

FROM scratch
COPY --from=builder /app/xcgo /
ENTRYPOINT ["/xcgo"]

In the first step xcgo is built by Go and musl-gcc under Alpine 3.8. For having libblkid.a, libuuid.a and blkid.h present we install the package util-linux-dev. The static binding, building and packaging of the three artifacts into one static binary is all done by:

go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" xcgo.go

Here we specify -ldflags to pass four options to Go’s linker:

1. -a forces to rebuild packages that are already up-to-date
2. -linkmode external instructs Go to always use the external linker, which is musl-gcc for Alpine
3. -extldflags '-static' instructs Go to pass the -static flag to the external linker, which in turn let musl-gcc build a static binary that also includes the three artifacts
4. -s strips the binary by omitting the symbol table and debug information
5. -w further strips the binary by also omitting the DWARF symbol table

The second step will make up our final image, which is built from scratch and only have the static cross-compiled xcgo binary included.

Here is an example of building and running xcgo within a Docker container:

docker build -t xcgo:alpine . && docker run --rm --privileged xcgo:alpine /dev/nvme0n1p1 UUID

It should output something like this:

Lookup "UUID" probe value of device /dev/nvme0n1p1
EEB3-29FA

Note that the container needs to be started in privileged mode since the access to a partition device needs proper permissions.

CentOS

As already mentioned CentOS does not use musl-gcc by default. We either need to install it first and then point to it when cross-compiling xcgo or use the default glibc C standard library.

We will show both variants:

musl-gcc

The following Dockerfile presents the first variant, using musl-gcc:

FROM centos:7 as builder
RUN yum update -y \
 && yum install -y \
    curl \
    gcc \
    make \
 && yum clean all \
 && curl -LSs https://dl.google.com/go/go1.11.4.linux-amd64.tar.gz -o go.tar.gz \
 && tar -xf go.tar.gz \
 && rm -v go.tar.gz \
 && mv go /usr/local/ \
 && chmod +x /usr/bin/gcc
ENV PATH=${PATH}:/usr/local/go/bin
RUN curl -LSs https://www.musl-libc.org/releases/musl-1.1.21.tar.gz -o musl.tar.gz \
 && tar -xvf musl.tar.gz \
 && cd musl-1.1.21 \
 && ./configure \
 && make \
 && make install \
 && cd .. \
 && rm -rf musl*
WORKDIR /app
COPY include/blkid.h /usr/local/musl/include/blkid/
COPY lib/libblkid.a lib/libuuid.a /usr/local/musl/lib/
COPY xcgo.go ./
ENV CC=/usr/local/musl/bin/musl-gcc
RUN go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" xcgo.go

FROM scratch
COPY --from=builder /app/xcgo /
ENTRYPOINT ["/xcgo"]

glibc

The second variant, using glibc, is achieved by the following Dockerfile:

FROM centos:7 as builder
RUN yum update -y \
 && yum install -y \
    curl \
    gcc \
    glibc-static \
 && yum clean all \
 && curl -LSs https://dl.google.com/go/go1.11.4.linux-amd64.tar.gz -o go.tar.gz \
 && tar -xf go.tar.gz \
 && rm -v go.tar.gz \
 && mv go /usr/local/ \
 && chmod +x /usr/bin/gcc
ENV PATH=${PATH}:/usr/local/go/bin
WORKDIR /app
COPY xcgo.go ./
RUN mkdir usr \
 && curl http://de.archive.ubuntu.com/ubuntu/pool/main/u/util-linux/libblkid-dev_2.20.1-5.1ubuntu20_amd64.deb -o usr/libblkid.deb \
 && curl http://de.archive.ubuntu.com/ubuntu/pool/main/u/util-linux/uuid-dev_2.20.1-5.1ubuntu20_amd64.deb -o usr/libuuid.deb \
 && ar x usr/libblkid.deb \
 && tar -x --xz -f data.tar.xz ./usr/lib/x86_64-linux-gnu/libblkid.a ./usr/include/blkid/blkid.h \
 && ar x usr/libuuid.deb \
 && tar -x --xz -f data.tar.xz ./usr/lib/x86_64-linux-gnu/libuuid.a \
 && mkdir -p /usr/include/blkid/ \
 && cp usr/include/blkid/blkid.h /usr/include/blkid/ \
 && cp usr/lib/x86_64-linux-gnu/libblkid.a usr/lib/x86_64-linux-gnu/libuuid.a /usr/lib64 \
 && rm -rf usr \
 && go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" xcgo.go

FROM scratch
COPY --from=builder /app/xcgo /
ENTRYPOINT ["/xcgo"]

Note that we need to download and extract the three artifacts libblkid.a, libuuid.a and blkid.h from a proper location,

since they are not included in any yum package nowadays.

Example

The following example works for both Dockerfiles:

docker build -t xcgo:centos . && docker run --rm --privileged xcgo:centos /dev/nvme0n1p1 UUID

Ubuntu

Ubuntu does not use musl-gcc by default too, thus as with CentOS we have also two variants to build our static Go program.

musl-gcc

The following Dockerfile presents the variant using musl-gcc:

FROM ubuntu:18.04 as builder
RUN apt update \
 && apt install -y \
    curl \
    gcc \
    make \
 && apt clean \
 && apt autoremove -y \
 && rm -rf /var/lib/apt/lists/* \
 && curl -LSs https://dl.google.com/go/go1.11.4.linux-amd64.tar.gz -o go.tar.gz \
 && tar -xf go.tar.gz \
 && rm -v go.tar.gz \
 && mv go /usr/local/ \
 && rm -rf /var/lib/apt/lists/*
ENV PATH=${PATH}:/usr/local/go/bin
RUN curl -LSs https://www.musl-libc.org/releases/musl-1.1.21.tar.gz -o musl.tar.gz \
 && tar -xvf musl.tar.gz \
 && cd musl-1.1.21 \
 && ./configure \
 && make \
 && make install \
 && cd .. \
 && rm -rf musl*
WORKDIR /app
COPY include/blkid.h /usr/local/musl/include/blkid/
COPY lib/libblkid.a lib/libuuid.a /usr/local/musl/lib/
COPY xcgo.go ./
ENV CC=/usr/local/musl/bin/musl-gcc
RUN go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" xcgo.go

FROM scratch
COPY --from=builder /app/xcgo /
ENTRYPOINT ["/xcgo"]

glibc

Our final Dockerfile shows the variant using glibc:

FROM ubuntu:18.04 as builder
RUN apt update \
 && apt install -y \
    curl \
 && apt clean \
 && apt autoremove -y \
 && rm -rf /var/lib/apt/lists/* \
 && curl -LSs https://dl.google.com/go/go1.11.4.linux-amd64.tar.gz -o go.tar.gz \
 && tar -xf go.tar.gz \
 && rm -v go.tar.gz \
 && mv go /usr/local/
ENV PATH=${PATH}:/usr/local/go/bin
WORKDIR /app
RUN apt update \
 && apt install -y \
    gcc \
    libblkid-dev \
 && apt clean \
 && apt autoremove -y \
 && rm -rf /var/lib/apt/lists/*
COPY xcgo.go ./
RUN go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" xcgo.go

FROM scratch
COPY --from=builder /app/xcgo /
ENTRYPOINT ["/xcgo"]

Unlike CentOS we do have the apt package libblkid-dev available, which includes the statically linked libblkid.a and libuuid.a libraries.

Example

The following example works for both Dockerfiles:

docker build -t xcgo:ubuntu . && docker run --rm --privileged xcgo:ubuntu /dev/nvme0n1p1 UUID