Functionality decoupling into microservices - part 4

11 May 2023

In the previous post we implemented the Fruits microservice. However, it is still not delivered to our users. To make everything as smooth as possible, we will keep the current monolith endpoints and route them to the microservice - implementing a strangler pattern ¹. In the future, we plan to instruct the users (or the frontend clients we publish) to route directly to the microservice, saving us some bandwidth and resources. For that we will also implement logging that will indicate how many users still use old endpoints. Current code is implemented as tag 1.2 on GitHub here.

Previous parts:

• Part 1 - Monolith implementation
• Part 2 - Data split into Redis
• Part 3 - Microservice implementation

Routing the endpoints

The first endpoint that is worth routing is the update function (PUT and POST HTTP verbs). It's safest to write to both databases for now. That way we will have everything up to date on the monolith side to serve current up to date state but also have everything kept in the microservice. For that we need to generate a token for the user that will authenticate them on the microservice side. We will use the function we implemented in part 2.

We replace the function setInRedis in fruits/update.go to the following one:

func sendToFruitsMicroservice(id int, username string, fruit string, super bool) error {
    client := http.Client{
        Timeout: 30 * time.Second,
    }

    form := url.Values{"fruit": []string{fruit}}.Encode()
    req, err := http.NewRequest("PUT", config.FruitsEndpoint, strings.NewReader(form))
    if err != nil {
        return err
    }
    token, err := CreateTokenForFruits(id, username, super) // Shorthand for setting up claims map
    if err != nil {
        return err
    }

    req.Header.Add("X-Auth-Token", token)
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
    req.Header.Set("Content-Length", strconv.Itoa(len(form)))

    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    if resp.StatusCode >= 400 {
        return fmt.Errorf("fruits microservice returned %d", resp.StatusCode)
    }
    return nil
}

So now we reached version 1.3 of our monolith. Let's write some test to verify if everything is working as expected. We will put Fruits microservice into a container for easier testing.

FROM golang:1.20-alpine AS builder
WORKDIR /app
COPY . /app/
RUN go build

FROM alpine:latest
WORKDIR /app
COPY --from=builder /app/fruits_microservice /usr/bin/fruits_microservice
EXPOSE 8081
ENTRYPOINT [ "/usr/bin/fruits_microservice" ]

Test the container by building it, starting a Redis container in the same network, starting the fruits microservice container and sending a request. It should reply with an empty array or anything else if your Redis has any data.

$ docker build -t fruits_microservice:latest .
$ docker network create fruits
$ docker run -d --name redis --network fruits redis:latest
$ docker run -d -v $(pwd)/monolith.pem:/app/monolith.pem --network fruits -e REDIS_ENDPOINT=redis:6379 -p 8081:8081 --rm fruits_microservice:latest
$ curl http://localhost:8081/
[]

In the fruits microservice I added some logging functions to see what's going on. So now if everything is working as expected in the container, let's use the bound port of fruits microservice as the target address in the monolith. To prepare getters for testing, we can use a header inside HTTP request to choose if we want to use the old database or pull data from the microservice. We will use X-Prefer-Data header with values monolith or microservice for that (this header will be read by the monolith).

// router/fruits.go
func ListAllFruits(w http.ResponseWriter, r *http.Request) {
    // Select the source based on `X-Prefer-Data`
    var fruits_map map[string]string
    var err error

    if r.Header.Get("X-Prefer-Data") == "microservice" {
        // Call the microservice
        fruits_map, err = fruits.GetFruitsFromMicroservice()
    } else {
        fruits_map, err = fruits.GetFruits()
    }
    [...]
// fruits/passthrough.go (skipping error checks and some other lines)
func GetFruitsFromMicroservice() (map[string]string, error) {
    client := http.Client{}
    req, err := http.NewRequest("GET", config.FruitsEndpoint, nil)
    resp, err := client.Do(req)
    if resp.StatusCode >= 400 {
        // error
    }
    jsonFruits := make([]struct{...}, 0, 32)
    err = json.NewDecoder(resp.Body).Decode(&jsonFruits)
    fruits := make(map[string]string)
    for _, f := range jsonFruits {
        fruits[f.username] = f.fruit
    }
    return fruits, nil
}

In order to test the microservice, I wrote an extensive end-to-end test, that:

1. Spins up the containers for microservice and Redis,
2. Creates users in the monolith,
3. Sets the fruits for the users in the monolith that should be reflected in the microservice (including trying to set a special one without super status),
4. Checks if the fruits are set correctly in the monolith,
5. Checks if the fruits are set correctly in the microservice when requested to the monolith with special header,
6. Checks if the fruits are set correctly in the microservice using its endpoint,
7. Checks if the fruits are set correctly in Redis.

The test uses cascadia library to find HTML nodes after rendering the page. Find the directory with all the test steps here.

When we know that everything is working all right, we can then change the code a bit so that it primarily uses the microservice. As we switch the code we can bump our version to 1.2.1. This tag and commit c6f2721 show exactly what needs to be changed in the code. The tests after a small change are also still passing so we can assume that the switchover was successful. Because we still write to the monolith database as a secondary, we should be able to run migration again and not lose any data. Diff for the getter can be seen below. For update.go, function that sends the request to update the fruit to the microservice was moved just after querying the monolith database for the username. If microservice errors out now, the data in the monolith will not be updated.

    var fruits_map map[string]string
    var err error

-   if r.Header.Get("X-Prefer-Data") == "microservice" {
+   if r.Header.Get("X-Prefer-Data") == "monolith" {
+       w.Header().Set("X-Data-Source", "monolith")
+       fruits_map, err = fruits.GetFruits()
+   } else {
        // Call the microservice
        fruits_map, err = fruits.GetFruitsFromMicroservice()
        w.Header().Set("X-Data-Source", "microservice")
-   } else {
-       fruits_map, err = fruits.GetFruits()
-       w.Header().Set("X-Data-Source", "monolith")
    }

    if err != nil {

If we have enough logging in the previous version 1.2, we know if there were any errors regarding the submission of the records to the microservice. If you are sure there are none and you did the migration before, we can assume that the service is stable, so deploying 1.2.1 can be done transparently without losing any data. On the other hand, if you saw some errors, it's safer to do a small downtime, migrate the data again from SQL to Redis and spin up 1.2.1.

As we have the latest monolith version running and serving users, the next step is to make changes in the client side - to call the microservice directly using a token. I will cover it the next post.