We're trying something new. Well, trying two new things.
We're starting to blog about Wattpad's journey into the world of microservices. And we're blogging on our own platform.
We're proud of the really hard problems we have to solve every day and w...
When Platform decided to embark on the journey towards a microservice architecture, we knew early on that simple unit testing would no longer be enough to ensure our services were production ready. As such, we began to write service tests. Last week, in case you missed it, we went over the concept of service tests. Today, we'll dive into a technique for writing test cases for processes that don't behave well when they share memory.
One of the most critical services we designed recently is to serve text to our readers and writers (see why I emphasized critical?). To make sure the service performed well at our scale, we decided to back it with groupcache - a distributed cache which also replicates hot data across nodes. For the purpose of this post, what we'll need to remember is that when a call is made to get data, a consistent hash of the key determines which cache node owns the data. If the node the request goes to is not the owner, it will request the data from the owner's cache to service the request. If the owner's cache misses, it will fetch the data from the database, write it through to its cache, then pass the data to the caller node.
Testing our text-cache service (a wrapper service around groupcache itself) sounds as easy as testing any cache library right? Try to get some data twice, the first time it will get the data from the database and store it in the cache and the second time your service should read directly from the cache.
К сожалению, это изображение не соответствует нашим правилам. Чтобы продолжить публикацию, пожалуйста, удалите изображение или загрузите другое.
This test case is completely valid, however, it's just not enough. What we're missing is a test for the actual group part of groupcache. When we noticed we were missing tests for the group part of groupcache, we decided to write a test like this:
К сожалению, это изображение не соответствует нашим правилам. Чтобы продолжить публикацию, пожалуйста, удалите изображение или загрузите другое.
It seems that this should work, however, when we implemented a similar test case, no matter who the data owner was (service1 or service2) the database would always be hit twice and the test would fail.
At first, it was assumed that we had a bug in the code and that the test case failing is a clear sign that we need to fix something. Unfortunately, after auditing the service carefully we couldn't find anything suspicious. Consciously being aware that once our program becomes a binary, any imported library is no different from our own code, we decided to also audit groupcache to see if there was anything in the library which would cause our test to fail.
