Comparing Event-Sourcing in F# and go - getting unstuck to ship that code
Hi,
last week I asked on Twitter:
Which of these projects does sound like most fun for you?
1) Help data scientists solve Python dependency mess in projects. 10+ VMs with Nix.
2) Porting logistic simulation from Python to C# for speed.
3) Comparing ES implementations in go and F#
4) Playing with new ML hardware
Comparison of ES implementations came up most often in the responses, so let's get back to it.
I'll provide you with a quick summary, then will talk about the place where I have been stuck for the last month or so.
What is this project about?
Working with startups showed me the value of achieving multiple objectives with a small amount of work (you might call it "the ultimate laziness"). This project is about doing a little work to:
learn new things and share them with the community;
practice event sourcing and inventory management domain in Trustbit;
raise awareness about the company brand;
perhaps run a very fun Hackathon later.
Here is how we could make this happen.
We publish a software kata for event sourcing in the area of inventory management. This kata includes domain, API and tests. It should be easy for the others to start working on the actual event-sourcing code right away. In a programming language of their choice.
All that is published as open source.
We also provide two sample solutions in different languages and programming approaches. We do the comparison: complexity, lines of code, performance and resistance to failures under the load.
If you decide to implement the kata as well, this would only improve the comparison.
I've been working on that project sporadically for 4 months using golang for this exercise (kata runner and actual implementation). Christian Folie joined along the way, focusing on domain coherence and F# implementation. You might have seen his last presentation with Daniel Weller on KanDDDinsky on "Juggling 300 integrations while scaling to 10M signals per day".
Currently the API is done, domain is fixed, tests are almost done, golang and F# implementations are almost done as well.
I've been stuck in that "almost" phase for a month...
Being stuck and not shipping
You see, I got carried away with my golang implementation. I wanted to get really nice performance results out of this implementation, just to prove a point. This meant optimising complex queries, going closer to the CPU and playing with the Data-oriented-programming.
In 3 minutes we can run a story of a growing company: 500k nested locations, 498k SKUs, 690k orders, more than a million entities and events🤯
All orders are resolved against strongly consistent availability engine. 1 thread 😉
The results are nice, but they come at a cost.
This caused the code to get more complicated to the point I can no longer work on it in small fast iterations. There are too many balls hanging in the air at once: event stream model, durable projection in SQLite and fast in-memory model for solving availability constraint riddles.
Ultimately, who cares about the performance, if that causes the project to stall and never be shipped? Nobody!
I've been working in startups for the most of my career, and should've learned that lesson by now. But playing with your own codebase on your own terms felt so nice for a change, that I got carried away.
Has this ever happened to you?
For me it is time to cut back the losses and get back on the track. Next steps:
- Roll back to the last commit before the performance tuning started.
- Carry over all new tests to that commit
- Finish the domain, even if it is horribly slow in some cases.
- Consider go implementation "good enough" and work to get the kata published.
I'll try to get as much done as possible before the update on the next weekend.
By the way my implementation is in golang with SQLite. Key takeways from this project so far:
- golang is a great little language. I have been underestimating its usefulness for complex business domains. It is more flexible and concise than it may look, even with all these "if err != nil" statements.
- SQLite is awesome. It is extremely fast and can even be made fault-resistant by using WAL replication.
- Nix and NixOS are becoming an indispensable tool in my workflows.
- gRPC is more practical than OpenAPI, in my opinion.
- ultimately, tech is irrelevant. Shipping is more important :)
I haven't seen the F# code, yet. So the most fun things take ways are still ahead!
See you next weekend.
With best regards,
Rinat
PS: Is there anything you want me to dive into in greater detail then? You could ping me on twitter or just reply to this email.