About software engineering

Standard

This blog post by Rob Pike, one of the fathers of the Go programming language, made me think in the shower so I wrote this list of things about building software:

  • as a software engineer you want to add value to your company building software solutions
  • software economy: you need to be agile to reduce time to market
  • devops is the culture where you build your solution AND ship it to production
  • also you are in charge of its maintenance - both infrastructure and code
  • since company requirements change fast, your code base must change to, and sometimes your infrastructure must too.
  • to adapt to these changes rapidly, you need to deliver those changes as fast as you can
  • if you want to be agile delivering those changes you need to automate as much as possible
  • Infrastructure-as-a-service providers (aka "The Cloud") automate building virtual infrastructure so use them
  • infrastructure-as-code tools automates defining the virtual infrastructure declaratively so use them
  • one of the best ways to improve and change the code is the red-green-refactor testing technique, therefore write good tests
  • Continuous Integration tools help you automating those tests
  • since you will probably work in a team, you want your solution to be as simplest as possible so it's understandable by the rest of the team
  • also the simpler the program, the easier is to test, refactor and change it.
  • making simple software solutions is hard, require years of practice and good tools
  • choosing the right programming language to build simple yet powerful tested software is key for success

I <3 Go

Some notes on Go error handling, FP and railway programming

Standard

These last months I've had the chance to implement some new microservices with the Social Point backend team. We have taken advantadge of some Go functional programming features to be more expressive when modeling our domain and be able to apply some techniques to make the code base simpler to understand, like implementing middlewares to decouple the domain logic from logging or tracing.

On the other hand, last year I attended Software Craftmanship BCN '16 conf and one of the talks I liked most was named Railway programming: all aboard! by Ignasi Marimon-Clos (thank you @ignasi35!) which explained how to use certain features found in some functional languages to avoid repeating error handling code bits all over your code base.

So with that in mind I tried to dive a little bit more in those concepts and I wrote some notes and references in this hackmd document to gather together techniques to apply this kind of approach when programming in Go.

I hope you find them interesting.

An Erlang journey, from start to... finish?

Standard

It's been more than a year since I wrote the last Erlang lines that went to production in a Social Point project and I thought that it would be cool to share the experience, so here it is. Also please keep in mind that this is my humble opinion, not from my employer.

First, we started with Erlang because we needed a high-concurrent platform to build a chat and some other real-time features for our games. Like many, did our research and found and studied ejabberd, but we found it too complex for our needs and decided to build it by ourselves from scratch.

So we went for it and invested a lot of time learning the things that surround the amazing world of distributed systems, concurrency and Erlang and OTP (the Open Telecom Platform): the behaviours, the BEAM VM, fault tolerance and the let it crash philosophy, functional programming, recursion, pattern matching, the concurrent programming model, the Actor Model VS the CSP model, the supervision tree, the benefits of message passing and immutability, distributed systems theory topics (high availability, replication, eventual consistency, CRDTs, consistent hashing rings, distributed hash tables, distributed process registers...), clustering BEAM VMs and epmd, benchmarking, OS tunning, compile/release/deploy strategies, dependency managers, remote debugging, learn and use amazing libraries (cowboy, ranch, riak core...), read some books and articles, went to a couple of Erlang User Conferences, did an in-house workshop with Loïc Hoguin...

Screenshot 2016-05-16 16.28.21

Lots and lots of really interesting and useful things there.

And the services we built worked rock-solid, no question.

But we found that to reach a good development speed and time-to-deploy a lot work had to be done, maybe too much. We found ourselves like we had hit a wall somehow. And we felt that it was not easy to introduce Erlang in other backend projects, there was too much friction. We couldn't engage the development team neither the systems team. We were fighting too much. And now I think it's somewhat related to the next tweet I posted the other day from the distributed framework Microsoft has built:

[tweet id="731560538543669250" align="center"]

Are you a distributed systems expert? Maybe, but I know certainly I'm not.

Sometimes it felt that one had to be almost an expert (maybe not a distributed systems one, but an expert nevertheless) to succeed in Erlang development. We wanted our whole development team to participate in this journey, but found that the developer experience wasn't very good, the documentation looked somewhat old, the syntax wasn't familiar, the open source ecosystem and libraries are very small, almost no one knows how to put a serious Erlang project in production, we couldn't find developers in Spain.... It was like we had a very powerful tool but that's very difficult to master. Remember the concepts I wrote in the second paragraph? Certainly they are not something that your learn in a week or a month.

So we tried, succeeded in some aspects, failed in others, and moved on.

As I said, learning Erlang brought a lot of knowledge and experiences about concurrency and distributed systems and many other things. We knew we wanted to move from some parts of our software from a stateless, request/response based, monocore model in PHP to a stateful multicore model and bring those big performance improvements we saw in some projects. We knew we wanted to implement new services with soft real-time capabilities because in game development this paradigm is key in making awesome game experiences.

And this is were Golang matched and now is succeeding in our company, because it was designed to be rapid: rapid to learn (the language syntax is super simple), rapid to develop (amazing standard library), rapid to build and deploy (compile times, no VM, just package and copy), rapid to execute (the performance is stunning) and rapid evolution (vibrant ecosystem, new release every 6 months).

We have been writing Golang code for almost a year and we are very happy about our development speed. We know it's not Erlang, the great platform which has been battle-tested for 30 years in production by some of the biggest telecoms, we don't have OTP and distribution and fault-tolerance for free so we have to do it ourselves if we want those features... but this is fine. Like everything, all decisions have trade-offs and we have decided to change, and in our experience, for the better.

Going back to Erlang, it seems that lately (i mean the last few years, since Erlang is from 1986!) Ericsson has taken more seriously some of these DX issues. They moved the code to GitHub and are more open to user's feedback, releases are faster and bring many goodies (maps, several performance improvements in many areas, new FSM behaviour)... Looks like exciting times are comming for the Erlang community. Maybe it's because the Elixir project, a Ruby-inspired language that compiles to the BEAM VM, is getting some traction lately and I think is trying in some ways to fix some of these Erlang developer experience issues. Or maybe not, anyway we all are lucky that it's happening. I haven't studied it deeply but it looks promising and its documentation looks very good. And the latest ElixirConfEU '16 videos look very interesting.

Finally I hope some day I can hack again in Erlang in my work time, because it's been an amazing experience and I think we are far better software engineers thanks to diving in it.

What i learnt today #1

Standard

I don't blog much, that's why i decided to do some microblogging series about things i learn from day to day. Hope someone likes it, anyway so i do ;-P

So i aliased today='vi ~/md/today/$(date "+%Y-%m-%d").md' from this tweet so i can take quick notes over the day quickly and easily with vim.

I also red about Project Orleans from Microsoft, which is more or less Erlang/OTP written in C# under the .NET framework umbrella with a better DX (Developer Experience). It started as a Azure cloud-based backend for the game Halo. Somewhat related i remembered one of the best talks i attented to at last year's Craft Conf from Caitie McCaffrey explaining the Saga Pattern and how they applied it in Halo. More info in this InfoQ article.

I still enjoy 3D FPS, i don't play anymore and i hope i don't play again (too addictive) but i enjoy watching videos from time to time, mostly Q3A 1vs1. I also whatch them in one of the Social Point's gym bicycles which is great because the time somewhat pass faster. I particularly have enjoyed The Story of Overwatch video series, which has great inside about how important is gameplay in shooters and how they have been inspired by the classics, and this latest DOOM video at 1080p from the NVidia guys.

Also i red somewhere that chapters 20 and 21 from the Google SRE book are a must if do microservices so i will take a look there. Now i'm in 4th chapter so i'll jump straight to there this weekend.

Finally there are some enthusiasts of the Rust language like Alberto Fernández and Guillem Nieto at Social Point backend team. Talking with Marcos Quesada i arrived at the conclusion that ideally we should learn all languages with concurrency primitives and pick the best one for the tool since all of them have its prons and cons. The Learn X in Y tutorials are great for having a bird-eye view if you are an experienced programmer. So far i think i can say i know Erlang and Golang, so others in my list are, in my order of preference: Elixir (since like Erlang compiles to the BEAM VM but looks like Ruby), Rust (i find its syntax ugly but being free of memory bugs at compile time seems amazing), Clojure (since i hadn't had the chance to learn LISP at the university), Java/Scala+Akka (JavaVM is everywhere and the ecosystem is huge), Pony (why not?) and Haskell (this is the last one because i don't know anything about Category theory). Oh, and this is why i didn't choose Stackless Python in the list nor NodeJS.