Go synchronization primitives

Go comes with two categories of synchronization primitives, traditional locking mechanisms like mutexes, and channels. When implementing bitcoin-network I decided to, as an experiment, go solely with channels and not utilize any of the traditional synchronization primitives. This was somewhat elegant in some situations, but really painful in others. One of the better ones were a straightforward module that already relied heavily on channels, the message dispatcher (essentially a demux). This is a module that reads messages from a single input channel and based on their type, forwards them to a channel to that type's subscriber. The (simplified) main loop looks something like this:

for {

	msg, ok := <-d.input
	if !ok {
		break
	}
	ch, found := d.subscribers[msg.Type]
	if found {
		ch <- msg
	}
}

With d being the dispatcher object:
type Dispatcher struct {
	subscribers map[string]chan<- Message
	input       <-chan Message
}


As long as all the subscribers are added to the map before the main loop is started, this module doesn't seem to need any form of synchronization. The issues start surfacing when looking at the shut down process. As it stands above, the dispatcher module can be shut down cleanly by closing the input channel. However, we have no feedback on this shut down, making it impossible to know how to handle the outgoing channels after the input has been closed.

Closing a channel from the reader side has the risk of crashing if the writer is still active. Shutting down the reader without closing the channel runs the risk of blocking the writer, causing us to leak memory and threads. Unless we let the dispatcher take ownership of the outgoing channels (ie. close them when shutting down), we can only solve this by adding a "shut down" method that blocks (or otherwise signals completion). As it greatly simplifies the downstream code to allow reuse of channels, I chose to go with the second option.

A more flexible approach than a shut down method is an Unsubscribe() method that removes one type from the subscribe map. Since a map structure is not thread safe, and we're not using any locks, the removal needs to happen in the main loop. We can achieve this by introducing a new structure and couple of new internal channels:
type unsubscription struct {
	key  string
	done chan<- bool
}

func (d *Dispatcher) Unsubscribe(key string) {
	done := make(chan bool)
	d.unsub <- unsubscription{
		key:  key,
		done: done,
	}
	<-done
}

New main loop:
for {
	select {
	case s := <- d.unsub:
		delete(d.subscribers, s.key)
		s.done <- true

	case msg, ok := <-d.input:
		if !ok {
			break
		}
		ch, found := d.subscribers[msg.Type]
		if found {
			ch <- msg
		}
	}
}

A call of Unsubscribe() will now block until the operation has completed, and as all the work is being done in the main loop, we don't have to worry about races. After unsubscribing all the subscribed keywords, it will be safe to close the subscription channels or just shut down the readers on those channels. This technique works, but we can already tell that more complex modules are going to need a lot of plumbing to get it right (look at the main loops here or here for examples). Even in the relatively simple dispatcher module we can already get simpler and more efficient code by utilizing a sync.Mutex to protect the subscribers map.

Obviously, this was all sort of a contrived example and the learnings don't transfer to channels in general, which certainly have their place alongside the traditional locking mechanisms (https://github.com/golang/go/wiki/MutexOrChannel provides some advice on when to use either mechanism). It does however show that the traditional locks still have a place in Go and that channel shut-down can get complicated quickly.