Dynamic multi-modular SBT project

As the Delving Culture-Hub project grows, I found myself adding module definitions by hand to the SBT more often than what I’d wish for, and for each adjustment of a common value, having to go through all modules and adjust them.

So after a while, I wondered whether it wouldn’t be possible to discover modules dynamically, rather than maintaining them by hand. It turns out that this is a bit tricky, but still possible.

The main limitation is that as by default, projects are discovered via reflection, we can’t dynamically provision the projects of a build from the same Build definition. Or in other words, we need two separate Build files. As pointed out by Mark Harrah in the comments, we can override the projectDefinitions method and use it in order to define a simple dynamic build.

Let’s take a simplified example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15



import sbt._
import Keys._

object Build extends sbt.Build {

  def modules(base: File) = for (x <- (base / "modules").listFiles if x.isDirectory) yield
    Project(id = "project-%s".format(x.getName), base = x)

  def toRef(base: File) = modules(base).map { x => x: ProjectReference }

  override def projectDefinitions(baseDirectory: File) = modules(baseDirectory) ++ Seq(root(baseDirectory))

  def root(base: File) = Project(id = "root", base = file("."), aggregate = toRef(base))
}

The modules method takes care of looking at the contents of the modules directory, and turns each directory into a SBT project definition. We then override the projectDefinition method to replace it with those dynamic definitions.

In practice, things get a little more complicated, mainly when there are dependencies between projects. I found that it makes sense to declare modules that many others depend upon explicitly anyway, rather than trying to do the dependency game dynamically. I suppose that this would be possible to achieve, but may in practice become quickly somewhat complex, given the restrictions brought in by reflection.

Initially, I had also planned to declare custom settings of modules that require them via the build.sbt mechanism, i.e. having a custom build.sbt file at each module root and pass in settings there. However, this does not seem to fare to well in this case, because the resulting Build includes import definitions twice. If I get this part to work, I’ll write about it in another post.

Contents