Dynamic multi-modular SBT project

As the Delving Culture-Hub project grows, I found myself adding module definitions by hand to the SBT more often than what I’d wish for, and for each adjustment of a common value, having to go through all modules and adjust them.

So after a while, I wondered whether it wouldn’t be possible to discover modules dynamically, rather than maintaining them by hand. It turns out that this is a bit tricky, but still possible.

The main limitation is that as by default, projects are discovered via reflection, we can’t dynamically provision the projects of a build from the same Build definition. Or in other words, we need two separate Build files. As pointed out by Mark Harrah in the comments, we can override the projectDefinitions method and use it in order to define a simple dynamic build.

Let’s take a simplified example:

[scala] import sbt._
import Keys._

object Build extends sbt.Build {

def modules(base: File) = for (x <- (base / "modules").listFiles if x.isDirectory) yield
Project(id = "project-%s".format(x.getName), base = x)

def toRef(base: File) = modules(base).map{x => x: ProjectReference}

override def projectDefinitions(baseDirectory: File) = modules(baseDirectory) ++ Seq(root(baseDirectory))

def root(base: File) = Project(id = "root", base = file("."), aggregate = toRef(base))
}
[/scala]

The modules method takes care of looking at the contents of the modules directory, and turns each directory into a SBT project definition. We then override the projectDefinition method to replace it with those dynamic definitons.

In practice, things get a little more complicated, mainly when there are dependencies between projects. I found that it makes sense to declare modules that many others depend upon explicitly anyway, rather than trying to do the dependency game dynamically. I suppose that this would be possible to achieve, but may in practice become quickly somewhat complex, given the restrictions brought in by reflection.

Initially, I had also planned to declare custom settings of modules that require them via the build.sbt mechanism, i.e. having a custom build.sbt file at each module root and pass in settings there. However, this does not seem to fare to well in this case, because the resulting Build includes import definitions twice. If I get this part to work, I’ll write about it in another post.


Liked this post? Subscribe to the mailing list to get regular updates on similar topics.

3 Comments on “Dynamic multi-modular SBT project”

  1. If you prefer, you can use the same Build by calling the super implementation:

    override def projects = modules ++ super.projects

    I’d actually recommend using the variant that provides the base directory, because there are situations where the current working directory (used to resolve a relative file) is not the base directory. So, it would look like:

    override def projectDefinitions(base: File) =
    modules(base) ++ projects

    and adjust modules to do base / “modules”.

    I don’t understand the problem with imports. Which ones get included twice?

    (Also, nice onLoadMessage.)

    1. Thanks for the feedback! I’ll try this out and update the post.

      About the imports: for example, if I move the settings of the cms module into its own build.sbt, I am getting the following error:


      /Users/manu/workspace/culture-hub/modules/cms/build.sbt:5: error: reference to routesImport is ambiguous;
      it is imported twice in the same scope by
      import _root_.sbt.PlayProject._
      and import _root_.play.Project._
      routesImport += "extensions.Binders._"
      ^
      /Users/manu/workspace/culture-hub/modules/cms/build.sbt:5: error: reassignment to val
      routesImport += "extensions.Binders._"
      ^
      [error] Type error in expression

      The setting causing this is:


      routesImport += "extensions.Binders._"

      The routesImport key is also used by another project definition in the same Build, which would explain the “reassignment to val” part of the message. And as far as I know, the build.sbt definitions import more or less everything in scope, so I suppose that would explain the duplicate import. Is there perhaps something special I have to do to make this work?

  2. It looks like that is a problem with play’s plugin. I think they deprecated sbt.PlayProject in favor of play.Project, but they are otherwise identical. They both extend Plugin and so they both get imported automatically. As with any ambiguous import, you can workaround it by explicitly referring to it, such as:

    play.Project.routesImport += “extensions.Binders._”

    I’m not sure if this is a well known problem or not, but you can probably follow up with the play project.

    The reassignment to val is a spurious error from scalac.

Leave a Reply

Your email address will not be published. Required fields are marked *