0

This is a "technology/concept identification" question.

I have a rather large project in a DCVS repository.

Build times are loooong (about an hour) from a full clean.

The project is already divided into subprojects, with incremental building. But it still takes a very a long time when switching branches, etc., since I'll often have to compile a lot.

I had an idea of caching the subproject outputs (I'm using Scala, so in this case, it's a bunch of .class files), and publishing them with the hash of the source as the key. Before a build, I would download any outputs that matches the hashes of any of the subprojects. The "pain" of building would be shared and reused across the organization, including the CI server.

I can't have been the first person to think about this.

Is there a name for this caching-by-source (rather than explicit version)? Is it commonly implemented? Any existing tools around this?


EDIT: I have found one example in scons, a tool used primarily for C/C++ builds.

  • I suspect that the build system is clever enough to only rebuild files that need rebuilding. It still could touch a number of class files to make sure that the changed sources are still correct with the regard of them. I'd rather throw some hardware on it: build on SSD + more RAM; it would probably cost 2-3 developer-weeks in money equivalent. – 9000 Dec 8 '14 at 20:08
  • 2
    Does sbt do what you want? And if not, why not? You might have to modify your build to create actual JARs for your sub-projects. But yes, the dependency management problem has been solved. – kdgregory Dec 8 '14 at 20:08
  • @9000, it is clever enough. That is what I meant by "incremental building". But even so, if I switch from branch A to B back to A, I have to do a lot of recompilation each step. And my coworkers have to as well. My idea was for a cache on our network that would recognize and pull down already built outputs, saving many minutes of build time. – Paul Draper Dec 8 '14 at 20:15
1

sbt

The standard build tool for scala is sbt (Scala Build Tool). Within sbt there are three ways to do dependencies:

  • Declarations in your project
  • Maven POM files
  • Ivy configuration

To use Maven, one would need to specify the resolvers for the location of your repository:

resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository"
resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

You could also override this by creating a local configuration file: ~/.sbt/repositories

[repositories]
local
my-maven-repo: https://example.org/repo
my-ivy-repo: https://example.org/ivy-repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]

If you really had to, you could also specify a url and set up some way to host it locally on a server you control. Note that this isn't an ideal solution

libraryDependencies += "slinky" % "slinky" % "2.1" from "https://slinky2.googlecode.com/svn/artifacts/2.1/slinky.jar"

However, this is only half of the battle. Sure you can pull the jars from some place, but you also need to put them there. If you are not putting the modules out there for someone else to download and use, they are going to have to recompile them.

You could set up a Sonotype nexus server and then publish to it:

publishTo := Some("Sonatype Snapshots Nexus" at "https://oss.sonatype.org/content/repositories/snapshots")

Or you could have a commonly mounted repository (I don't think this would be a best solution):

publishTo := Some(Resolver.file("file",  new File( "path/to/my/maven-repo/releases" )) )

You can configure it to publish some things here and some things there:

publishTo := {
  val nexus = "https://oss.sonatype.org/"
  if (isSnapshot.value)
    Some("snapshots" at nexus + "content/repositories/snapshots") 
  else
    Some("releases"  at nexus + "service/local/staging/deploy/maven2")
}

But the thing there is that you need to publish it in order for other people to download it.

maven

Maven is (or should be) well known to Java developers. It isn't something that is commonly used with scala. But, it is possible to get scala and maven to work together. Along with src/main/java you will have src/main/scala. You will also have a plugin in your .pom:

<plugin>
  <groupId>net.alchim31.maven</groupId>
  <artifactId>scala-maven-plugin</artifactId>
  <version>3.1.3</version>
  <executions>
    <execution>
      <goals>
        <goal>compile</goal>
        <goal>testCompile</goal>
      </goals>
    </execution>
  </executions>
</plugin>

At this point, you are using maven and you are deploying maven artifacts to the Sonotype nexus server.

general things

All of this, with either approach depends on something else though. You need to have your project modularized so that small jars can be built, compiled, and deployed to the artifact server.

If your project is not configured this way, you will need to change it so that small independent parts are distinct.

You may find that the easiest thing to do (so you don't have to remember to do a publish or deploy) is to set up a continuous integration server that is fetching changes from your version control server, building them, and automatically deploying successful builds to the server.

This also implies some degree of discipline with versioning of the artifacts. You may wish to look at semantic versioning. The identifier for the release info so that you don't step on each other's toes and deploy competing snapshots or overwrite someone else's version number. You will find yourself with versions like 1.0.3-pauld.0.1 so that they are namespaced.

And thus, you will specify with sbt or maven that this project uses some artifact at some version. That pre-release version number will likely have the branch or author name in it. It will be deployed to a server where you can then pull it. When someone else switches to that branch those modules will be available for the build tool to download and use without having to recompile them.

languages and tools

A key thing to remember here is that one doesn't deploy .class files. Instead, you are deploying .jars. Everything is built around either locally compiled .class or downloaded libraries. While one could conceive of doing this with other languages, with Java and Scala the IDEs would get very annoyed if the .class files and the .java or .scala files became out of sync. If you're using vim and javac or scalac, then you might find other tools that pull singly compiled assets useful.

However, I'm going to point out that this isn't as trivial as it would first appear. You will find things like Outer$Inner.class and Outer$1$3.class in instances where you've got inner classes (the second example is the compiled file for the third anonymous inner class in the first anonymous inner class in the class Outer). With anonymous inner classes, this is position dependent in the file - add some code and Outer$1.class becomes Outer$2.class. This makes caching by source rather problematic and may not work at all with many JVM derived languages. To give an example:

object App {

    def isEven(i: Int) = i % 2 == 0

    def isOdd(i: Int) = i % 2 == 1

    def main(args: Array[String]): Unit = {
        val n = (1 to 10).toList
        n.filter(isEven).flatMap(i => n.filter(isOdd).map(j => i * j))
    }

}

when compiled with scalac App.scala you get:

App$$anonfun$main$1.class
App$$anonfun$main$2$$anonfun$apply$1.class
App$$anonfun$main$2$$anonfun$apply$2.class
App$$anonfun$main$2.class
App$.class
App.class

And you can see the issue of the order of the class things in the code can easily change. This would make tools like scons much less useful and possibly even problematic.

  • Totally right about the caching-by-.scala file problem (Java has the same "problem", FYI, with inner classes and such). My caching would be by subproject. So I would be caching a bunch of .class files if a bunch of .scala files matched. – Paul Draper Dec 10 '14 at 2:01
  • @PaulDraper you'd still be fighting against the IDE in many cases. I've had eclipse completely get upset with me when someone checked in some .class files in the project and I did a sync and pulled those in the workspace. Sure, a clean build cleaned it, but then whats the point? – user40980 Dec 10 '14 at 2:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.