9

I'm trying to get a Java programming project set up in a way that it can be built completely offline just from the files in version control, on any machine, with no prerequisites to install. (Sole exception: the JDK)

I don't know much of Maven, but I'm learning Gradle currently. For the build I'm using Gradle with the wrapper, I modified the gradle-wrapper.properties so the distributionUrl doesn't point to a remote URL but instead a file within the project's folder. So Gradle itself is done already.

For dependencies though, this is a pure nightmare. I've already tried managing it with simply directly pointing to single JARs or collections of JARs, but as you sure know this ends up in total confusion if you have interdependencies and/or transitive dependencies.

Basically what I want to have is Gradle/Maven configured (just for this project) in such a way that they use a different directory for their local dependency cache - not one that is shared by anything you do on your user account, in your home directory, but instead a folder within the project folder (so I can check that into version control as well) which is only used for that one project. I would then only once take Gradle/Maven in online mode, run the build once to download all the currently needed dependencies into the local cache directory within the project folder (whose contents then are put into VC), then put it in offline mode for all subsequent builds.

I haven't found a way yet to reconfigure that local cache path just for this one project (has to be something I can just put into my build.gradle). Anyone have a lead for me?

Edit: Another thing would be to have this local repository within the project folder be a "real" Maven repository (instead of just a cache). But that would involve manual work for one way I imagined: You start a local copy of Sonatype Nexus that has a proxying repository set up for any Maven repos you're referring to in your build.gradle; you modify the repo URLs in your build script to point to your local Nexus, run the build once so Nexus downloads the deps and puts them into its work directory in the "usual" Maven repo structure; then you copy that structure to a folder within your project folder and change the buildscript repo URLs to point to the folder you just copied the files to. Quite a lot of work that can barely be automated.

5
  • Stuffing all those .JARs into a source-code repository bloats the repository and is explicitly not recommended for decentralized repositories like Git. Sep 29 '16 at 0:22
  • I am aware it bloats the repository. Yet I prefer portable builds with minimal dependencies, stuff I can build anywhere, anytime - without having to rely on external services which might not be available for whatever reason, or serve corrupt files. I don't want to rely on volatile, badly portable local caches (which by the way have to be populated once as well).
    – Julian
    Sep 29 '16 at 2:24
  • For the projects I set up this way, I don't expect to add too many deps, and those I do add won't be updated often, if at all. So things should stay manageable. Whoever makes a fresh working copy clone of the repo will only download all the big binary files once; and for building the project they'd need to download them either way - doesn't matter much if it's from an external maven server or directly from the source repo.
    – Julian
    Sep 29 '16 at 2:24
  • Not exactly what you ask for since the cached dependencies will be stored in default directories (probably in hidden dirs somewhere in your home directory, at least on Linux) rather than inside the project, but you can run ./gradlew --offline to make gradle only use cached dependencies and not access the network when it builds the project. Sep 29 '16 at 7:17
  • I've tried your solution, but didn't work. Can you take a look here?
    – MTZ
    Sep 10 '18 at 9:30
7

Alright, so I found a way which works for me, and it's automatable.

I found this snippet which allows you to create a Gradle task that will copy certain project dependencies to an arbitrary folder in Maven directory structure. I needed to tweak it slightly (link broken) so it doesn't just do that for one single configuration, but for any and all configurations the project defines.

My build script looks like this now:

apply plugin: 'java'

repositories {
    maven {
        url './dependencies-maven/'
    }
    mavenCentral()
    jcenter()
    // whatever other external repos your deps come from
}

dependencies {
    // ...
}

task offlineRepo (type:OfflineMavenRepository) {
    repoDir = new File(project.projectDir, 'dependencies-maven')
}

You then simply fire the offlineRepo task once: all current project dependencies will be downloaded to Gradle's local cache and then copied into a subfolder in the project directory (in Maven repo structure). The next time you run the build with the project directory in this state, the just-populated local repo will be queried first, and since it contains (should contain) all the deps, nothing should be taken from the Gradle's user folder cache nor from the web.

At this point you can take the whole project folder to any other machine with just Gradle installed and it should be able to do the build without ever going online to fetch some additional data.

Whenever you want to update some project dependency or add new ones, you'll just need to fire the offlineRepo task once again. Be aware that this won't delete the old version of the dependency you update, so if you don't want to end up with a cluttered repo, you need to do some manual cleaning here.

With this, plus putting the Gradle wrapper offline as well (described in my original question), I've now made my project buildable completely offline and with minimal prerequisites: no internet connectivity required, not even a Gradle install required. Merely the JDK.

Note: When I set this up in a completely empty dummy project, I had to create a java source directory and some sample java file, otherwise Gradle would only download/copy some POMs but not the corresponding JARs.

Addition: The tweaked snippet will only take care of project dependencies. But if you use a custom plugin, which is a buildscript dependency, the script will miss it. You'll need to add more code to the build() method:

for(Configuration configuration : project.buildscript.configurations.findAll())
{
    copyJars(configuration)
    copyPoms(configuration)
}

Edit, December 2019: Three years later after a good bunch of upvotes over the whole time and now an explicit request, it appears this topic is still important not only to me.

To avoid link breakage, I will post the current version of my project skeleton setup right here.

The following assumes you are starting with a blank slate / empty folder that is supposed to be your future project directory. All paths I specify here are relative to the project root folder.

  1. Get a gradle-wrapper.jar and place it in /gradle/wrapper. Place the accompanying gradlew and gradlew.bat run scripts in the project's root folder. Either grab those directly if you have them around somewhere, or install Gradle on your local system and run gradle wrapper in your project's root folder to generate these files. (You do not need to install Gradle locally if you can get the wrapper jar and scripts from somewhere else, and even if not, this is a one-time job. Once the wrapper is in place, there is no more need for a local Gradle installation.)
  2. Grab a Gradle binary distribution package (file name like gradle-X.X-bin.zip) from somewhere and place it in /gradle/wrapper/.
  3. Create (or modify an existing) /gradle/wrapper/gradle-wrapper.properties to contain the following:

    distributionBase=PROJECT
    distributionPath=gradle/wrapper/dists
    zipStoreBase=PROJECT
    zipStorePath=gradle/wrapper/dists
    distributionUrl=./gradle-X.X-bin.zip
    

    (The file name specified in distributionUrl obviously is just a placeholder and has to match the actual filename or the binary distribution package you used in step 2.) (You can alternatively set both distributionBase and zipStoreBase to GRADLE_USER_HOME to avoid having those temp files created in your project's folder; they will be created in your home folder instead.)

  4. Make sure you exclude certain temporary files and directories from version control. I myself am using Git, so my example ignore-file is for Git. Adjust as needed if you are using a different version control system:

    .gradle/
    gradle/wrapper/dists/
    build/
    buildSrc/.gradle
    buildSrc/build/
    
  5. Add the file /buildSrc/src/main/groovy/buildutils/OfflineMavenRepository.groovy with following contents (this is bmuschko's code linked above with all my proposed modifications applied to it):

    package buildutils
    
    import org.gradle.api.tasks.Input
    import org.gradle.api.tasks.Optional
    import org.gradle.api.tasks.OutputDirectory
    import org.gradle.api.tasks.TaskAction
    import org.gradle.api.DefaultTask
    import org.gradle.util.GFileUtils
    import org.gradle.api.artifacts.Configuration
    import org.gradle.api.artifacts.component.ModuleComponentIdentifier
    import org.gradle.maven.MavenModule
    import org.gradle.maven.MavenPomArtifact
    
    class OfflineMavenRepository extends DefaultTask {
        @OutputDirectory
        File repoDir = new File(project.projectDir, 'dependencies/maven')
    
        @TaskAction
        void build() {
            // Plugin/Buildscript dependencies
            for(Configuration configuration : project.buildscript.configurations.findAll())
            {
                copyJars(configuration)
                copyPoms(configuration)
            }
    
            // Normal dependencies
            for(Configuration configuration : project.configurations.findAll())
            {
                copyJars(configuration)
                copyPoms(configuration)
            }
        }
    
        private void copyJars(Configuration configuration) {
            configuration.resolvedConfiguration.resolvedArtifacts.each { artifact ->
                def moduleVersionId = artifact.moduleVersion.id
                File moduleDir = new File(repoDir, "${moduleVersionId.group.replace('.','/')}/${moduleVersionId.name}/${moduleVersionId.version}")
                GFileUtils.mkdirs(moduleDir)
                GFileUtils.copyFile(artifact.file, new File(moduleDir, artifact.file.name))
            }
        }
    
        private void copyPoms(Configuration configuration) {
            def componentIds = configuration.incoming.resolutionResult.allDependencies.collect { it.selected.id }
    
            def result = project.dependencies.createArtifactResolutionQuery()
                .forComponents(componentIds)
                .withArtifacts(MavenModule, MavenPomArtifact)
                .execute()
    
            for(component in result.resolvedComponents) {
                def componentId = component.id
    
                if(componentId instanceof ModuleComponentIdentifier) {
                    File moduleDir = new File(repoDir, "${componentId.group.replace('.','/')}/${componentId.module}/${componentId.version}")
                    GFileUtils.mkdirs(moduleDir)                
                    File pomFile = component.getArtifacts(MavenPomArtifact)[0].file
                    GFileUtils.copyFile(pomFile, new File(moduleDir, pomFile.name))
                }          
            }
        }
    }
    
  6. Create your /build.gradle skeleton as follows:

    repositories {
        maven {
            url './dependencies/'
        }
    
        mavenLocal()
        mavenCentral()
        jcenter()
    }
    
    buildscript {
        repositories {
            maven {
                url './dependencies/'
            }
    
            mavenLocal()
            mavenCentral()
            jcenter()
        }
    }
    
    import buildutils.OfflineMavenRepository
    task offlineDependencies (type:OfflineMavenRepository) {
        repoDir = new File(project.projectDir, 'dependencies/')
    }
    
  7. Add your additional online dependency repos, dependencies and other build logic to your build.gradle, then run ./gradlew offlineDependencies. This will create a folder structure within /dependencies/ and download and place all the dependency JARs inside. You only have to do this once to get the files there; once they are in place, you will only need to run this command again if your project's dependencies change (i.e. additional dependency or version change).

  8. Check everything in version control and you should be good. Only requirement for building the project is a working Java JDK on your system. Gradle is contained within your project's directory (usable via the wrapper), so are all your Maven dependencies.
5
  • Your small tweak pastebin is offline... would you happen to have it anywhere else? :) Dec 17 '19 at 21:15
  • 3
    I've edited my solution to be fully self-contained.
    – Julian
    Dec 18 '19 at 23:01
  • Unfortunately, you also need to download POM files that are pointing to other POM files (e.g. com.example.foo that will be resolved as com.example.bar). In current implementation only com/example/bar/*.pom will be downloaded and offline build will fail.
    – Denys
    May 12 at 14:26
  • @Denys do you have a solution/fix I can add to the script?
    – Julian
    May 18 at 11:35
  • @Julian :( Sadly, for now Gradle has only two official solutions for this problem: 1. "Warming up" your cache before transferring it to container / another machine. Gradle will create lock-files when using this cache - this might cause problems in read-only containers; 2. Incubating "shared read-only dependency cache" which is lock-free and can used for sharing cache between few containers. Details are here: docs.gradle.org/current/userguide/…
    – Denys
    May 19 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.