Maven: One Phase per CI Stage

Have you ever tried running Maven phases in different stages/jobs of a CI pipeline? If so, you have probably run into an annoying issue where Maven re-runs the phases already executed in previous jobs, even though the files produced by the previous phase are present. Let’s see how to fix that!

Context

Let’s say you want to run Maven’s compile, test and package phases in three separate jobs. This is a legitimate use case, for example in a monorepo with several modules written in different languages. In this kind of situation, and for better readability, I like to have a fine-grained split of my pipeline. A Build stage runs all the build jobs, a test stage runs all the test jobs, and so on. Optimizing this kind of pipeline is a different topic that we will cover in another article. With Maven, during the test stage, it will re-run the whole compile phase. Even if you restore the entire target folder containing the output of the compile phase in the test job before running it, Maven won’t care. This completely defeats the purpose of splitting phases into separate jobs.

Why does Maven behave this way?

The maven-compiler-plugin uses two criteria to decide whether it needs to recompile:

• The list of files already processed in a previous phase (sources compiled during the test phase, tests compiled during the package phase)
• The timestamp of the sources compared to the produced .class files

Both criteria can cause problems when using Maven in a CI with Docker agents.

Changed paths of already-processed files

If your CI mounts your project sources into a Docker container before running the job, the path inside the container may not be the same as in the container of the previous phase. I recently ran into this with TFS. The Build container running mvn compile had the project sources mounted under /__w/<n>/s, n being a number assigned by each CI agent. The Test container therefore had a /__w/<n>/s path with a different n.

Maven compares the absolute paths recorded in target/maven-status/maven-compiler-plugin/**/inputFiles.lst with the sources actually present in the workspace, won’t find them, and will therefore trigger a new compilation.

To solve this problem, you need to rewrite the source paths in the inputFiles.lst files:

1
2
3
4
5
6
7
8

# Example for TFS. Adapt it if your CI is affected by this issue
COMPILE_STATUS_DIR="$(Build.SourceDirectory)/<modulename>/target/maven-status/maven-compiler-plugin/compile/default-compile"
sed -i "s|/__w/[0-9]*/s/|$(Build.SourceDirectory)/|g" "$COMPILE_STATUS_DIR/inputFiles.lst"

TEST_STATUS_DIR="$(Build.SourceDirectory)/<modulename>/target/maven-status/maven-compiler-plugin/compile/default-testCompile"
if [ -d "$TEST_STATUS_DIR" ]; then
  sed -i "s|/__w/[0-9]*/s/|$(Build.SourceDirectory)/|g" "$TEST_STATUS_DIR/inputFiles.lst"
fi

Changed timestamps

The second mechanism makes Maven recompile all .class files if their timestamp is older than the sources. This happens when you restore previously compiled .class files directly into the project folder mounted in your container. To solve this problem, simply touch the files to reset their timestamp.

1
2

# Example for TFS. Adapt it with your CI's variables.
find $(Build.SourceDirectory)/<modulename>/target -type f -exec touch {} +

Conclusion

This problem gave me quite a headache, and I’m glad to have found a solution. It’s probably not the cleanest one, but it has the merit of working. I can now have a CI with clearly separated jobs.