How to share bamboo Java specs between several projects

Thu 21 May 2020

Forewords
Reusable project organization
- Folder structure
- Maven structure
Running the commands
Conclusion
Addendum

Forewords

Atlassian Bamboo CI/CD is a wonderful tool both for building your projects and deploying them. To help the development and the maintenance of the various plans and jobs and pipelines, Bamboo features the "CI/CD-as-code" approach through the so-called Bamboo-Specs, and with two possibilities: YAML and Java.

YAML is quite common for CI/CD in cloud services such as Travis or Appveyor. It is a nice descriptive language that outstands by its simplicity, but can quickly become hard to get done right as your projects scale up. On the other hand, Java, much less common in the CI/CD world, is a powerful programming language that comes with a tremendous amount of support for programming:

Full fledged object oriented programming language with conditions, inheritance, loops and so on
Reflexivity which allows excellent tooling such as refactoring
IDE great support (on-the-fly compilation and error checking, type inference, documentation, highlighting etc)
unit testing
etc

All in all, Java turns out to be much more powerful as a language than YAML. In this short article, the focus will be on showing how to structure your project such that reusable components on one side, and plan specific businesses on the other, are separated. This will let you reuse this library in other projects, compose and maintain a lot of things at once, and amortize the developments efforts across several projects.

Reusable project organization

Let's dig now into an possible organization for reusable CI/CD code. But first ...

Disclaimer I am not a Java nor a Maven expert: there are certainly better ways of doing what we need to achieve. Any hints that I can understand are most welcome!

The main steps are as follow:

first you need a Bamboo Java-spec Maven project. Atlassian's documentation gives you a way to get started and this part will be skipped.
we then need to change a bit the folder structure
finally we embed the initial Maven project in a bigger/umbrella Maven project that has connection with other existing projects. Those other projects are obviously the common libraries that are bringing us relevant and reusable services for developing further our current CI/CD project.

To make things clearer, let's call the main CI/CD project yayi (that contains the build definition for my project yayi) while the reusable CI/CD library will be called simply common.

Folder structure

The folder structure is as follow. I deliberately left another project, code_doc, to highlight the organization together with the same common project.

├── common
│   ├── pom.xml
│   └── src
├── code_doc
│   ├── code_doc-specs
│   │   ├── pom.xml
│   │   └── src
│   └── pom.xml
└── yayi
    ├── pom.xml
    └── yayi-specs
        ├── pom.xml
        └── src

The explanations are:

the yayi-specs (or code_doc-specs and not to be confused with the yayi or code_doc folders) contain the project as created by running the skeleton (see Atlassian's documentation).
the folder yayi contains the parent/umbrella project that we will discuss in the next section, and the yayi-spec maven project is a subfolder of it,
the folder common will contain our CI/CD toolbox code that we will share among several of our projects.

Maven structure

The umbrella project

Maven works with pom.xml files that describes the project structure in an XML form. Among the various sections this XML can contain, there is a section called modules: each module in there can refer to another Maven folders, which means folders that contain another pom.xml. This Maven feature seems to be a good start for what we need.

Just as a reminder, this is where this file is:

├── common
│   ├── pom.xml
│   └── src
├── code_doc
│   ├── code_doc-specs
│   │   ├── pom.xml
│   │   └── src
│   └── pom.xml
└── yayi
    ├── pom.xml         <-- here
    └── yayi-specs
        ├── pom.xml
        └── src

and this is the content of the parent Maven project:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>com.atlassian.bamboo</groupId>
    <artifactId>bamboo-specs-parent</artifactId>
    <version>6.10.3</version> <!-- this is the version of your Bamboo instance,
                                   please change accordingly -->
    <relativePath/>
  </parent>

  <groupId>org.yayimorphology.nan</groupId>
  <artifactId>yayi-specs-parent</artifactId> <!-- name of the maven project as it appears in my IDE -->
  <version>1.0.0</version>
  <packaging>pom</packaging>

  <modules>
      <module>../common</module>
      <module>yayi-specs</module>
  </modules>
</project>

The important parts are:

The type of the packaging that should be pom as it links to other pom.xml files:
```
<packaging>pom</packaging>
```
Indeed, our umbrella/parent project does not produce any artifact by itself, it is there to make a single Maven project out of several ones: this is what this pom packaging is about.
The links to the other projects that are relative to the current (parent) pom.xml file, and within a <modules> section:
```
<modules>
    <module>../common</module>
    <module>yayi-specs</module>
</modules>
```
Of course this layout can be adapted to your own.
The parent section as a copy-paste of the skeleton project. Omitting this makes the execution of the deployment difficult.

A specific groupId, artifactId and version are also needed. I use a dummy version 1.0.0 everywhere as I am not distributing compiled artifacts, a groupId that is similar to the website containing my Bamboo instance and an artifactId that is derived from the name of the project. There is no particular constraint I know of for those values as no distribution is involved.

Adapting the project skeleton

We now need to advertise a direct dependency of our yayi project to the common one. This is done by changing the yayi-specs project's pom.xml. As a friendly reminder, we are now here:

├── common
│   ├── pom.xml
│   └── src
├── code_doc
│   ├── code_doc-specs
│   │   ├── pom.xml
│   │   └── src
│   └── pom.xml
└── yayi
    ├── pom.xml         <- we were here
    └── yayi-specs
        ├── pom.xml     <-- now we are here
        └── src

The dependency injection to the common project is simply done by adding a dependency section, as follow:

<dependencies>
  <!-- those dependencies come from the skeleton -->
  <dependency>
    <groupId>com.atlassian.bamboo</groupId>
    <artifactId>bamboo-specs-api</artifactId>
  </dependency>

  <dependency>
    <groupId>com.atlassian.bamboo</groupId>
    <artifactId>bamboo-specs</artifactId>
  </dependency>

  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <scope>test</scope>
  </dependency>

  <!-- the new section -->
  <dependency>
    <groupId>org.yayimorphology.nan</groupId>
    <artifactId>common-specs</artifactId>
    <version>1.0.0</version>
  </dependency>

</dependencies>

Again there is no particular constraints, but the artifactId and version should match the one provided by the common project. We took also the same groupId as for the umbrella project or any of our projects.

Note that without this explicit dependency, the functions that are needed from yayi-specs to common might not be found. This omission may appear clearly as a compilation failure only when you actually have code in yayi-specs referencing code in common and you may miss this configuration at the beginning of your developments.

The common project

The common project is also a Maven project and contains a pom.xml. In fact, you can virtually make it a copy paste of the yayi-specs Maven project:

it generates a jar,
it inherits from the same bamboo-specs-parent such that we have nice automated commands,
only the groupId, artifactId and (maybe) the version should be specific, and match the ones of the dependency we injected into yayi-specs's pom.xml file.

A relatively good start would be to just copy/paste (yeah, you heard me) the pom.xml of the yayi-specs without the (self) dependency to common:

├── common
│   ├── pom.xml         <-- now we are here
│   └── src
├── code_doc
│   ├── code_doc-specs
│   │   ├── pom.xml
│   │   └── src
│   └── pom.xml
└── yayi
    ├── pom.xml
    └── yayi-specs
        ├── pom.xml     <- copy pasted from here
        └── src

The source code cannot be exactly copy/pasted from the yayi-specs though: common should not contain any class annotated with @BambooSpec, as this is a specific instruction for the Bamboo commands.

How it looks like

We like pictures, even when it is to show Eclipse. This is now how it looks like when importing the main yayi project into this IDE:

Eclipse maven java spec

Here we see clearly the 3 projects:

common
yayi-specs
yayi-specs-parent, which is at the end just a kind of "super project"

Running the commands

Since you know everything about OOP, I leave your first hello world as an exercise. However, the structure above has one drawback: we need to run a specific install command every time we make a change in the common project.

So before:

cd yayi
mvn test
mvn -Ppublish-specs

Now becomes

cd yayi
mvn test
mvn install          # this is the new command
mvn -Ppublish-specs

I spent quite some time to avoid that, but I gave up :) . The install command places all the artifacts of the projects into the MAven local cache (usually in ~/.m2), and the -Ppublish-specs profile takes common's jar file from there.

There is one gotcha and I was caught by it several times already, so be warned: you may publish your projects to Bamboo with an outdated version of common. Maybe it would be a good thing to create a script that does all this mechanically.

Conclusion

There you are with your first, reusable, Bamboo Java spec project. As you will develop further in this path and see the number of projects in Bamboo grow, you will quickly see the benefits of creating shareable components. As examples:

AWS setup for deploying "things" (docker images, CloudFront, etc)
composition of short pipelines into longer ones, such as python virtual environment (creation and sourcing for each task), yarn installation and configuration , etc
dynamic docker image building and direct injection in the subsequent build tasks
local cache management
conan
and many more!

In addition to all this, you know already a lot about library life cycles and management from your OOP experience, unit testing and so on, and this library creation will not be too much effort.

Cherry on top: Bamboo is free for open source projects and can run on a Raspberry Pi 4 with low effort!

Addendum

This project example can be found on Bitbucket (it would have been odd to put a link about Atlassian on a GitHub repository).

As you see, I did not invest too much time in having the perfect integration, this is absolutely not my focus: I want my project to build on Bamboo and to be able to make evolution to those project and my builds easily. That comes at the cost of running sometimes extra commands, so be it.

yayi C++, python, image processing, hacking, etc