Dogen: The Package Management Saga

Over the last few years I’ve had a little project on the side called Dogen. Dogen is a code generator designed to target domain models, with the lofty ambition of automating the modelling process as much as possible: users create domain models using a supported UML tool and respecting a set of predefined restrictions; Dogen uses the tool’s diagram files to generate the source code representation. The generated code contains most of the services required from a typical domain object such as serialisation, hashing, streaming and so on. Dogen is written in C++ 14 and generates C++ 14 too, but other languages will eventually be supported as well. You can find our repo at https://github.com/DomainDrivenConsulting/dogen .

Now, for a four-year-old project, I guess it’s fair to say that Dogen hasn’t exactly set the Open Source world on fire. Nevertheless, it has proven to be a personal fountain of lessons and experiences on software development; one such lesson was package management and that’s what I shall reminisce about in this article.

The conundrum

Like any other part-time C++ developer whose professional mainstay is C# and Java, I have keenly felt the need for a package manager when in C++-land. The problem is less visible when you are working with mature libraries and dealing with just Linux, due to the huge size of the package repositories and the great tooling built around them. However, things get messier when you start to go cross-platform, and messier still when you are coding on the bleeding edge of C++: either the package you need is not available in the distro’s repos or even PPA’s; or, when it is, its rarely at the version you require. Alas, for all our sins, that’s exactly where we were when Dogen got started.

A spoonful of Dogen history

Dogen sprung to life just a tad after C++-0x became C++-11, so we experienced first hand the highs of a quasi-new-language followed by the lows of feeling the brunt of the bleeding edge pain. For starters, nothing we ever wanted was available out of the box, on any of the platforms we were interested in. Even Debian Testing was a fair bit behind the curve –probably stalled due to a compiler transition or other. In those days, Real Programmers were Real Programmers and mice were mice: we had to build and install the C++ compilers ourselves and, even then, C++-11 support was new, a bit flaky and limited. We then had to use those compilers to compile all of the dependencies in C++-11 mode.

The PFH days

After doing this manually once or twice, it soon stopped being fun. And so we solved this problem by creating the PFH – the Private Filesystem Hierarchy – a gloriously over-ambitious name to describe a set of wrapper scripts that helped with the process of downloading tarballs, unpacking, building and finally installing them into well-defined locations. It worked well enough in the confines of its remit, but we were often outside those, having to apply out-of-tree patches, adding new dependencies and so on. We also didn’t use Travis then; not even sure it existed, but if it did, the rigmarole of the bleeding edge experience would certainly put a stop to any ideas of using it. So we used a local install of CDash with a number of build agents on OSX, Windows (MinGW) and Linux (32-bit and 64-bit). Things worked beautifully when nothing changed and the setup was stable; but every time a new version of a library – or God forbid, of a compiler – was released, one had that sense of dread: do I really need to upgrade? And yet we often did, because we needed the features.

Since one of the main objectives of Dogen was to learn about C++-11, one has to say that the pain was worth it. But all of the moving parts described above were not ideal and they were not the thing you want to be wasting your precious time on when it is very scarce. They were certainly not scalable.

The good days and the bad days

Things improved slightly for a year or two when distros started shipping C++-11 compliant compilers and recent boost versions. This led to an attack of pragmatism, during which we ditched all platforms except for Linux, got rid of almost all our private infrastructure and moved over to Travis. For a while things looked really good. However, due to Travis’ Ubuntu LTS policy, we were stuck with a rapidly ageing Boost version. At first PPAs were a good solution, but over time these became stale too.

Soon we were stuck, unable to afford to revert back to the bad old days of the PFH but also unable to freeze all dependencies in time, as it would provide a worse development experience. So it was that the only route left was to break the build on Travis and hope that a solution would manifest itself. The red build painfully lingered on, commit after commit, whilst alternatives such as Drone.io and GitLab were unsuccessfully tried.

Finally, there was nothing else for it. We simply needed a package manager to manage the development dependencies.

Nuget hopes dashed

Having used Nuget in anger for both C# and C++ projects – and given Microsoft’s recent change of heart with regards to open source – I was secretly hoping that Nuget would get some traction in the wider C++ world. Nuget works well enough in Mono, and C++ support for Windows was added fairly early on. It was somewhat limited and a bit quirky at the start but it kept on getting better, to the point of being actually usable; we now use Nuget to manage our C++ dependencies at work – a Windows shop in the main – and it has improved our quality of life dramatically.

Unfortunately, the troubles begun on closer inspection. The truth is that Microsoft’s current Nuget focus is C# and Visual Studio, not Linux and C++. Also, it seems that outside Microsoft and Xamarin, there just isn’t enough traction for this tool at present.

However, there have been a couple of recent announcements from Microsoft that give me hope things may change in the future:

• Clang with Microsoft CodeGen in VS 2015 Update 1
• Support for Android CMake projects in Visual Studio

Surely the logical consequence is to be able to manage packages in a consistent way across platforms? We can but hope.

Biicode comes to the rescue?

Nuget did not pan out but what did happen was even more unlikely: some crazy-cool Spaniards had decided to create a stand alone package manager. Being from the same peninsula, I felt compelled to use their wares, and was joyful as they went from strength to strength – including the success of their open source campaign. And I loved the fact that it integrated really well with CMake, and that CLion provided Biicode integration very early on.

However, my biggest problem with Biicode was that it was just too complicated. I don’t mean to say the creators of the product didn’t have very good reasons for their technical choices – Lord knows creating a product is hard enough, so I have nothing but praise to anyone who tries. However, for me personally, I never had the time to understand why Biicode needed its own version of CMake, nor did I want to modify my CMake files too much in order to fit properly with Biicode and so on. Basically, I needed a solution that worked well and required minimal changes at my end. Having been brought up with Maven and Nuget, I just could not understand why there wasn’t a simple packages.xml file that specified the dependencies and then some non-intrusive CMake support to expose those into the CMake files. As you can see from some of my posts, it just seemed it required ‘getting’ Biicode in order to make use of it, which for me was not an option.

Another thing that annoyed me was the difficulty on knowing what the ‘real’ version of a library was. I wrote, at the time:

One slightly confusing thing about the process of adding dependencies is that there may be more than one page for a given dependency and it is not clear which one is the ‘best’ one. For RapidJson there are three options, presumably from three different Biicode users:

• fenix: authored on 2015-Apr-28, v1.0.1.
• hithwen: authored 2014-Jul-30
• denis: authored 2014-Oct-09

The ‘fenix’ option appeared to be the most up-to-date so I went with that one. However, this illustrates a deeper issue: how do you know you can trust a package? In the ideal setup, the project owners would add Biicode support and that would then be the one true version. However, like any other project, Biicode faces the initial adoption conundrum: people are not going to be willing to spend time adding support for Biicode if there aren’t a lot of users of Biicode out there already, but without a large library of dependencies there is nothing to draw users in. In this light, one can understand that it makes sense for Biicode to allow anyone to add new packages as a way to bootstrap their user base; but sooner or later they will face the same issues as all distributions face.

A few features would be helpful in the mean time:

• popularity/number of downloads
• user ratings

These metrics would help in deciding which package to depend on.

For all these reasons, I never found the time to get Biicode setup and these stories lingered in Dogen’s backlog. And the build continued to be red.

Sadly Biicode the company didn’t make it either. I feel very sad for the guys behind it, because their heart was on the right place.

Which brings us right up to date.

Enter Conan

When I was a kid, we were all big fans of Conan. No, not the barbarian, the Japanese Manga Future Boy Conan. For me the name Conan will always bring back great memories of this show, which we watched in the original Japanese with Portuguese subtitles. So I was secretly pleased when I found conan.io, a new package management system for C++. The guy behind it seems to be one of the original Biicode developers, so a lot of lessons from Biicode were learned.

To cut a short story short, the great news is I managed to add Conan support to Dogen in roughly 3 hours and with very minimal knowledge about Conan. This to me was a litmus test of sorts, because I have very little interest in package management – creating my own product has proven to be challenging enough, so the last thing I need is to divert my energy further. The other interesting thing is that roughly half of that time was taken by trying to get Travis to behave, so its not quite fair to impute it to Conan.

Setting up Dogen for Conan

So, what changes did I do to get it all working? It was a very simple 3-step process. First I installed Conan using a Debian package from their site.

I then created a conanfile.txt on my top-level directory:

  [requires]
  Boost/1.60.0@lasote/stable
  [generators]
  cmake

Finally I modified my top-level CMakeLists.txt :

  # conan support
  if(EXISTS "${CMAKE_BINARY_DIR}/conanbuildinfo.cmake")
    message(STATUS "Setting up Conan support.")
    include("${CMAKE_BINARY_DIR}/conanbuildinfo.cmake")
    CONAN_BASIC_SETUP()
  else()
    message(STATUS "Conan build file not found, skipping include")
  endif()

This means that it is entirely possible to build Dogen without Conan, but if it is present, it will be used. With these two changes, all that was left to do was to build:

  $ cd dogen/build/output
  $ mkdir gcc-5-conan
  $ cd gcc-5-conan
  $ conan install ../../..
  $ make -j5 run_all_specs

Et voilà, I had a brand spanking new build of Dogen using Conan. Well, actually, not quite. I’ve omitted a couple of problems that are a bit of a distraction on the Conan success story. Let’s look at them now.

Problems and their solutions

The first problem was that Boost 1.59 does not appear to have an overridden FindBoost, which means that I was not able to link. I moved to Boost 1.60 – which I wanted to do anyway – and it worked out of the box.

The second problem was that Conan seems to get confused with Ninja, my build system of choice. For whatever reason, when I use the Ninja generator, it fails like so:

  $ cmake ../../../ -G Ninja
  $ ninja -j5
  $ ninja: error: '~/.conan/data/Boost/1.60.0/lasote/stable/package/\
  ebdc9c0c0164b54c29125127c75297f6607946c5/lib/libboost_system.so', needed by 'stage/bin/dogen_utility_spec', \
  missing and no known rule to make it

This is very strange because Boost System is clearly available in the Conan download folder. Going back to make solved this problem. I’ve opened an issue in Conan (#56) and its currently under investigation.

The third problem is more Boost related than anything else. Boost Graph has not been as well maintained as it should, really. Thus users now find themselves carrying patches, and all because no one seems to be able to apply them upstream. Dogen is in this situation as we’ve hit the issue described at Stack Overflow: ‘Compile error with boost.graph 1.56.0 and g++ 4.6.4’. Sadly this is still present on Boost 1.60; the patch exists in Trac but remains unapplied (#10382). This is a tad worrying as we make a lot of use of Boost Graph and intend to increase the usage in the future.

At any rate, as you can see, none of the problems were showstoppers, nor can they all be attributed to Conan.

Getting Travis to behave

Once I got Dogen building locally, I then went on a mission to convince Travis to use it. It was painful, but mainly because of the lag between commits and hitting an error. The core of the changes to my YML file were as in Listing 1.

install:
<snip>
  # conan
  - wget https://s3-eu-west-1.amazonaws.com/conanio-production/downloads/conan-ubuntu-64_0_5_0.deb -O conan.deb
  - sudo dpkg -i conan.deb
  - rm conan.deb
<snip>
script:
  - export GIT_REPO="`pwd`"
  - cd ${GIT_REPO}/build
  - mkdir output
  - cd output
  - conan install ${GIT_REPO}
  - hash=`ls ~/.conan/data/Boost/1.60.0/lasote/stable/package/`
  - cd ~/.conan/data/Boost/1.60.0/lasote/stable/package/${hash}/include/
  - sudo patch -p0 < ${GIT_REPO}/patches/boost_1_59_graph.patch
  - cmake ${GIT_REPO} -DWITH_MINIMAL_PACKAGING=on
  - make -j2 run_all_specs
<snip>
			

I probably should have a bash script by now, given the size of the YML, but hey – if it works. The changes above deal with installation of the package, applying the boost patch and using Make instead of Ninja. Quite trivial in the end, even though it required a lot of iterations to get there.

Conclusions

Having a red build is a very distressful event for a developer, so you can imagine how painful it has been to have red builds for several months . So it is with unmitigated pleasure that I got to see build #628 in a shiny emerald green. As far as that goes, it has been an unmitigated success.

In a broader sense though, what can we say about Conan? There are many positives to take home, even at this early stage of Dogen usage:

• it is a lot less intrusive than Biicode and easier to setup. Biicode was very well documented, but it was easy to stray from the beaten track and that then required reading a lot of different wiki pages. It seems easier to stay on the beaten track with Conan.
• as with Biicode, it seems to provide solutions to Debug/Release, multi-platforms and multiple compilers. We shall be testing it on Windows soon and reporting back.
• hopefully, since it started Open Source from the beginning, it will form a community of developers around the source with the know-how required to maintain it. It would also be great to see if a business forms around it, since someone will have to pay the cloud bill. It certainly is gaining popularity, as the recent CppCast attests.

In terms of negatives:

• I still believe the most scalable approach would have been to extend Nuget for the C++ Linux use case, since Microsoft is willing to take patches and since they foot the bill for the public repo. However, I can understand why one would prefer to have total control over the solution rather than depend on the whims of some middle-manager in order to commit.
• it seems publishing packages requires getting down into Python. Haven’t tried it yet, but I’m hoping it will be made as easy as importing packages with a simple text file. The more complexity around these flows the tool adds, the less likely they are to be used.
• there still are no ‘official builds’ from projects. As explained above, this is a chicken and egg problem, because people are only willing to dedicate time to it once there are enough users complaining. Having said that, since Conan is easy to setup, one hopes to see some adoption in the near future.
• even when using a GitHub profile, one still has to define a Conan specific password. This was not required with Biicode. Minor pain, but still, if they want to increase traction, this is probably an unnecessary stumbling block. It was sufficient to make me think twice about setting up a login, for one.

In truth, these are all very minor negative points, but still worth making them. All and all, I am quite pleased with Conan thus far.