Clawpack turns 20

Happy Birthday!

Twenty years ago, version 1.0 of the Conservation LAWs PACKage (CLAWPACK, now Clawpack was first released by Randy LeVeque. It seems fitting to take the occasion to look back on the intervening years. What follows are my thoughts on some of the great things that have resulted.

As far as I can tell, this item in the NA-Digest is the first public announcement of its existence. It was also announced more verbosely the same year in this conference paper, from the proceedings of the 5th HYP conference. Reading that conference paper now, I am struck by how it incorporated many of the ideals of scientific software development that we now discuss as if they were new ideas. For instance,

  • Code that is easy to read and use, with plentiful documentation and examples.
  • Modular design, that allows low-level functions to be reused and disparate parts of the code to be modified independently. In the case of Clawpack, this is epitomized by the fact that it allows the solution of any system of hyperbolic PDEs by changing just a single routine (the Riemann solver).
  • An interface that allows methods and parameters to be changed easily, so that different methods can be conveniently compared.
  • Clawpack was proposed as a benchmark against which to easily test new algorithms.
  • Clawpack was released open source and for free on a public FTP server (netlib).

In this day of so much ado about credit for software, it’s also interesting to view this paper as an early example of a mathematical publication that is all about software.

Looking through the code snippets in the paper, I was astonished to recognize how much of the original Fortran 77 code remains virtually unaltered – including many variable names, function interfaces, and overall design. This is a testament to the quality of the original code design.

The central algorithms in Clawpack have also stood the test of time. The 80’s saw the heyday of research into second-order TVD methods for conservation laws, and Clawpack was released just as that era came to a close. Since then, research has gone in other directions – high-order methods, well-balancing, and positivity preservation, to name a few. While these new directions have provided additions to Clawpack, the “classic” algorithms have not changed and are still hard to beat as a robust general-purpose tool.

Of course, much has happened in the intervening twenty years. The original library handled 1- and 2-dimensional problems on regular cartesian grids. In the next few years, subsequent versions added algorithms for 3D, mapped grids, and adaptively refined meshes.

Additional algorithmic innovations are too numerous to try to list, but one that has had a lot of impact is the f-wave technique.

The problems to which Clawpack has been applied are certainly much too numerous to list. But you can start to get an idea by looking at citations of major Clawpack papers like this, this, this, this, and this. Perhaps the heaviest use in recent years has involved geophysical flows such as tsunamis and storm surges, in GeoClaw.

The Clawpack family of codes

Clawpack has spawned numerous offshoots and extensions, including (but not limited to) AMRClaw, BearClaw, ZPLClaw, SharpClaw, CUDAClaw, ForestClaw, ManyClaw, PyClaw, and GeoClaw. Some of these have become part of the Clawpack-5 suite while others have forked and gone in other directions.

Nowadays, the term Clawpack refers to a collection of interrelated packages that are maintained and developed at github.com/clawpack. They include:

  • The original (“Classic”) Clawpack;
  • AMRClaw (with adaptive mesh refinement)
  • GeoClaw (with special tools for geophysical flows)
  • PyClaw (a Python interface to both the classic code and the high-order “SharpClaw” algorithms)
  • Riemann (a library of approximate Riemann solvers, which can be used with all of the above codes)
  • VisClaw (an in-house visualization tool)

The Github organization also includes repositories for the docs and for contributed applications.

The Clawpack community

As far as I know, the original release was a one-person effort. But like most open-source projects, Clawpack quickly became a broader collaboration. I won’t attempt to credit everyone here; you can see some of the major contributors here, and many more by looking at the contributors pages on Github.

I was surprised to realize that I’ve now been involved with Clawpack for half of its existence – ten years! During those years I’ve gotten to work with an group of exceptional researchers who are also just outstanding people. They say that the culture of an open-source software community is shaped strongly by its founder, and I think Clawpack is no exception.
It seems to me that its creator is not only a great applied mathematician, but also somene who consistently leads the way in terms of improving the way we do science. Clawpack exemplifies his commitment to reproducibility and sustainable scientific software development, long before those words came into scientific vogue. He was an advocate for publishing in journals with low subscription prices, long before open access became a movement.
Most significantly, he has always been interested first in finding and solving interesting problems, and only secondarily in publishing papers. Both through his personal influence and as chair of the SIAM Journals Committee, he has been influential in making progress in these directions, including the establishment of a Software section in SISC, the acceptance by SIAM journals of supplementary materials (including code), and a new policy allowing authors to post published articles on their own or institutional websites.

As a result, the culture surrounding Clawpack has always encouraged openness and a willingness to accept new contributions. Furthermore, I think that the Clawpack developers have maintained a healthy skepticism toward our own algorithms and code. Although we try to make our code useful to as many people as possible, there has never been any attempt to evangelize the community in order to increase use of a particular set of algorithms or to increase metrics like citation counts. Because of this attitude, the code is continually improved through incorporation of new algorithmic innovations.

Lessons learned

Of course, it would be wrong to say that Clawpack has been a perfect model for scientific code development. There are plenty of things we’ve done wrong or could learn to improve.

The original announcement says that “contributions from other users will be gratefully accepted,” and that has always been true. Nevertheless, the widely accepted development model for a long time was that most users would take the code, fork it, and make their own enhancements that would never get back to the main codebase. While this prevented feature bloat, it also meant that a great wealth of knowledge – largely in the form of sophisticated approximate Riemann solvers – will perish on some dusty hard drive rather than benefitting the larger community. We’re trying to change that now by encouraging users to submit pull requests for Riemann solvers and for entire applications.

Another example of where I see room for improvement is in output and visualization, where we have, to some degree, reinvented the wheel. Clawpack has long used custom ASCII and binary file formats, that can only be read in by Clawpack (or by reverse-engineering code for the relatively simple formats). We are now pushing to move to a more standard default format (probably HDF5), which would allow easier integration with standard visualization and post-processing libraries.

On the visualization side, the Clawpack developers have created some extremely useful tools for plotting time-dependent data on structured grids (including block-structured AMR). These tools sit on top of MATLAB and matplotlib. A large amount of work has gone into these “in-house” tools rather than into leveraging and contributing to dedicated visualization tools. Meanwhile, individual users have occasionally connected Clawpack to powerful visualization tools, but their custom code never got back to the main codebase. The limited capabilities of matplotlib in 3D seem to finally be providing sufficient impetus to force us to integrate with a sophisticated visualization library. I have been working lately on integration with yt.

The next 20 years?

It may come as a surprise for a code that’s so long in the tooth, but I think Clawpack development at present is more vibrant than ever. Since 2011, we’ve held annual developer workshops, the latest of which took place last week here at KAUST. The pictures on this page are from those workshops (the cake in the first photo, which shows the Clawpack logo, was made by my wife, and is a fondant version of a fluid-dynamical shockwave hitting a low-density bubble).

As for the future, I won’t claim enough clairvoyance to see 20 years ahead. But here are some things I hope we can accomplish in the next few years:

  • Massively parallel adaptive mesh refinement (at present, it exists only in the unreleased ForestClaw code; a concurrent effort is bring this to PyClaw through BoxLib);
  • An ever-growing library of Riemann solvers for increasingly complex systems;
  • Code-generation for solving problems where a custom Riemann solver is not yet available;
  • Incorporation of code that runs on accelerators (like CUDAClaw and ManyClaw) into the main library in a way that allows users to change hardware seamlessly;
  • More teaching tools based on Clawpack and IPython notebooks, including a book showcasing Riemann solutions of important physical systems;
  • Additional algorithms (such as Discontinuous Galerkin methods and new time stepping techniques) that can be accessed through the same problem setup and use the same Riemann solvers;
  • Better interoperability between Clawpack and other codes (such as Proteus), by making Clawpack more of a true library.

Are you excited yet? I certainly am. Come join the fun!