For both reproducibility and interoperability containerization of models can be very useful. All dependencies are packaged alongside the model, and the containers are easy to share and version. After creating a container image for your model, running it anywhere is as easy as doing docker run mymodel:1.0.0.
However, when a model is in a container you cannot interface with it directly anymore. To be able to communicate with containerized models, we developed grpc4bmi in the eWaterCycle project. This enables you to start a gRPC server inside the container, which other processes on your system (or anywhere!) can communicate to.
grpc4bmi is available in Python and C++, so any language that can directly interface with these is supported.
Not all languages have a gRPC server implemented though, so for this reason we recently created Remote BMI, an OpenAPI based RESTful API for BMI. We currently have implemented servers/clients in Python, Julia, and R.
These two tools allow for coupling models across many languages (Python, C, C++, Fortran, Julia, R).
To summarize some advantages:
Communicate with containerized models
No need for matching dependencies/interpreter versions
Couple across languages
Some disadvantages:
You cannot pass pointers: very tight coupling is not possible
There is a communication overhead, both in grpc4bmi and Remote BMI
If anyone would be interested, I could give a short demo presentation showcasing what is possible at some point.
I would be interested in a demo at some point! May it could be useful as a CSDMS webinar?
Can you give a rough idea of what the communication overhead is, e.g. per average call on an average machine? I appreciate that might be difficult to answer but I’m mostly interested in order of magnitude!
It is indeed very difficult to answer! It depends on the protocol used, the server language, and the client language.
I have not done a lot of benchmarking, but gRPC should be much more efficient than the RESTful API of RemoteBMI (7-10x apparently).
A gRPC benchmark is available, where the Rust and Java implementations have a 3 - 5 ms call time. Python 50 ms.
For some quick tests on my laptop, a Python client & Julia Remote BMI server, the return time is 0.5 - 1 s. Note sure why it’s quite slow in this case. In any case, if your .update() steps are very fast, this will be too much overhead. Gridded hydrological models can take a few seconds per time step, so then this is acceptable.
It would probably be possible to use a Rust gRPC server and client, which you interface with the other languages to get this to be more performant. However, we have for now had to focus on getting things to be simple and to work
