I started a new topic to make it more organized.
The way the code is now (V8.3.9-18-geeae287), the streams used in the consumer groups code ( library(redis_stream) ) will grow unbounded, and all the data will remain even when the processes are terminated. This is the default redis way of operating also.
However, I think this is a little dangerous for the regular user. We have some options that I can think of:
Make the MAXLEN ~ option for the XADD command have a large default value, to prevent a nightmare with users asking why the machine is running out of memory.
Maybe provide a call-back option when the stream grows close to a certain size, but this one seems less desirable, since we would need to check the size of the stream regularly.
Provide an option to call XTRIM regularly at user specified time intervals, with a user-specified limit for the stream.
Probably we should also add a fairly noticeable paragraph in the documentation explaining that the stream will grow unbound unless MAXLEN has the proper value or XTRIM is called. The user can also call
DEL <streamname> to delete the entire stream when the process is finishing.
In any case, I think leaving the stream with unbound growth by default is rather dangerous.
EDIT: to see the danger of this, run the demo and then terminate all processes. then in redis-cli run
XRANGE candidates - + and you will see the messages still there.