Redis and binary blobs

Help! Data Structure
1

Hi,

I’m working with vector searches in Redis. I’ve been able to use the Redis library and Janus to create document vectors (using sentence_transformers) in Redis as JSON objects. And I’m able to create a Redis index.

When it comes to searching, Redis requires that the query vector be provided as a binary blob. In Python it looks like this:
res = r.ft("doc_idx").search(q, query_params={"vec": model.encode(sentence).astype(np.float32).tobytes()})

Note tobytes() which returns a Python bytes object.

Is there an equivalent in SWI Prolog? Alternatively, if I have a ref to a Python blob (via Janus, obtained by py_call with py_object(true) option), I’m not clear on how I can serialize that out to Redis, in Prolog. If I don’t use py_object(true), then I get a list of integers (the bytes) back from Janus. Can that be blobbified somehow?

Thanks.

2

As is, the SWI-Prolog Redis interface does not support Redis binary blobs. I guess we could add that by supporting e.g., Data as blob in the same context as where we now support Term as prolog. Data would then be bytes, encoded as list of integers or as an 8-bit string. That should be fairly trivial to add. If you provide a PR, I’m happy to accept that.

Second problem is how to do binary encoding. There are many options for binary encoding data. Some of them are implemented by the protobufs package. If you can get a list of bytes from Python, the above should be enough. If this is some specific encoding for Redis, we could consider adding a support predicate to the redis package.