Library(redis) pipelining?

[swi]

@jan

I’m happy that X as Type applies to arrays, but we need something else for maps/hashes, etc. Notably the value translation should typically depend on the key, e.g, receiving

[ "label", "Hello world", "image", <JPEG bytes> ]

Could not this be something like MyDictImg as dict(_{label: atom, image: codes})?

@jan

I think the idea of a user type is asking too much. It complicates stuff a lot and as is you can get a good type to do the final step. So, unless convincing use cases show up I think the set of types will be fixed.

A typical use case, since this will be used heavily for web apps, is JSON data. JSON data will often be converted into some user defined prolog type, so a hook will be very useful. Another typical case is RPC between distributed processes when redis is used like a message broker between distributed apps: formats like JSONRPC, MSGPACK, PROTOBUFS are heavily used here and would need to be converted to the user prolog type.

An example for simple JSON would be something like redis(get(login:person), MyPerson as person).

Where RedisData is a JSON object like “{ “name”: Name, “age”: Integer, “face”: Base64Data }” (for redis it is just a string). The hook will convert it to the prolog data type.

The hook could be something like:
redis_convert_type(Command, Type, RedisData, UserData) (where RedisData is trivial for bulk data/arrays, but for maps it can simply be a dict of the form: _{ AtomKey1: String, AtomKey2: String} } where String is binary safe. Then the user can convert it to whatever prolog data structure they have for the incoming JSON.

EDIT: Although the hook will improve readability and the use cases are common, I agree that it can be done without the hook also.

[jan]

Good points. Need to think about this. There are also some disadvantages I’ll talk about next time.

[jan]

Making slow progress on the “as Type” handling. I think this looks pretty neat. Main progress:

• a new type auto(AsText,AsNumber), i.e., auto(atom, integer) which will return the bulk data as an integer if it conplies to the integer syntax and an atom otherwise.
• a type auto that is a shorthand for auto(atom,number)
• a new type pairs(AsKey, AsValue), which is used on maps and arrays, demanding the latter to have even length. This returns the map or array as a list of Key-Value pairs, respecting the two types.
• new types dict(AsKey, AsValue) or simply dict(AsValue) to return the map/array into a dict.

We probably still want for dict(AsKey,AsValue) the AsValue to be a dict mapping keys to types as @swi proposed.

I’m still a bit unsure about user defined conversion hooks. I’m a little worried about the complexity as well as disliking hooks that change global behavior. The latter might not be that much of a problem as I’d assume most programs will talk to one redis server that satisfies some conventions.

I’m wondering whether the default translation for bulk should be changed to auto. This is most likely a lot friendlier to 99% of the users and applications.

Opinions?

[swi]

@jan

I think this looks pretty neat. Main progress:

• a new type auto(AsText,AsNumber), i.e., auto(atom, integer) which will return the bulk data as an integer if it conplies to the integer syntax and an atom otherwise.
• a type auto that is a shorthand for auto(atom,number)
• a new type pairs(AsKey, AsValue), which is used on maps and arrays, demanding the latter to have even length. This returns the map or array as a list of Key-Value pairs, respecting the two types.
• new types dict(AsKey, AsValue) or simply dict(AsValue) to return the map/array into a dict.

This is great!

@jan

I’m still a bit unsure about user defined conversion hooks

Thinking more about it, maybe it does complicate things more than necessary, and we can still easily create a predicate that produces the user type, without using a hook.

@jan

I’m wondering whether the default translation for bulk should be changed to auto .

I think this is the right way to go. The addition of the auto type really improves things.

[jan]

@swi

@jan

I’m wondering whether the default translation for bulk should be changed to auto .

I think this is the right way to go. The addition of the auto type really improves things.

Pushed changes to realize this. This also includes reporting status as e.g., status(ok). That is, as a lowercase atom rather than an uppercase string. As far as I can tell, the status indications are short identifier-like strings. I think this makes it all a lot more pleasant to use. Getting close …