I have a few million facts (mostly triples, except the node labels have 5 components) that I run some exhaustive validation queries over, using forall/2. I’ve found that the time to run these queries is sensitive to which query is first, which seems to affect the JIT indexes, and I’d like to know h…

[image] peter.ludemann: Is this working-as-intended? I guess you read the docs ? I don’t have much to add. The key thing is that it only creates a new index if it considers its existing indexes poor. That way the result can depend on the order. [image] peter.ludemann: once(node(,,,,,,…

Running the “create JITI queries” in parallel worked very nicely – thank-you. Loading 2 million facts from a .qlf file took 0.5 seconds (vs 12.6 seconds from a .pl file); adding 4 indexes in parallel took 1.4 seconds (thank-you concurrent_maplist/2). I’m seeing wide variation in the time my queries…

Sounds good. I don’t really recall 20% wide. Just slightly little different data characteristics may already cause larger differences. I guess the ultimate route to these things is to monitor bottlenecks and take measures at runtime. For example, the running thread could count calls with poor i…

I get at least 20% variation in timings with the same data. My hypothesis is that memory usage (both within swipl and with other running things such as chrome) is a factor; but statistics/2 for that doesn’t seem to work – e.g.: 1 ?- statistics(core, M). M = [0, -1]. 2 ?- statistics(memory, M). M = …

These are old compatibility keys that are hard to support in modern OSes. If you want to know overall resource usage, use the OS tools. All the other stuff is somewhat documented. I doubt anyone wants to describe exhaustively what is possible …

[image] peter.ludemann: How should I “prime” the JITI to get maximum speedup? Answering myself, this worked, to prime the indexes (except I ran them in parallel): ( node(sig,corpus,root,path,lang,_,_) -> true ; true ), ( node(_,_,_,_,_,fact,value) -> true ; true), ( edge(vname(sig,corpus,roo…

Optimizing the JIT indexes

Help!

jan Split this topic March 24, 2020, 1:22pm 7

5 posts were split to a new topic: Poor memory reuse (tcmalloc?)

Topic		Replies	Views
JIT indexing question Help!	19	320	February 14, 2023
First and second argument indexing -- which should be a first argument Help!	21	1235	July 21, 2021
How to write data for fast indexing? Help!	2	716	August 12, 2019
A little benchmark shows a slow first access to tabled predicates Help!	0	62	January 23, 2025
How to build a table in parallel General tabling , threads	18	499	January 29, 2024

Optimizing the JIT indexes

Related topics