# Can you please provide advice on big data handling strategy?

**URL:** <https://swi-prolog.discourse.group/t/can-you-please-provide-advice-on-big-data-handling-strategy/2997>\
**Category:** Algorithm\
**Created:** [October 2, 2020, 6:00pm UTC](https://swi-prolog.discourse.group/t/can-you-please-provide-advice-on-big-data-handling-strategy/2997 "2020-10-02T18:00:48Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://swi-prolog.discourse.group/u/EricGT)\
**Post date:** [October 2, 2020, 6:52pm UTC](https://swi-prolog.discourse.group/t/can-you-please-provide-advice-on-big-data-handling-strategy/2997/2 "2020-10-02T18:52:22Z")

</div>

> [@Rscho314](#):
>
> My idea is to use lib persistency

While I am a fan of [library(persistency)](https://www.swi-prolog.org/pldoc/man?section=persistency) personally I would look at [Quick Load Files](https://www.swi-prolog.org/pldoc/man?section=qlf) first.  
See related topic: [Quick Load Files](https://swi-prolog.discourse.group/t/quick-load-files/1239)

The topic of large fact files has also been discussed before, see: [Scaling to billions of facts?](https://swi-prolog.discourse.group/t/scaling-to-billions-of-facts/380)

While you may need [library(persistency)](https://www.swi-prolog.org/pldoc/man?section=persistency) to create the first fact file. Once the file is created I would convert the data into a qlf.

> [@Rscho314](#):
>
> I’d like to use [phrase\_from\_stream/2](https://www.swi-prolog.org/pldoc/doc_for?object=phrase_from_stream/2) to stream the data in constant memory from SQL to fact files. I read that the stream must be buffered. What does that mean in practice?

I can’t help with this as I have not done that.

> [@Rscho314](#):
>
> Is using memory files the way to go?

> [@Rscho314](#):
>
> Also regarding lib persistency, my understanding is that it won’t require to keep all facts in memory at all times. Is that correct?

AFAIK any fact you want access to has to be in memory. If the fact is not in memory then Prolog will not know about it. As I use Windows I would think that Windows Virtual Memory should solve this, see: [How To Manage Virtual Memory (Pagefile) In Windows 10](https://www.tomshardware.com/news/how-to-manage-virtual-memory-pagefile-windows-10,36929.html)

> [@Rscho314](#):
>
> Can you give me an example on how to proceed to stream the data, please?

While I do have some [experience](https://swi-prolog.discourse.group/t/quick-load-files/1239/4) parsing very large files into Prolog facts using DCGs, the one thing I know for sure is to make sure your DCG is deterministic. If it is not it will take quite some time to run though the variations which might not be what you want.

> [@Rscho314](#):
>
> Would you be able to provide a ballpark estimate of how much RAM I’ll need?

I can’t.

* * *

While these answers are based on my experience, you should wait to see if Jan W. gives an answer. If he has a different answer then both you and I will have learned something. 🙂

* * *

**EDIT**

If you just want to save the facts, then I would use library(persistency) as used [here](https://swi-prolog.discourse.group/t/wiki-discussion-generating-cytoscape-js-graphs-with-swi-prolog/2816/4).

After you have parsed a single entry, then just persist it using something like

```prolog
add_imports(Import_module,Export_module) :-
    (
        imports(Import_module,Export_module), !
    ;
        assert_imports(Import_module,Export_module)
    ).

```

where the functor for the fact is `imports` and values for fact is `Import_module,Export_module`.

The line

```prolog
imports(Import_module,Export_module), !

```

checks to see if the fact is already persisted and if so, the cut then prunes the running the next statement.

The next statement

```prolog
assert_imports(Import_module,Export_module)

```

is executed if the fact is not persisted and just adds an assert to the persistency journal file.

HTH

* * *

**EDIT**

```prolog
atom_codes(Ay, Ey), atom_number(Ay, Ny)

```

can this not be replaced with number\_codes/2 ?

The whole list thing had me confused until I remembered that the DCG tutorials do it that way, e.g. [Using Definite Clause Grammars in SWI-Prolog](http://www.pathwayslms.com/swipltuts/dcg/) or [Prolog DCG Primer](https://www.metalevel.at/prolog/dcg)

I think you might find [basics.pl – Various general DCG utilities](https://www.swi-prolog.org/pldoc/doc/_SWI_/library/dcg/basics.pl) of more value, especially string\_without//2.

Personally I prefer the way Wouter Beek does parsing with difference list, [dcg.pl](https://github.com/wouterbeek/prolog_library_collection/blob/master/prolog/dcg.pl)

---

_[View the full topic](https://swi-prolog.discourse.group/t/can-you-please-provide-advice-on-big-data-handling-strategy/2997)._