# How to detect duplicate facts in the knowledge base?

**URL:** <https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547>\
**Category:** Help!\
**Tags:** how-to\
**Created:** [June 23, 2022, 1:25am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547 "2022-06-23T01:25:53Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![sgodoyc](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/sgodoyc/32/3158_2.png) [@sgodoyc](https://swi-prolog.discourse.group/u/sgodoyc)\
**Post date:** [June 23, 2022, 1:25am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/1 "2022-06-23T01:25:53Z")

</div>

Hi,  
Just by error, I found that you can **assert** the exact same fact (same functor and same argument) as many times as you want.

So, my question is How can I detect and remove the duplicate clauses? If the clauses are identical, does not look like **retract** can help.

```prolog
% your code here
insect(fly).
insect(bee).
insect(ant).
insect(fly).
insect(fly).
insect(fly).

```

---

<div class="post-metadata">

**Author:** ![kuniaki.mukai](https://avatars.discourse-cdn.com/v4/letter/k/c67d28/32.png) [@kuniaki.mukai](https://swi-prolog.discourse.group/u/kuniaki.mukai)\
**Post date:** [June 23, 2022, 3:10am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/2 "2022-06-23T03:10:20Z")

</div>

This is a partial solution to your post, which, I doubt,  
what you intended.

```prolog
% ?- find_multiples(S, S0, Dups).
%@ S = [fly, bee, ant, fly, fly, fly],
%@ S0 = [ant, bee, fly],
%@ Dups = [fly, fly, fly] 

find_multiples(S, S0, Dups):- findall(X, insect(X), S),
	sort(S, S0),
	foldl(select, S0, S, Dups).

insect(fly).
insect(bee).
insect(ant).
insect(fly).
insect(fly).
insect(fly).

```

---

<div class="post-metadata">

**Author:** ![chansey97](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@chansey97](https://swi-prolog.discourse.group/u/chansey97)\
**Post date:** [June 23, 2022, 3:10am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/3 "2022-06-23T03:10:52Z")

</div>

IMO, a better approach is to avoid duplicate facts instead of detecting and removing them afterward.

The following are some possible options:

1. When asserting a new fact, check and throw it away if it’s a duplicate.
2. Maintain a unique key for each fact.

That would make your life easier.

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [June 23, 2022, 5:43am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/4 "2022-06-23T05:43:14Z")

</div>

If you want to detect and remove duplicates:

```prolog
?- forall((clause(insect(A), Body, Ref), clause(insect(A), Body, Ref2), Ref @< Ref2), format('~q ~w ~w~n', [insect(A), Ref, Ref2])).
insect(fly) <clause>(0x566ed049e8a0) <clause>(0x566ed05cede0)
insect(fly) <clause>(0x566ed049e8a0) <clause>(0x566ed05cecc0)
insect(fly) <clause>(0x566ed049e8a0) <clause>(0x566ed05ceba0)
insect(fly) <clause>(0x566ed05cecc0) <clause>(0x566ed05cede0)
insect(fly) <clause>(0x566ed05ceba0) <clause>(0x566ed05cede0)
insect(fly) <clause>(0x566ed05ceba0) <clause>(0x566ed05cecc0)

```

You can use erase/1 to remove the duplicates.

---

<div class="post-metadata">

**Author:** ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://swi-prolog.discourse.group/u/EricGT)\
**Post date:** [June 23, 2022, 6:27am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/5 "2022-06-23T06:27:29Z")

</div>

> [@sgodoyc](#):
>
> Just by error, I found that you can **assert** the exact same fact (same functor and same argument) as many times as you want.
> 
> So, my question is How can I detect and remove the duplicate clauses?

Quick answer:

You want to check for the fact or assert the fact. The or as used in the sentence means logical or which translates to Prolog as the `;` operator.

Here is an example

```prolog
add_imports(Module,Import) :-
    (
        imports(Module,Import), !
    ;
        assertz(imports(Module,Import))
    ).

```

This is the first part of the statement that checks if the fact already exist.

```prolog
imports(Module,Import)

```

This is the second part of the statement that asserts the fact if the first part of the `or` fails.

```prolog
assertz(imports(Module,Import))

```

Note the use of logical or (`;`) and the cut (`!`). The cut is used to avoid leaving behind unnecessary choice points.

* * *

The example used is actually from real working code from the post

[Basic persistency](https://swi-prolog.discourse.group/t/basic-persistency/2808)

which in the example is

```prolog
add_imports(Module,Import) :-
    (
        imports(Module,Import), !
    ;
        assert_imports(Module,Import)
    ).

```

but you have to understand that when using [library(persistency)](https://www.swi-prolog.org/pldoc/man?section=persistency) that it will create the predicate `assert_imports/2`, thus why the example code for library(persistency) is different.

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [June 23, 2022, 7:06am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/6 "2022-06-23T07:06:30Z")

</div>

> [@EricGT](#):
>
> Note the use of logical or (`;`) and the cut (`!`). The cut is used to avoid leaving behind unnecessary choice points.

But has a larger scope than needed. What is wrong with this?

```prolog
add_imports(Module,Import) :-
    ( imports(Module,Import)
    -> true
    ; assert_imports(Module,Import)
    ).

```

---

<div class="post-metadata">

**Author:** ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://swi-prolog.discourse.group/u/EricGT)\
**Post date:** [June 23, 2022, 7:10am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/7 "2022-06-23T07:10:52Z")

</div>

> [@jan](#):
>
> What is wrong with this?
> 
> ```prolog
> add_imports(Module,Import) :-
> ( imports(Module,Import)
> -> true
> ; assert_imports(Module,Import)
> ).
> 
> ```

Nothing.

* * *

In my few years of using it the way I did (`;` instead of conditional `->` with `;`) with library(persistency) it works as needed.

* * *

On a side note, it might be useful to add another generated predicate for library(persistency) that asserts only unique facts, then about a third to half of the lines of code I write for large modules that use library(persistency) can be removed.

I know, pull requests are desired. 🙂

---

<div class="post-metadata">

**Author:** ![chansey97](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@chansey97](https://swi-prolog.discourse.group/u/chansey97)\
**Post date:** [June 23, 2022, 8:20am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/8 "2022-06-23T08:20:37Z")

</div>

The following is the approach that implement @peter.ludemann 's idea.

```prolog
:- dynamic insect/1.

insect(fly).
insect(bee).
insect(ant).
insect(fly).
insect(fly).
insect(fly).

?- setof(Ref2, Ref^A^Body^(clause(insect(A), Body, Ref), clause(insect(A), Body, Ref2), Ref @< Ref2), RList),
   maplist([R] >> erase(R), RList), listing(insect(_)).
:- dynamic insect/1.

insect(fly).
insect(bee).
insect(ant).

RList = [<clause>(00000000026AD930),<clause>(00000000026AE350),<clause>(00000000026AE3B0)].

```

Here the `Ref` plays the role of the unique key for each fact.

---

<div class="post-metadata">

**Author:** ![sgodoyc](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/sgodoyc/32/3158_2.png) [@sgodoyc](https://swi-prolog.discourse.group/u/sgodoyc)\
**Post date:** [June 23, 2022, 5:05pm UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/9 "2022-06-23T17:05:14Z")

</div>

Ohh…  
I see.  
Aparently a carefull combination of **clause/3** with ducplicate checking is all that is needed…

Thank you very much all for your insights…

---

<div class="post-metadata">

**Author:** ![drspro](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/drspro/32/3509_2.png) [@drspro](https://swi-prolog.discourse.group/u/drspro)\
**Post date:** [June 24, 2022, 10:55am UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/10 "2022-06-24T10:55:00Z")

</div>

or this?

```prolog
asser_insect( Insect_atom ):- insect( Insect_atom ),!. % already asserted
asser_insect( Insect_atom ):- assert( insect( Insect_atom ) ).

```

---

<div class="post-metadata">

**Author:** ![sgodoyc](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/sgodoyc/32/3158_2.png) [@sgodoyc](https://swi-prolog.discourse.group/u/sgodoyc)\
**Post date:** [June 24, 2022, 7:30pm UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/11 "2022-06-24T19:30:20Z")

</div>

That last one from **dspro** looks incredibly simple… but it seems to work fine.

Worth studying it…

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [June 24, 2022, 7:46pm UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/12 "2022-06-24T19:46:23Z")

</div>

> [@drspro](#):
>
> ```prolog
> asser_insect( Insect_atom ):- insect( Insect_atom ),!. % already asserted
> asser_insect( Insect_atom ):- assert( insect( Insect_atom ) ).
> 
> ```

I’m not a fan of cuts, so I’d write it:

```prolog
assert_insect(Insect) :-
    assertion(ground(Insect)),
    ( insect(Insect)
    -> true
    ; assertz(insect(Insect))
    ).

```

The assertion/1 is because this code could have unexpected results if `Insect` is a term that contains a variable.

Another way of doing this is to add the table/1 directive:

```prolog
?- dynamic insect/1.
true.

?- table insect/1.
true.

?- assertz(insect(fly)).
true.

?- assertz(insect(flea)).
true.

?- assertz(insect(fly)).
true.

?- bagof(Insect, insect(Insect), Insects).
Insects = [flea, fly].

```

If the table/1 directive is removed, the bagof/3 query gets `Insects = [fly, flea, fly]`.

PS: assert/1 is deprecated and assertz/1 is recommended instead.

---

<div class="post-metadata">

**Author:** ![sgodoyc](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/sgodoyc/32/3158_2.png) [@sgodoyc](https://swi-prolog.discourse.group/u/sgodoyc)\
**Post date:** [June 24, 2022, 9:29pm UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/13 "2022-06-24T21:29:18Z")

</div>

Ok,  
I guess the cut ruins the non-determinism of the predicate.  
But yes, **peter.ludemann** ’s version with **if-then-else** looks more elegant…

However the behavior of **assertion()** will require some more experiments to clearly understand it…  
The tabling solution is very clear, although not so sure if the memory requirement makes it worth…

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [June 24, 2022, 10:12pm UTC](https://swi-prolog.discourse.group/t/how-to-detect-duplicate-facts-in-the-knowledge-base/5547/14 "2022-06-24T22:12:07Z")

</div>

> [@sgodoyc](#):
>
> the behavior of **assertion()** will require some more experiments to clearly understand it…

Suppose that instead of `insect/1`, you had `insect(CommonName, ScientificName)`, and there’s no `assertion((ground(CommonName), ground(ScientificName)))`:

```prolog
assert_insect(CommonName, ScientificName) :-
    ( insect(CommonName, ScientificName)
    -> true
    ; assertz(insect(CommonName, ScientificName))
    ).

?- assert_insect(cockroach, blattodea).
?- assert_insect(ant, _).
?- assert_insect(ant, formicidae).

```