What and how to get information on silent quit after heavy use of memory to consult experts

Help! Predicate
1

I use functor/3 for a large array which grows incrementally by double. There is annoying trouble that it is often prolog query quits silently without message after running long (e.g. 24 hours ). Is there a way to get information to reason these silent quits.

I have a naive question about functor/3. For example functor(T, f, N) for a big number N, what will happen when a consecutive large block of memory is not available. I expect modern (SWI-Prolog) memory management can prepare such a large block. I am not sure on this, but still I am afraid that the silent quit is a result caused by a failure for the system to do so, though this is merely a mental guess of mine. I want to get some information with which I could bring to an expert to consult.

2

I think this is quite unusual, unless maybe the OS terminated the SWI-Prolog process. Is this query being run from the top-level or as a script?

AFAIU the compound term T we’ll be allocated on the Prolog stacks, which means it relies on an underlying malloc implementation. There is some relevant documentation here. If malloc fails, SWI-Prolog detects the failure and raises a resource_error(stack) exception (via the C function outOfStack in pl-alloc.c), which is translated to a user-facing error as in:

?- N is 1024 * 1024 * 1024, functor(F, f, N).
ERROR: Stack limit (1.0Gb) exceeded
ERROR:   Stack sizes: local: 1Kb, global: 8Kb, trail: 0Kb
ERROR:   Stack depth: 11, last-call: 18%, Choice points: 3
ERROR:   In:
ERROR:     [11] system:functor(_2354, f, 1073741824)
ERROR:     [9] '$toplevel':toplevel_call(<compound (:)/2>)
ERROR:     [8] '$toplevel':stop_backtrace(<compound (:)/2>, _2402)
ERROR:     [7] '$tabling':'$wfs_call'(<compound (:)/2>, <compound (:)/2>)
ERROR:     [5] '$toplevel':'$execute_goal2'(<compound (:)/2>, [length:2], true)
ERROR: 
ERROR: Use the --stack_limit=size[KMG] command line option or
ERROR: ?- set_prolog_flag(stack_limit, 2_147_483_648). to double the limit.
?-
3

Thanks. So it is worth for me to investigate, who is challenging “the world record” which uses large memory.

So far, I found no difference on top-level, script, and ediprolog query, though I have not tested in a rigourous way.

Also as far as I remember, I haven’t seen any Prolog message about memory exceeding errors.

I will take time to insert some snapshot codes before and after functor calls for creating large array. I have little skill to find and manage information about prolog resources, e.g. about handling malloc.

4

EDIT:2022-Jul-23
With the same setting, I found a different results between query in terminal and Emacs Ediprolog mode. Detail log on terminal and ‘current_prolog_flag’ info are at the bottom. Ediprog-mode continues displaying the created long term after success. I had to kill the buffer with Emacs menu. Thanks to your comment, I feel making some step, though still in cloud seeing nothing to go.
end of EDIT. (The Details are at the bottom.)

I was wrong. I can see the ERROR message at terminal,
but still not at ediprolog mode on Emacs. Perhaps, a certain related prolog_flag is set wrongly. I will check it.

% swipl

Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.14-17-g851d08ab6)

SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.

Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org

For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- between(1, 30, J), L is 2^J, writeln(2^J), functor(T, f, L), fail.

2^1
2^2
2^3
2^4
2^5
2^6
2^7
2^8
2^9
2^10
2^11
2^12
2^13
2^14
2^15
2^16
2^17
2^18
2^19
2^20
2^21
2^22
2^23
2^24
2^25
2^26
2^27
ERROR: Stack limit (1.0Gb) exceeded
ERROR: Stack sizes: local: 2Kb, global: 59Kb, trail: 2Kb
ERROR: Stack depth: 11, last-call: 0%, Choice points: 4
ERROR: In:
ERROR: [11] system:functor(_15190, f, 134217728)
ERROR: [10] system:'<meta-call>'(<compound (:)/2>)
ERROR: [9] '$toplevel':toplevel_call(<compound (:)/2>)
ERROR: [8] '$toplevel':stop_backtrace(<compound (:)/2>, _15262)
ERROR: [7] '$tabling':'$wfs_call'(<compound (:)/2>, <compound (:)/2>)
ERROR:
ERROR: Use the --stack_limit=size[KMG] command line option or
ERROR: ?- set_prolog_flag(stack_limit, 2_147_483_648). to double the limit.
?-

EDIT (continued from the top) :2022-Jul-23

% swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.14-17-g851d08ab6)

?- use_module('/Users/cantor/devel/zdd/prolog/pac').

?- call_with_time_limit(100, (J is 2^30, functor(T, f, J))).
ERROR: Stack limit (1.0Gb) exceeded
ERROR:   Stack sizes: local: 1Kb, global: 0.3Mb, trail: 2Kb
ERROR:   Stack depth: 15, last-call: 20%, Choice points: 4
ERROR:   In:
ERROR:     [15] system:functor(_85510, f, 1073741824)
ERROR:     [13] time:run_alarm_goal(<compound '$alarm'/1>, <compound (:)/2>)
ERROR:     [12] system:setup_call_catcher_cleanup(<compound (:)/2>, <compound (:)/2>, _85568, <compound (:)/2>)
ERROR:     [10] time:call_with_time_limit(100, <compound (:)/2>)
ERROR:     [9] '$toplevel':toplevel_call(<compound (:)/2>)
ERROR: 
ERROR: Use the --stack_limit=size[KMG] command line option or
ERROR: ?- set_prolog_flag(stack_limit, 2_147_483_648). to double the limit.
^  Exception: (12) setup_call_catcher_cleanup(time:alarm(100, time_limit_exceeded(100), '$alarm'(105553178149024), [install(false)]), time:run_alarm_goal('$alarm'(105553178149024), user:(_81780 is 2^30, functor(_81786, f, _81780))), _89812, time:remove_alarm_notrace('$alarm'(105553178149024))) ? creep
?- 

?- current_prolog_flag(X, Y), writeln((X, Y)), fail.
xref,false
verbose_load,silent
debug,false
editor,emacsclient
encoding,utf8
sandboxed_load,false
query_debug_settings,debug(false,false)
qcompile,auto
generate_debug_info,true
optimise,false
save_history,true
emulated_dialect,swi
var_prefix,false
quasi_quotations,true
stream_type_check,loose
hyperlink_term,false
gui,true
backtrace_depth,20
prompt_alternatives_on,determinism
allow_variable_name_as_functor,false
protect_static_code,false
toplevel_goal,default
agc_margin,10000
unknown,error
fileerrors,true
apple,true
toplevel_print_anon,true
toplevel_prompt,~m~d~l~! ?- 
mitigate_spectre,false
write_attributes,ignore
table_incremental,false
packs,true
argv,[]
answer_write_options,[quoted(true),portray(true),max_depth(10),spacing(next_argument)]
rational_syntax,compatibility
table_space,1073741824
table_shared,false
version_data,swi(8,5,14,[])
message_language,ja_JP
system_thread_id,4708038144
res_keep_foreign,false
os_argv,[swipl]
c_cflags,-fPIC -pthread
autoload,true
float_zero_div,error
unload_foreign_libraries,false
tmp_dir,/tmp
debugger_show_context,false
debug_on_interrupt,true
vmi_builtin,true
float_overflow,error
float_undefined,error
c_libs,
shared_table_space,1073741824
c_cc,cc
file_name_variables,false
on_warning,print
debug_term_position,false
history,0
optimise_unify,true
threads,true
occurs_check,false
emacs_inferior_process,false
toplevel_list_wfs_residual_program,true
version,80514
break_level,0
arch,x86_64-darwin
toplevel_residue_vars,false
max_procedure_arity,1024
verbose_autoload,false
verbose,normal
color_term,true
toplevel_var_size,1000
print_write_options,[portray(true),quoted(true),numbervars(true)]
max_rational_size_action,error
portable_vmi,true
stack_limit,1073741824
char_conversion,false
bounded,false
double_quotes,string
max_tagged_integer,72057594037927935
trace_gc,false
abi_version,abi{built_in:3335887428,foreign_interface:2,qlf:68,qlf_min_load:68,record:3,vmi:2678345669}
pipe,true
gmp_version,6
float_max,1.7976931348623157e+308
max_table_answer_size_action,error
resource_database,/Users/cantor/lib/swipl/boot.prc
open_shared_object,true
posix_shell,/bin/sh
gc_thread,true
character_escapes,true
max_table_subgoal_size_action,error
max_answers_for_subgoal_action,error
message_context,[thread]
access_level,user
user_flags,silent
float_min,2.2250738585072014e-308
odict,true
clpfd_monotonic,false
backtrace_show_lines,true
backtrace_goal_depth,3
unix,true
xpce,true
signals,true
toplevel_name_variables,true
table_subsumptive,false
report_error,true
compiled_at,Jul 21 2022, 20:50:15
error_ambiguous_stream_pair,false
cmake_build_type,RelWithDebInfo
file_name_case_handling,case_preserving
cpu_count,16
table_monotonic,eager
version_git,8.5.14-17-g851d08ab6
iso,false
readline,editline
back_quotes,codes
c_ldflags,
float_underflow,ignore
c_libplso,-lswipl
float_rounding,to_nearest
large_files,true
timezone,-32400
debug_on_error,true
last_call_optimisation,true
optimise_debug,default
debugger_write_options,[quoted(true),portray(true),max_depth(10),attributes(portray),spacing(next_argument)]
compile_meta_arguments,false
file_search_cache_time,10
warn_override_implicit_import,true
backtrace,true
home,/Users/cantor/lib/swipl
on_error,print
dialect,swi
pac,true
pid,86496
executable,/Users/cantor/lib/swipl/bin/x86_64-darwin/swipl
toplevel_extra_white_line,true
determinism_error,error
toplevel_print_factorized,false
traditional,false
allow_dot_in_atom,false
gc,true
verbose_file_search,false
tty_control,true
answer_format,~p
max_char_code,1114111
shift_check,false
max_arity,unbounded
colon_sets_calling_context,true
min_tagged_integer,-72057594037927936
url_encoding,utf8
integer_rounding_function,toward_zero
prefer_rationals,false
float_max_integer,9.007199254740992e+15
shared_object_extension,so
agc_close_streams,false
address_bits,64
toplevel_mode,backtracking
shared_object_search_path,DYLD_LIBRARY_PATH
character_escapes_unicode,true
false.
5

Weird. It is just a normal error message printed to stderr/user_error. Do other errors appear normally?

6

As far as on terminal, I can see “stack size exceeded” message, and also see for swipl script.
However, Gnu Emacs in ediprolog mode, it won’t show such message even much after the specified time limit. macOS gnu Emacs on iMac intell.

% ?- call_with_time_limit(100, (J is 2^30, functor(T, f, J))).
%@ Action (h for help) ? 
%@ Action (h for help) ? abort
%@ EOF: exit (status 4)
7

I’ve seen some problems with displaying messages in the past when using Emacs’ shell, but they were mainly because SWI-Prolog didn’t know some characteristics of the shell (e.g., it’s a “dumb” terminal and doesn’t support the ANSI colour codes). I never use ediprolog but I notice it intercepts some messages with its own DWIM, so perhaps that’s the problem? (Assuming that you’re referring to this: GNU ELPA - ediprolog )

8

Yes. GNU ELPA - ediprolog. Thank you. I will read its source code with your hints in mind.
EDIT:

I have checked “ediprlog-history”, the last line of which is strange for me. I will have to check it later.

*ediprolog-history* buffer

?ediprolog- call_with_time_limit(100, (J is 2^30, functor(T, f, J))).
Action (h for help) ? 
Action (h for help) ? aabort
EOF: exit (status 4)

/Users/cantor/local/bin/pac: line 11: -g: command not found
Jul 28 01:05:12: exited abnormally with code 127.

EDIT: 2022-Jul-28

A shell script for swipl-process in ediprolog mode seems wrong in some point. I edited to drop -g option, then I could see the stack size error as on terminal. There remains a silent quit problem, but this small step is good for me for now. Thank you for hints on ediprolog.

#! /bin/sh
SWIPL=/usr/local/bin/swipl
PACLIB=${HOME}/devel/zdd/prolog
PACSETUP=${PACLIB}/pac.pl
SCRIPT=${PACLIB}/zdd/zdd.pl

${SWIPL} -q -f ${PACSETUP}

Stack Size Error in ediprolog mode buffer.

% ?- call_with_time_limit(100, (J is 2^30, functor(T, f, J))).
%@ ERROR: Stack limit (1.0Gb) exceeded
%@ ERROR:   Stack sizes: local: 1Kb, global: 23Kb, trail: 1Kb
%@ ERROR:   Stack depth: 15, last-call: 27%, Choice points: 4
%@ ERROR:   In:
%@ ERROR:     [15] system:functor(_5994, f, 1073741824)
%@ ERROR:     [13] time:run_alarm_goal(<compound '$alarm'/1>, <compound (:)/2>)
%@ ERROR:     [12] system:setup_call_catcher_cleanup(<compound (:)/2>, <compound (:)/2>, _6052, <compound (:)/2>)
%@ ERROR:     [9] '$toplevel':toplevel_call(<compound (:)/2>)
%@ ERROR:     [8] '$toplevel':stop_backtrace(<compound (:)/2>, _6116)
%@ ERROR: 
%@ ERROR: Use the --stack_limit=size[KMG] command line option or
%@ ERROR: ?- set_prolog_flag(stack_limit, 2_147_483_648). to double the limit.
%@ ^  Exception: (12) setup_call_catcher_cleanup(time:alarm(100, time_limit_exceeded(100), '$alarm'(105553139302720), [install(false)]), time:run_alarm_goal('$alarm'(105553139302720), user:(_4762 is 2^30, functor(_4768, f, _4762))), _10104, time:remove_alarm_notrace('$alarm'(105553139302720))) ? creep