Thank you for your valuable feedback! To be honest, I am quite flattered. I am a complete beginner and have only been learning coding from scratch for about 2 months now, so hearing this from the community means a lot to me.
Regarding the refactoring, you are absolutely right. I actually planned to split the module when it hit around 3,000 lines. However, as a beginner, I was deeply afraid of breaking the interconnected logic and side-effects across dependencies, so I kept putting it off, and now it has snowballed into 6,000 lines! I definitely plan to refactor it carefully using modules moving forward.
As for the performance and how I managed the execution speed without hitting memory exhaustion (OOM), my approach was purely empirical. For every block of logic or page I built, I immediately stress-tested it by injecting millions of dummy data rows into the RAM.
I literally benchmarked different iteration mechanisms and higher-order predicates against each other to see how they behave with millions of tracking points. Specifically, I compared:
findall/3, bagof/3, and setof/3 for data accumulation.aggregate_all/3 for memory-optimized statistical calculations.forall/2 and failure-driven loops (fail) to achieve a zero-memory footprint by forcing immediate internal backtracking and clearing the stacks.- Manual recursion combined with strict Tail-Call Optimization (TCO) and proper cuts (
!) to ensure the virtual machine reuses stack frames instead of bloating the global stack.
Whichever implementation clocked the fastest runtime and cleanest memory footprint under heavy stress was the one I kept. In this specific cryptography verification module (SHA-256 & AES-256), stripping away choicepoints and ensuring strict determinism was the key to handling 2,000,000 records smoothly under a tight hardware constraint (8GB RAM shared with GPU).
I am glad this sparked an interesting technical discussion across tools and languages!
To give you a concrete example, here is a snippet of how I structured the verification loop to ensure strict determinism and zero-memory bloat during the real-time console print:
cek_integritas_nota_manual(IDTx) :-
% 1. Fetch data sequentially from in-memory facts
riwayat_tx(IDTx, _Barcode, _Qty, LabaNet, Th, Bl, Tgl),
% 2. Execute the SHA-256 cryptographic verification
buat_hash_keamanan_nota(IDTx, Th, Bl, Tgl, LabaNet, HashBaru),
% 3. Print verified status directly to console
format('[INTEGRITY OK] Tx: ~w | SHA-256 Verified: ~w~n', [IDTx, HashBaru]),
!. % Strict cut to eliminate choicepoints immediately
print_verifikasi_hash_nota(Limit) :-
% 1. Extract only the IDs into a list to minimize memory footprint
findall(ID, retail_brain:riwayat_tx(ID, _, _, _, _, _, _), SemuaID),
( length(SemuaID, TotalDatabase), TotalDatabase > Limit
-> length(SampelID, Limit), append(SampelID, _, SemuaID)
; SampelID = SemuaID
),
length(SampelID, TotalAktual),
get_time(WaktuMulai),
nl, writeln('=== STARTING STEP-BY-STEP INTEGRITY PRINT ==='),
% 2. Leverage forall/2 for a failure-driven loop to clear stacks on every iteration
forall(member(IDTx, SampelID), (
retail_brain:cek_integritas_nota_manual(IDTx)
)),
writeln('=== END OF STEP-BY-STEP INTEGRITY PRINT ==='),
get_time(WaktuSelesai),
WaktuProses is WaktuSelesai - WaktuMulai,
nl,
writeln('-----------------------------------------------------------------'),
format(' MEASUREMENT TARGET : SHA-256 Data Integrity Verification~n', []),
format(' TOTAL INVOICES HASHED : ~w Invoices~n', [TotalAktual]),
format(' EXECUTION TIME : ~4f seconds~n', [WaktuProses]),
writeln('-----------------------------------------------------------------'), nl, !.
By isolating the list to just IDs, cutting choicepoints at the leaf predicate, and utilizing `forall/2` for internal backtracking, the SWI-Prolog virtual machine can keep recycling the same stack frames efficiently even when scaled up to millions of records. I used that code to test the retrieval of 2,000,000 records.
