A program written in C++ library of swipl crashes

[ArthurWang]

I wrote a test for my application in modern C++ based on C++ lib of swipl. The test program runs very well on itself in pure C++:

int main() {
  srand(time(0));
  Person person;
  auto start = steady_clock::now();
  string persons = person.concGenPersons(1, 1000000);
  auto end = steady_clock::now();
  cout << "time used in milliseconds:"
       << duration_cast<milliseconds>(end - start).count() << endl;
  return 0;
}

--------------
Arthurs-MacBook-Pro:swipl-dict-db laowang$ test/gendata 
time used in milliseconds:565
Arthurs-MacBook-Pro:swipl-dict-db laowang$ test/gendata 
time used in milliseconds:520
Arthurs-MacBook-Pro:swipl-dict-db laowang$ test/gendata 
time used in milliseconds:519
Arthurs-MacBook-Pro:swipl-dict-db laowang$ test/gendata 
time used in milliseconds:511
Arthurs-MacBook-Pro:swipl-dict-db laowang$ test/gendata 
time used in milliseconds:524

But when I run it in swipl, it often crashed even using much less data.

PREDICATE(conc_gen_persons, 3) {
  srand(time(0));
  int fromId((int)A1);
  int toId((int)A2);
  try {
    Person p;
    string dictsStr = p.concGenPersons(fromId, toId);
    cout << "length:" << dictsStr.size() << endl;
    A3 = (PlCompound)(dictsStr.c_str());
  } catch (const exception &e) {
    throw Utils::genPlException("conc_gen_persons_exception", string(e.what()));
  } catch (...) {
    throw Utils::genPlException("conc_gen_persons_exception",
                          string("unknown reason"));
  }
  return TRUE;
}
-------------------------
?- time(ddtest:conc_gen_persons(1,100000,D)),length(D,L).

SWI-Prolog [thread -1 () at Tue Aug 20 15:10:11 2019]: received fatal signal 11 (segv)
C-stack trace labeled "crash":
  [0] /usr/local/Cellar/swi-prolog/HEAD-ad9b759/libexec/lib/swipl/lib/x86_64-darwin/libswipl.8.dylib(save_backtrace+0x62) [0x102112042]
  [1] /usr/local/Cellar/swi-prolog/HEAD-ad9b759/libexec/lib/swipl/lib/x86_64-darwin/libswipl.8.dylib(print_c_backtrace+0x15) [0x102112915]
  [2] /usr/local/Cellar/swi-prolog/HEAD-ad9b759/libexec/lib/swipl/lib/x86_64-darwin/libswipl.8.dylib(sigCrashHandler+0xf4) [0x102112824]
  [3] /usr/local/Cellar/swi-prolog/HEAD-ad9b759/libexec/lib/swipl/lib/x86_64-darwin/libswipl.8.dylib(dispatch_signal+0x6d) [0x10207336d]
  [4] /usr/local/Cellar/swi-prolog/HEAD-ad9b759/libexec/lib/swipl/lib/x86_64-darwin/libswipl.8.dylib(pl_signal_handler+0x15) [0x102076155]
  [5] /usr/lib/system/libsystem_platform.dylib(_sigtramp+0x1d) [0x7fff63408b5d]
  [7] /usr/local/opt/llvm/lib/libc++.1.dylib(_ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC2ERKS5_+0x87) [0x1026c0713]
  [8] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZN6Person13getARandomStrERKNSt3__16vectorINS0_12basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEENS5_IS7_EEEE+0x67) [0x102405cf7]
  [9] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZN6Person9getALNameEv+0x24) [0x102405a64]
  [10] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZN6Person10genAPersonEm+0x1df) [0x10240539f]
  [11] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZN6Person10genPersonsEii+0x5b) [0x102404c6b]
  [12] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZN6Person16genPersonsThreadEii+0x30) [0x102404f10]
  [13] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__18__invokeIM6PersonFviiEPS1_JiiEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS5_DpOS6_+0x9d) [0x10240794d]
  [14] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__112__async_funcIM6PersonFviiEJPS1_iiEE9__executeIJLm1ELm2ELm3EEEEvNS_15__tuple_indicesIJXspT_EEEE+0x74) [0x1024078a4]
  [15] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__112__async_funcIM6PersonFviiEJPS1_iiEEclEv+0x15) [0x102407825]
  [16] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__119__async_assoc_stateIvNS_12__async_funcIM6PersonFviiEJPS2_iiEEEE9__executeEv+0x25) [0x1024076b5]
  [17] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__18__invokeIMNS_19__async_assoc_stateIvNS_12__async_funcIM6PersonFviiEJPS3_iiEEEEEFvvEPS8_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OSC_DpOSD_+0x71) [0x102408851]
  [18] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMNS_19__async_assoc_stateIvNS_12__async_funcIM6PersonFviiEJPS8_iiEEEEEFvvEJPSD_EJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE+0x3e) [0x10240875e]
  [19] /Users/laowang/workspace/swipl-dict-db/lib/dbdict.dylib(_ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMNS_19__async_assoc_stateIvNS_12__async_funcIM6PersonFviiEJPS9_iiEEEEEFvvEPSE_EEEEEPvSJ_+0x76) [0x102407f56]
  [20] /usr/lib/system/libsystem_pthread.dylib(_pthread_body+0x7e) [0x7fff634112eb]
  [21] /usr/lib/system/libsystem_pthread.dylib(_pthread_start+0x42) [0x7fff63414249]
  [22] /usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff6341040d]
Running on_halt hooks with status 139
Killing 50544 with default signal handlers
Segmentation fault: 11
--------------------------------------
Somtimes it sucesses as follows:
?- time(ddtest:conc_gen_persons(1,100000,D)),length(D,L).
length:8814874
% 2 inferences, 0.772 CPU in 0.830 seconds (93% CPU, 3 Lips)
D = [person{dob:'1952-5-27', first_name:'Tracy', gender:male, id:25001, last_name:'Sellards'}, person{dob:'1984-5-7', first_name:'Sarai', gender:female, id:25002, last_name:'Royer'}, person{dob:'1975-11-3', first_name:'Shelby', gender:male, id:25003, last_name:'Wainer'}, person{dob:'1988-1-13', first_name:'Juan', gender:male, id:25004, last_name:'Drinkwater'}, person{dob:'1985-10-7', first_name:'Nickie', gender:female, id:25005, last_name:'Plesnarski'}, person{dob:'1980-8-22', first_name:'Spring', gender:female, id:25006, last_name:'Delee'}, person{dob:'1958-3-29', first_name:'Harry', gender:male, id:25007, last_name:'Falkner'}, person{dob:'1982-12-18', first_name:'Kaycee', gender:female, id:25008, last_name:'Darrell'}, ...{... : ..., ... : ..., ... : ..., ... : ..., ... : ...}|...],
L = 100000.
----------------------------------
ps.   I've set a big stack_limit value:
?- current_prolog_flag(stack_limit, V).
V = 8589934592.

BTW, ddtest:conc_gen_persons/3 is a member function in concurrency.

The whole source file is attached below for reference:

#define PROLOG_MODULE "ddtest"

// #include "../cxxsrc/Utils.hpp"
// #include <SWI-cpp.h>
#include <chrono>
#include <condition_variable>
#include <cstdlib>
#include <ctime>
#include <filesystem>
#include <fstream>
#include <future>
#include <iostream>
#include <mutex>
#include <string>
#include <thread>
#include <vector>

using namespace std;
using namespace std::chrono;

class Person {
private:
  vector<string> lnames_;
  vector<string> mfnames_;
  vector<string> ffnames_;
  unsigned int concurrency_;
  mutex readyMutex_;
  condition_variable readyCV_;
  string resultStr_;
  int readyCount_;

  bool getLinesToVector(const char *file, vector<string> &strings) {
    ifstream in(file);
    if (!in) {
      cerr << "Error opening file:" << file << endl;
      return false;
    }
    string str;
    while (getline(in, str)) {
      if (str.size() > 0)
        strings.push_back(str);
    }
    in.close();
    return true;
  }

  bool initData() {
    concurrency_ = thread::hardware_concurrency();
    if (concurrency_ == 0)
      concurrency_ = 2;

    return getLinesToVector("test/last-names.txt", lnames_) &&
           getLinesToVector("test/male-first-names.txt", mfnames_) &&
           getLinesToVector("test/female-first-names.txt", ffnames_);
  }

  string getARandomStr(vector<string> const &strs) {
    int n = rand() % strs.size() + 1;
    return strs[n];
  }

  string getALName() { return getARandomStr(lnames_); }
  string getAMFame() { return getARandomStr(mfnames_); }
  string getAFFame() { return getARandomStr(ffnames_); }

  int juliandate(int year, int month, int day) {
    int a = (14 - month) / 12;
    int m = month + 12 * a - 3;
    int y = year + 4800 - a;
    return day + (153 * m + 2) / 5 + 365 * y + y / 4 - y / 100 + y / 400 -
           32045;
  }

  void calendardate(const int jdate, int &year, int &month, int &day) {
    int a = jdate + 32044;
    int b = (4 * a + 3) / 146097;
    int c = a - 146097 * b / 4;
    int d = (4 * c + 3) / 1461;
    int e = c - 1461 * d / 4;
    int m = (5 * e + 2) / 153;
    day = e - (153 * m + 2) / 5 + 1;
    month = m + 3 - 12 * (m / 10);
    year = 100 * b + d - 4800 + m / 10;
  }

  struct Date {
    int year;
    int month;
    int day;
  };
  string genRandomDate(const Date date1, const Date date2) {
    int jdate1 = juliandate(date1.year, date1.month, date1.day);
    int jdate2 = juliandate(date2.year, date2.month, date2.day);
    int jdate = rand() % (jdate2 - jdate1 + 1) + jdate1;
    Date date;
    calendardate(jdate, date.year, date.month, date.day);
    return to_string(date.year) + "-" + to_string(date.month) + "-" +
           to_string(date.day);
  }

  string genAPerson(const size_t id) {
    float r = ((double)rand() / (RAND_MAX));
    string gender = r < 0.68 ? "male" : "female";
    Date date1{1950, 1, 1}, date2{1999, 12, 31};
    string date = genRandomDate(date1, date2);
    string p = string("person{id:") + to_string(id) + ", gender:" + gender +
               ", last_name:'" + getALName() + "', first_name:'";
    p += gender == "male" ? getAMFame() : getAFFame();
    p += "',dob:'" + date + "'},";
    return p;
  }

  void genPersonsThread(const int fromId, const int toId) {
    string persons = genPersons(fromId, toId);
    {
      lock_guard<mutex> lg(readyMutex_);
      resultStr_ += persons;
      readyCount_--;
    }
    readyCV_.notify_one();
  }

public:
  Person() {
    if (!initData())
      throw "Failed init data";
  }

  string genPersons(const int fromId, const int toId) {
    string persons = "";
    for (int i = fromId; i <= toId; ++i) {
      persons += genAPerson(i);
    }
    return persons;
  }

  string concGenPersons(const int fromId, const int toId) {
    int num = toId - fromId + 1;
    if (num < concurrency_ * 2)
      return genPersons(fromId, toId);

    {
      lock_guard<mutex> lg(readyMutex_);
      resultStr_ = "[";
      readyCount_ = concurrency_;
    }
    int numPerCore = num / concurrency_;
    vector<future<void>> futures;
    for (int i = 0; i < concurrency_ - 1; ++i) {
      futures.push_back(async(&Person::genPersonsThread, this,
                              numPerCore * i + 1, numPerCore * (i + 1)));
    }
    futures.push_back(async(&Person::genPersonsThread, this,
                            numPerCore * (concurrency_ - 1) + 1, toId));

    {
      unique_lock<mutex> ul(readyMutex_);
      this->readyCV_.wait(ul, [this] { return readyCount_ == 0; });
    }
    resultStr_.pop_back();
    return resultStr_ + "]";
    // return dicts_;
  }
};
/*
PREDICATE(conc_gen_persons, 3) {
  srand(time(0));
  int fromId((int)A1);
  int toId((int)A2);
  try {
    Person p;
    string dictsStr = p.concGenPersons(fromId, toId);
    cout << "length:" << dictsStr.size() << endl;
    A3 = (PlCompound)(dictsStr.c_str());
  } catch (const exception &e) {
    Utils::genPlException("conc_gen_persons_exception", string(e.what()));
  } catch (...) {
    Utils::genPlException("conc_gen_persons_exception",
                          string("unknown reason"));
  }
  return TRUE;
}
*/
int main() {
  srand(time(0));
  Person person;
  auto start = steady_clock::now();
  string persons = person.concGenPersons(1, 1000000);
  auto end = steady_clock::now();
  cout << "time used in milliseconds:"
       << duration_cast<milliseconds>(end - start).count() << endl;
  return 0;
}

Any helps are greatly appreciated!
With best regards,
Arthur Wang

[CapelliC]

Would be better to have also these files:

test/last-names.txt
test/male-first-names.txt
test/female-first-names.txt

but anyway… I’m suspicious of this line
int n = rand() % strs.size() + 1;
of getARandomStr: adding 1 to mod could access past end()

For example in JS:

var strs=[1,2,3]
for (var rand = 0; rand < 10; ++rand) console.log(strs[rand % strs.length + 1])
2
3
undefined
2
3
undefined
2
3
undefined
2
undefined

[ArthurWang]

Thank you very much!
It seems the “+1” is the culprit.
BTW, the 3 txt’s file are very big files for human names used to generate random test persons data.

With kind regards,
Arthur Wang