Web query help needed

Help! Predicate
1

I have access to a huge on-line database that I want to query. I do know how to set up the queries as a web browser URL, and I know how to parse the replies. Every query will be in the general form:

https://www.zzz.org/index.php?option=com_api&app=zzz&clientid=zzz&key=zzz&resource=zzz&...

And the reply from to these replies (raw source) on a web browser has the general form:

{"result":{"id":"15","location_id":"10","location": … }

I have no concerns about generating the URLs, or analyzing the replies in Prolog. However, I do not know how to use SWI-Prolog to set up my computer and to send and receive web data.

I am new to SWI-Prolog, but have it running great in a terminal window on my Mac. This is an awesome Prolog system! No concerns getting up to speed with it and its debugger. I have 20 years professional Prolog experience prior to my retirement 8 years ago, and now want to work on a personal project that requires me doing these very simple web queries, but I have no prior web programming experience. All help will, of course, be tremendously appreciated!

– Alan Newman

2

If you want to know how to handle JSON, here’s the documentation.

For communicating between a JavaScript client and a Prolog HTTP server, using JSON, here’s some “tutorial” code: GitHub - kamahen/swipl-server-js-client: Sample SWI-Prolog server with JavaScript client (elsewhere in this discussion group there are pointers to Prolog making HTTP queries; there is also an ODBC interface).

3

Thank you, Peter! The kamahen/swipl-server-js-client tutorial looks very appropriate.

Thanks also for the JSON reference. It looks like it will be very useful. Learning JSON with SWI support should be reasonably straightforward, as I have already been able to run my own examples.

However, I am not using Linux, so the getting started instructions from kamahen are not clear to me. I’m guessing that many of the installation steps are already handled in the Mac install package, but I am not sure which steps I can skip. Do I need to install software-properties-common, or add repository (?) spa:swi-prolog/devel, and if so, how do I do that with a Mac?

4

Linux (Debian, Ubuntu) uses the Advanced Package Tool or APT to install packages; it has a command line interface that is invoked with sudo apt install ...
The sudo apt-add-repository is used to make a repository (“PPA”) available.

I don’t know how to install packages on the Mac, but this page might give you some hints (assuming you use Macports or Homebrew):
https://www.swi-prolog.org/build/macos.html

5

Thanks again, Peter. I have had good initial success. The default Mac simple install seems to work fine with my very rudimentary HTTP client program. I loaded the library modules http/http_client, and http_open, and was able to invoke http_get('https://www. ... ', X, []) to instantiate X to the exact external server reply I expected, X = '{"result":{"id":"15", ... '.

My next step is to use other JSON library predicates to simplify the generation and parsing of the HTTP queries and replies before doing my analysis. This has been much easier than I expected, significantly due to your help.

If my program gets the data I need and the research shows anything useful, I will post a link to my results here, in case there is interest. The topic is worldwide gun violence as related to gun density (but with neither a pro- or anti-gun intent). If I get it published, be assured that both the collection of the source data at www.gunpolicy.org, and SWI-Prolog will get exemplary credits.

6

Hi Alan :slight_smile:
Another source that you might find useful:

Hope it helps!
/JCR

7

Anne’s tutorial is good, but it’s a bit out of date (there have been some improvements to the http library since she wrote the tutorial). That’s why I wrote my tutorial code, which was extracted from a more complex application that I wrote, initially using Anne’s tutorial.

8

I hadn’t linked the tutorial at kamahen with the author peter.ludemann. Where is that {embarrassed} emoji when you need it?

9

No worries – that’s what “kamahen” means, more or less :wink:
One of the first words I learned when I lived in Ōsaka, many years ago.

10

I finished my studies on gun violence. The hypothesis was gun violence is strongly correlated to gun density (gun deaths/100K related to number of guns/100, for all developed nations), as was presented on a graph I was given. I wanted to prove the correlation existed, but the analysis failed to demonstrate a significant correlation. On the other hand, the exercise did provide me with a great opportunity to enjoy some Prolog coding, learn SWI-Prolog, and learn how to do API web interfacing.

Since the hypothesis was not supported, I will not be doing a full writeup for publication, but I did do a quick summary of the goal, procedures, results, supporting data, and the Prolog code. I will email that write-up to anyone interested (just provide me your email address).

Here are the required libraries and the key predicate clause for accessing the web data:

:- use_module(library(http/http_open)) .
:- use_module(library(http/http_client)) . 
:- use_module(library(http/json)) .
:- use_module(library(http/json_convert)) .

ask_gunpolicy(Topic, Location, Country, Reply) :- 
   credentials(User, Key) ,
   atomic_list_concat(
      [ 'https://www .gunpolicy .org/index.php?option=com_api',
        '&app=gpodatapage',
        '&clientid=',User,
        '&key=', Key,
        '&format=raw&resource=getcategorydata',
        '&category=', Topic, 
        '&location_id=', Location
      ], Query) ,
   http_get(Query, Reply, []) , 
   atom_json_term(Reply, json([result=json(JT)]), []) ,
    {rest omitted}

This predicates takes three inputs, and writes one line of formatted data to output per invocation. Data is returned as a fourth argument mostly to handle exceptions from incorrectly formatted web responses.

Additional code generates the numerous ask_gunpolicy/4 calls for each of 50 countries, and 8 different data categories per country. The final output is comma separated data (“Country, Category1, …, Category 8,”) that is imported directly into Excel for statistical analysis and graphing. Also, each returned Category data has multiple elements that include source reference, and year/value data pairs, and the Prolog separates all the values and returns an average value. Had the study shown a significant correlation, then data would have been further broken down by year, and more of the stat analysis would have been done in Prolog (rather than just a simple averaging).’

Thanks for the much appreciated help and guidance!