# A tokeniser I've written. Any suggestions on how to improve it?

**URL:** <https://swi-prolog.discourse.group/t/a-tokeniser-ive-written-any-suggestions-on-how-to-improve-it/367>\
**Category:** Help!\
**Created:** [March 7, 2019, 9:37am UTC](https://swi-prolog.discourse.group/t/a-tokeniser-ive-written-any-suggestions-on-how-to-improve-it/367 "2019-03-07T09:37:10Z")\
**Posts on this page:** 1\
**Showing post:** 27

<div class="post-metadata">

**Author:** ![joeblog](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/joeblog/32/122_2.png) [@joeblog](https://swi-prolog.discourse.group/u/joeblog)\
**Post date:** [April 3, 2019, 12:33pm UTC](https://swi-prolog.discourse.group/t/a-tokeniser-ive-written-any-suggestions-on-how-to-improve-it/367/27 "2019-04-03T12:33:56Z")

</div>

Many thanks for those pointers Jan.

I’ve realised a DCG is the most elegant approach after getting to grips with the various ways of iterating in Prolog which I described here [Six ways to iterate in Prolog](https://swi-prolog.discourse.group/t/six-ways-to-iterate-in-prolog/477)

I’ve also realised [A parsing example](https://swi-prolog.discourse.group/t/a-parsing-example/425/5) I took from The Art of Prolog is overcomplicated and could be boiled down to one DCG (something I’ll tackle next).

Here’s the simplified tokeniser I’ve settled on.

```prolog
:- module(tokeniser, [string_tokens/2]).

%% tokenise(+String, -Tokens) is det

string_tokens(String, Tokens) :-
  string_chars(String, Chars),
  phrase(tokens(Tokens), Chars).

% Definite Clause Grammar (DCG)

tokens([Token|Tokens]) --> ws, token(Token), ws, !, tokens(Tokens).
tokens([]) --> [].

ws --> [W], { char_type(W, space) }, ws.
ws --> [].

token(P) --> [C], { char_type(C, punct), \+char_type(C, quote), string_chars(P, [C]) }.
token(Q) --> quote(Cs), { string_chars(Q, Cs) }.
token(W) --> word(Cs), { string_chars(W, Cs) }.

quote([Quote|Ls]) --> [Quote], { char_type(Quote, quote) }, quote_rest(Quote, Ls).
quote_rest(Quote, [L|Ls]) --> [L], { L \= Quote }, quote_rest(Quote, Ls).
quote_rest(Quote, [Quote]) --> [Quote]. 

word([L|Ls]) --> [L], { char_type(L, alnum) }, word_rest(Ls).
word_rest([L|Ls]) --> [L], { char_type(L, alnum) }, word_rest(Ls).
word_rest([]) --> [].

```

---

_[View the full topic](https://swi-prolog.discourse.group/t/a-tokeniser-ive-written-any-suggestions-on-how-to-improve-it/367)._