# Improving contributor guide discoverability (was: Consolidating the 71 GitHub repositories to simplify maintenance and contribution) **URL:** \ **Category:** General\ **Created:** [April 5, 2019, 2:54pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492 "2019-04-05T14:54:10Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![edom](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/edom/32/522_2.png) [@edom](https://swi-prolog.discourse.group/u/edom)\ **Post date:** [April 5, 2019, 2:54pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/1 "2019-04-05T14:54:10Z")
Dear SWI-Prolog maintainers, SWI-Prolog has [71 Git repositories](https://github.com/SWI-Prolog) and 4 maintainers. This sparsity presents some difficulties to both maintainers, contributors, and would-be contributors. Fortunately there is a reasonably easy solution. We can combine all those repositories into one (or some small number) using `git-subtree` which preserves history. For the issue tracker, we can make a label for each package. Such consolidation will simplify maintenance and reduce the barrier to people who want to contribute. How it will simplify maintenance: Now we only need to make one commit to update CMake version instead of tens of commits spread in tens of repositories. How it will simplify issue reporting: Now people don’t have to find out which repository to file the issues in. They open an issue in swipl-devel, and a maintainer attaches a suitable label that indicates the package. What do you think? Best regards, Erik ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [April 5, 2019, 3:22pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/2 "2019-04-05T15:22:54Z")
Interesting idea. I kind of got used to submodules. The model might seem a bit complicated, but is easy to understand and once you get it is quite trivial what to do. The number of modules is also not that bad: just 37 🙂 make up SWI-Prolog. The rest is stuff that is (no longer) related to the core system. That said, the submodule system surely leads to confusion, some of which you already mention. I’ll do some reading and see which problems it solved and creates. Might take some time though, but there is no hurry. Thanks for the tip — Jan ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 5, 2019, 5:16pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/3 "2019-04-05T17:16:32Z")
Right now the repository is not contributor friendly --if you need to contribute to one of the packages/submodules (which is the most common case). It took me almost an hour googling answers to find out the solution. `.gitmodules` needs to be changed as done in this PR: [https://github.com/SWI-Prolog/swipl-devel/pull/356](https://github.com/SWI-Prolog/swipl-devel/pull/356) . The `.gitmodules` modification is meant for travis, but it works for any user. The reason it is easy for Jan is because he is the owner of the repo, and this problem doesn’t show up for owners of the repo, and also because he always has the proper parent directory structure. The problem only shows up for contributors. ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 5, 2019, 5:37pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/4 "2019-04-05T17:37:19Z")
In particular, I mean this line from the PR referenced above: ```prolog if ["$TRAVIS_OS_NAME" == "linux"] ; then sed -i 's~url = ..~url = https://github.com/SWI-Prolog~' .gitmodules; fi ``` ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 5, 2019, 5:47pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/5 "2019-04-05T17:47:26Z")
I think Erik’s idea has merit, look at the docs from git: ```prolog Unlike submodules, subtrees do not need any special constructions (like .gitmodules files or gitlinks) be present in your repository, and do not force end-users of your repository to do anything special or to understand how subtrees work. A subtree is just a subdirectory that can be committed to, branched, and merged along with your project in any way you want. ``` It will solve a lot of the module problems experienced by contributors. But it will be painful for Jan in the beginning 😭 But I think it is worth it to expand the contributions. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [April 5, 2019, 5:54pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/6 "2019-04-05T17:54:34Z")
I’ll surely read into it. As is, the simple way is to clone the whole thing, init all submodules. That allows you to build the system. To make a PR for a module, fork the module on github, use `git remote add` to make your fork accessible from the cloned submodule. Then do your edit work and push to your added remote. First thing is to understand what exactly `git subtree` does, what gets easier and what gets harder. ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 5, 2019, 6:24pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/7 "2019-04-05T18:24:15Z")
The normal quick contributor just would like to do a quick patch of the documentation, or some small bug with a one-line change, etc. I’ll show you what happens to that contributor **who has forked swipl-devel** in his github user account: ```prolog $ git clone https://github.com/someuser/swipl-devel Cloning into 'swipl-devel'... remote: Enumerating objects: 184191, done. remote: Total 184191 (delta 0), reused 0 (delta 0), pack-reused 184191 Receiving objects: 100% (184191/184191), 80.62 MiB | 3.26 MiB/s, done. Resolving deltas: 100% (147549/147549), done. $ cd swipl-devel $ git submodule update --init Submodule 'bench' (https://github.com/erlanger/bench.git) registered for path 'bench' Submodule 'debian' (https://github.com/erlanger/distro-debian.git) registered for path 'debian' Submodule 'packages/PDT' (https://github.com/erlanger/packages-PDT.git) [....submodule registration...] Submodule 'packages/zlib' (https://github.com/erlanger/packages-zlib.git) registered for path 'packages/zlib' Cloning into '/tmp/swipl-devel/bench'... Username for 'https://github.com': <<<--------- LOOK HERE ``` Uhh? It is asking for the user name? The user who just wants to make a one-line change will simply say: “why is it asking me for the user name? This is too hard I’ll do it sometime later”, the end result: we’ll never get the contribution. **The more persistent** user will start googling around, and figure out that it is asking for the user name because of the way `.gitmodules` is set up. Then he will figure out an hour later, that he has to change `.gitmodules` the way it is described in the PR I showed above. This is why travis can’t build SWI-Prolog without the patch in the PR I mentioned. The reason why Jan has never experienced this is because he is the owner of the repo. Jan, you would see the above if you fired up a VM, fork swipl-devel from a new github account, and try to make a one line patch as if you were not the author of the project. ---
**Author:** ![edom](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/edom/32/522_2.png) [@edom](https://swi-prolog.discourse.group/u/edom)\ **Post date:** [April 5, 2019, 6:39pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/8 "2019-04-05T18:39:27Z")
One problem: Before we merge, we have to make sure that there is no code left behind in the branches of the submodules. For example, the [packages-ssl](https://github.com/SWI-Prolog/packages-ssl) repository has several branches (base64\_newline, cmake, etc.). We have to merge all them into master if we don’t want to lose the changes in those branches. One way to simplify development (which I use myself) is to not use branches: 1. Do everything in the `master` branch. 2. The `master` branch must always work. It does not have to be Jan who merges all the repositories. 🙂 As an [example](https://github.com/edom/swipl-devel/tree/master/packages/ssl), I have merged `package-ssl:master` into my fork of `swipl-devel:master`. The difference from the original is: Everyone who clones this repository also immediately gets the contents of `package-ssl:e9d0a9e` in the `swipl-devel/package/ssl` directory. That is, vanilla `git-clone` works as expected. Then, we can delete the `packages-ssl` GitHub repository. I can easily merge the other packages (it’s just `git subtree add -P packages/ `), but only Jan can verify that all branches of child repositories have been correctly merged to their respective master branches, or discarded if those codes are unwanted. One downside of subtree: It may slow down the repository if there are too many files. In my experience, with a 100000-file repository, git rebase is unbearably slow. ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 5, 2019, 6:49pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/9 "2019-04-05T18:49:23Z")
An intermediate step, without so much work, is to simply convert modules into subtrees, and leave the merging for later. Even if we don’t use subtrees, I think `.gitmodules` has to be fixed if we want to make contribution easy. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [April 6, 2019, 7:43am UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/10 "2019-04-06T07:43:12Z")
After a bit of _bench reading_ 🙂 I’m not convinced `git subtree` is worth the trouble. It all feels a little like “I think (Prolog) modules are too complicated, put everything in a single file”. Enough people program that way anyway ☹ Submodules have had their value in the past when several of the modules were practically managed by other people. At the moment all package modules are practically in maintenance stage and this doesn’t matter too much, but I still like to be able to do so. Submodules were also intended to be shared with other Prolog systems. That too isn’t active right now, but work is going on between XSB and SWI, so who knows? I’m a big fan of branching and rebasing and the warnings do not make me very happy (we have about 45,000 files). Git is not a distributed file system. I more like, if I recall correctly, Linus Torvald’s view that a software system is a set of patches. So for now, I think we should educate people how to contribute in a comfortable way. ---
**Author:** ![edom](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/edom/32/522_2.png) [@edom](https://swi-prolog.discourse.group/u/edom)\ **Post date:** [April 6, 2019, 8:11am UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/11 "2019-04-06T08:11:26Z")
I am doing `git submodule update` when I encounter this error message: ```prolog Fetched in submodule path 'debian', but it did not contain c4718ab1a3fddaa0b2e02d52694a94c641df3b48. Direct fetching of that commit failed. ``` It only appears when `git submodule update`. When I git clone `debian` normally, I can git-log that commit, and it indeed exists. This page suggests that you may have forgotten to push something to swipl-devel. Is this the case? > **[Direct fetching of commit failed](https://www.deployhq.com/support/common-repository-errors/direct-fetching-of-commit-failed)** > > Troubleshooting submodule change errors when configuring the repository in your DeployHQ project. (Update: This solved the problem. It seems that `git submodule update` only fetches the remote `master`.) ```prolog cd debian git fetch origin devel ``` ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [April 6, 2019, 1:43pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/12 "2019-04-06T13:43:37Z")
> [@edom](#): > > Update: This solved the problem. It seems that `git submodule update` only fetches the remote `master` .) Hmm. I always do a `git pull` on the main repo and that also fetches the submodules. Possible because `devel` is also a local branch for me? In fact the only submodule you probably do not want is `debian` as it is only used to build the Ubuntu PPAs. ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 6, 2019, 3:50pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/13 "2019-04-06T15:50:22Z")
> [@jan](#): > > Submodules have had their value in the past when several of the modules were practically managed by other people. This makes sense. ---
**Author:** ![edom](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/edom/32/522_2.png) [@edom](https://swi-prolog.discourse.group/u/edom)\ **Post date:** [April 6, 2019, 4:28pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/14 "2019-04-06T16:28:25Z")
Agree. I did not foresee those use cases. Let’s stick to submodules and forget about subtrees for now. Submodules are not too hard. We can help future contributors avoid swi’s problem by putting a prominent note in the contributor guide: _clone before fork_, and do not fork before clone. It turns out that this instruction is already in [unix.html](http://www.swi-prolog.org/build/unix.html), but it is two clicks away from [SubmitPatch.html](http://www.swi-prolog.org/howto/SubmitPatch.html). That is, it exists, but it is hard to find. It turns out that this is not the first time Jan has written the instructions. He did write it [once in 2015 in Google Groups](https://groups.google.com/forum/#!topic/swi-prolog/sly4UOpM42Y). Thus Jan has written it at least three times: once in the website, once in Google Groups, and once in Discourse. 🙂 Thus, I think we have found the real problem: _the newcomers cannot find the instructions_, because the instructions are three clicks away from the home page. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [April 6, 2019, 5:03pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/15 "2019-04-06T17:03:16Z")
I added a _clone before fork_ to SubmitPatch.html (may take an hour for the CDN to update). That saves one click. Still, people tend not to read these things. There is already a link from _COMMUNITY_ ☹ So, I guess the question becomes _"what is a good place for people to find this info_"? ---
**Author:** ![edom](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/edom/32/522_2.png) [@edom](https://swi-prolog.discourse.group/u/edom)\ **Post date:** [April 6, 2019, 9:40pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/16 "2019-04-06T21:40:32Z")
> […] people tend not to read these things […] > > So, I guess the question becomes _"what is a good place for people to find this info_ "? That question is insightful. People tend not to read these things because they are not cloning when they are at that page. The information should be at where they are when they are cloning: the swipl-devel GitHub repository. The information, the person, and the task must be _near to each other in space and time_. Ideally, the information is presented right where people need it when they need it. The question becomes “_Where are they when they need that information?_” The answer: They probably are at [swipl-devel](https://github.com/SWI-Prolog/swipl-devel) at GitHub, after searching for “swi prolog source code” in Google. (I may be wrong. You may have a more accurate answer from the website statistics.) Thus, I think _the best place for that information is the README.md file in swipl-devel_, because people will be looking at that when they are cloning. The readme is as close as possible to the “Fork” and “Clone” button as GitHub allows. The readme is the only place that is _zero_ clicks away from where people are when they are cloning. Also, we can assume that people want to build the source _right after_ they clone it, so the information about building should be placed _right after_ the information about cloning. Then, they will want to install it, run it, learn about it, play with it, write big programs in it, contribute to it, and so on. Thus the _sequence of information_ in README.md should _follow_ that most likely _sequence of tasks_ done by a new contributor. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [April 7, 2019, 8:02am UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/17 "2019-04-07T08:02:23Z")
Makes sense. See [https://github.com/SWI-Prolog/swipl-devel](https://github.com/SWI-Prolog/swipl-devel). Please suggest improvements if you think this can be done better. ---
**Author:** ![edom](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/edom/32/522_2.png) [@edom](https://swi-prolog.discourse.group/u/edom)\ **Post date:** [April 7, 2019, 9:43am UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/18 "2019-04-07T09:43:00Z")
I made a draft pull request. Let’s continue the discussion there. > ---
**Author:** ![swi](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@swi](https://swi-prolog.discourse.group/u/swi)\ **Post date:** [April 7, 2019, 2:41pm UTC](https://swi-prolog.discourse.group/t/improving-contributor-guide-discoverability-was-consolidating-the-71-github-repositories-to-simplify-maintenance-and-contribution/492/19 "2019-04-07T14:41:38Z")
Thr new README.md is much better to solve this problem, I think this will help much. Especially because it is on the top of the README.