## Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

## Site Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

• CommentRowNumber1.
• CommentAuthorAndrew Stacey
• CommentTimeAug 17th 2011

There was a query at HowTo on downloading the nLab. It’s no longer relevant since when updating the software, I re-enabled “all pages” and “recently revised” (but not export) since I wanted a clean sheet to measure the performance better. I’ve replied at HowTo but I’m copying it here since I’ll remove that query box after a short time as I don’t think that HowTo is a good place for query boxes.

The following command line magic depends on the list of all pages which no longer works. Is there any other way to download a copy of the nLab?

Andrew: The “all pages” is currently reenabled, but there are alternative methods depending on exactly what you want to do with the copy. Best route would be to join us in the nForum and discuss it so that we can come up with the best method.

(For housekeeping, I’ll delete this query box in a few days.)

• CommentRowNumber2.
• CommentAuthorMike Shulman
• CommentTimeJul 31st 2017

I have deleted this query box. (Six years seems long enough to count as “a few days”.)

• CommentRowNumber3.
• CommentAuthorMike Shulman
• CommentTimeJul 31st 2017

Although it does seem that the method suggested at HowTo doesn’t currently work. At least, http://ncatlab.org/nlab/list gives me a bad gateway.

• CommentRowNumber4.
• CommentAuthorUrs
• CommentTimeAug 1st 2017
• (edited Aug 1st 2017)

I don’t know about any of this, except that since many months Bas Spitters has been highlighting by email that there is no means anymore to download the nLab. Bas in turn was writing in reaction to some people he is in contact with, people with interest in web-based mathematics, who requested instructions for such downloads. Adeel has not been reacting to inquiries to this extent. I suppose he is busy with more important tasks.

We need somebody with time and energy who could lend Adeel a hand with being system administrator.

Some of you must be in closer contact with students of computer science than I am. Could it be an interesting thesis project for a student with interest in web technology to look into some aspects of the $n$Lab installation?

• CommentRowNumber5.
• CommentAuthorDmitri Pavlov
• CommentTimeAug 1st 2017

As far as I understand, Instiki stores all articles internally in an SQL database.

At one point in the past this SQL database was periodically exported to a Bazaar repository, which could then be pulled externally, thus providing a convenient way to fetch (and backup) the entire nLab: once the database was downloaded, one simply had to do bzr pull once in a while to keep it updated. This is what I did, anyway.

It seems to me that after the mgiration to a different server the Bazaar export feature was dropped (it is not in the Instiki code as far as I can see).

Perhaps the SQL database could be made available via HTTP as a read-only file? This would require very little effort.

Alternatively, if somebody gives me the SQL file, I can write a very short script that would periodically export the database to a git repository, which can then be made publicly accessible in the same manner as the bzr repository, in particular it can be updated with git pull. This requires very little effort and can be done very quickly.

Urs, what do you think?

• CommentRowNumber6.
• CommentAuthorUrs
• CommentTimeAug 1st 2017

Urs, what do you think?

Thanks for offering help! You should please contact Adeel Khan with this, let him know what you offer to do and what you need for that. If you don’t have his email address, let me know.

• CommentRowNumber7.
• CommentAuthorDmitri Pavlov
• CommentTimeAug 1st 2017

I emailed Adeel and you, let’s see how it works out.

• CommentRowNumber8.
• CommentAuthorRichard Williamson
• CommentTimeAug 1st 2017
• (edited Aug 1st 2017)

To state the obvious, one should certainly not expose the SQL database itself (via HTTP or any other way). If a copy of it is made on the server every so often (via a cron job, say), and this copy is made available, that would be perfectly fine of course.

• CommentRowNumber9.
• CommentAuthorspitters
• CommentTimeAug 23rd 2017

Dmitri, did you manage to get this to work?

• CommentRowNumber10.
• CommentAuthorDmitri Pavlov
• CommentTimeAug 28th 2017

Once he is back, we will implement it.

• CommentRowNumber11.
• CommentTimeSep 3rd 2017
• (edited Sep 3rd 2017)

I apologize for the delay on this. I finally set this up again, using a git repository instead of the bazaar repository we had before. Anyone interested in using this should e-mail me their SSH public key (follow these instructions) and I’ll give you access. You will then be able to clone the git repository from nlab-git@saunders.phil.cmu.edu:nlab-content. To save server resources you can download a “bare” version of the repository with

git clone --bare nlab-git@saunders.phil.cmu.edu:nlab-content


and then run

git clone nlab-content.git nlab-content
rm nlab-content.git
cd nlab-content
git remote set-url origin nlab-git@saunders.phil.cmu.edu:nlab-content


to get a “normal” repository with a working tree. Use git pull inside the working directory to sync it with the server (you can set up a https://help.ubuntu.com/community/CronHowto to do this every day, for example).

• CommentRowNumber12.
• CommentAuthorUrs
• CommentTimeSep 3rd 2017
• (edited Sep 3rd 2017)

• CommentRowNumber13.
• CommentAuthorkohlhase
• CommentTimeSep 5th 2017
Thanks this the git repos. This makes access much simpler.

I am assuming that the git repository is updated periodically? Could you clarify?
• CommentRowNumber14.
• CommentTimeSep 5th 2017

Yes, there is a cronjob running on the server to commit any new changes every hour.

• CommentRowNumber15.
• CommentAuthorRodMcGuire
• CommentTimeSep 5th 2017
• (edited Sep 5th 2017)

Yes, there is a cronjob running on the server to commit any new changes every hour.

Could that be what causes the nLab to apparently hang when I try to submit an edit, or is that some other problem? Canceling and then re-editing with the text that you have hopefully saved seems to work.

Right now I had hanging submit that eventually returned the page

Secure Connection Failed

• CommentRowNumber16.
• CommentTimeSep 5th 2017

The git repository is now mirrored at GitHub: https://github.com/ncatlab/nlab-content.

This means in particular that you can skip sending me your public key and just run

git clone git@github.com:ncatlab/nlab-content.git


It also means that there will always be a relatively reliable mirror of the nLab on GitHub, unless something breaks at some point.

• CommentRowNumber17.
• CommentTimeSep 5th 2017

Yes, there is a cronjob running on the server to commit any new changes every hour.

Could that be what causes the nLab to apparently hang when I try to submit an edit, or is that some other problem?

Were you eventually able to save your changes or is the problem still persistent? The cronjob takes very little server resources, so more likely the reason was different; around that time I was running a script on the server to replace “bad” author names from the git repository, which GitHub wasn’t accepting as valid (in this case, “bad” meant that they contained certain invalid characters).

• CommentRowNumber18.
• CommentAuthorDmitri Pavlov
• CommentTimeSep 5th 2017

Probably a better command for cloning is

git clone https://github.com/ncatlab/nlab-content.git/

The native git protocol is faster than ssh.

• CommentRowNumber19.
• CommentAuthorDmitri Pavlov
• CommentTimeSep 5th 2017
• (edited Sep 5th 2017)
I cloned the github repo and it seems to work fine!

One difference from the Bazaar repository is that there are no metadata files,
so all metadata such as page titles, for example, must be pulled out of git history.

I like to have an offline copy of nLab for traveling,
which is why I mentioned this---I need to pull out all page titles to create an index to all nLab pages,
which I can then use to quickly access the desired page.

For the record, here is a script I currently use to create such an index:

git -C nlab-content log --name-only --grep='Page name:' --format=format:$'\n'%B | sed -n 's/^Page name: //p;s@pages/@@;/^[0-9]*$/p' | tr \\n $'\x1c' | sed 's/\x1c\x1c/\n/g' | sed -n 's/^\x1c//;s/\x1c$//;s/^$$[^\x1c]*$$\x1c$$[^\x1c]*$$\$/\2 \1/p' | sort -u -t' ' -k1,1 -n >nindex

I also discovered a Unicode encoding problem: for instance, type

git show 478b6cba2f8f00b99b326942905cd5800ac30735

and notice how "Kähler manifold" has garbage instead of ä.

Can we fix this? I presume this is a simple matter of adding an option to the python script.
• CommentRowNumber20.
• CommentAuthorkohlhase
• CommentTimeSep 6th 2017
I agree with Dmitri,

it would be good to have a version of the files with all the metadata, preferrably in one file. I am having (as a newbie) problems mapping the pages to html files.

BTW, your script gives an empty file nindex for me (on MacOS).

Another reason to have metadata preprocessed on the server :-).
• CommentRowNumber21.
• CommentAuthorDmitri Pavlov
• CommentTimeSep 6th 2017
macOS (via FreeBSD) has its own implementations of many Unix utilities, which sometimes have a different behavior from their Linux counterparts.
So the script might not work on macOS.
• CommentRowNumber22.
• CommentTimeSep 16th 2017
• (edited Sep 16th 2017)

I’ve added metadata files to the repository. It’s only the page title, let me know if you need anything else.

I looked into the unicode issue Dmitri mentioned. It turns out that the database encoding was set to latin1 for some reason. I’ve converted everything over to utf8 and I’m now setting up the git repository again from scratch (it will take a day or two I guess).

After that I’ll look into adding html format files. I think it’s best to put them in a separate branch.

• CommentRowNumber23.
• CommentAuthorDmitri Pavlov
• CommentTimeSep 16th 2017
Re #22: Thanks, Adeel! We appreciate all the work you are doing for the nLab!
• CommentRowNumber24.
• CommentAuthorDmitri Pavlov
• CommentTimeSep 16th 2017
Re #22: A minor thing: it would be nice to set up proper Unix permissions on all files
so that text files are not executable.
• CommentRowNumber25.
• CommentTimeSep 18th 2017

Ok, the new repository has been set up (at the same URL: https://github.com/ncatlab/nlab-content). If you’ve already downloaded it, you should delete your local copy and clone it again; git pull will give you a lot of errors.

• CommentRowNumber26.