Not signed in (Sign In)

Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

  • Sign in using OpenID

Site Tag Cloud

2-category 2-category-theory abelian-categories adjoint algebra algebraic algebraic-geometry algebraic-topology analysis analytic-geometry arithmetic arithmetic-geometry book bundles calculus categorical categories category category-theory chern-weil-theory cohesion cohesive-homotopy-type-theory cohomology colimits combinatorics comma complex complex-geometry computable-mathematics computer-science constructive cosmology deformation-theory descent diagrams differential differential-cohomology differential-equations differential-geometry digraphs duality elliptic-cohomology enriched fibration finite foundation foundations functional-analysis functor gauge-theory gebra geometric-quantization geometry graph graphs gravity grothendieck group group-theory harmonic-analysis higher higher-algebra higher-category-theory higher-differential-geometry higher-geometry higher-lie-theory higher-topos-theory homological homological-algebra homotopy homotopy-theory homotopy-type-theory index-theory integration integration-theory k-theory lie-theory limits linear linear-algebra locale localization logic mathematics measure-theory modal modal-logic model model-category-theory monad monads monoidal monoidal-category-theory morphism motives motivic-cohomology nlab noncommutative noncommutative-geometry number-theory of operads operator operator-algebra order-theory pages pasting philosophy physics pro-object probability probability-theory quantization quantum quantum-field quantum-field-theory quantum-mechanics quantum-physics quantum-theory question representation representation-theory riemannian-geometry scheme schemes set set-theory sheaf simplicial space spin-geometry stable-homotopy-theory stack string string-theory superalgebra supergeometry svg symplectic-geometry synthetic-differential-geometry terminology theory topology topos topos-theory tqft type type-theory universal variational-calculus

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

Welcome to nForum
If you want to take part in these discussions either sign in now (if you have an account), apply for one now (if you don't).
    • CommentRowNumber1.
    • CommentAuthorAndrew Stacey
    • CommentTimeAug 17th 2011

    There was a query at HowTo on downloading the nLab. It’s no longer relevant since when updating the software, I re-enabled “all pages” and “recently revised” (but not export) since I wanted a clean sheet to measure the performance better. I’ve replied at HowTo but I’m copying it here since I’ll remove that query box after a short time as I don’t think that HowTo is a good place for query boxes.

    The following command line magic depends on the list of all pages which no longer works. Is there any other way to download a copy of the nLab?

    Andrew: The “all pages” is currently reenabled, but there are alternative methods depending on exactly what you want to do with the copy. Best route would be to join us in the nForum and discuss it so that we can come up with the best method.

    (For housekeeping, I’ll delete this query box in a few days.)

    • CommentRowNumber2.
    • CommentAuthorMike Shulman
    • CommentTimeJul 31st 2017

    I have deleted this query box. (Six years seems long enough to count as “a few days”.)

    • CommentRowNumber3.
    • CommentAuthorMike Shulman
    • CommentTimeJul 31st 2017

    Although it does seem that the method suggested at HowTo doesn’t currently work. At least, http://ncatlab.org/nlab/list gives me a bad gateway.

    It seems like the most recent discussion about downloading the nlab was this one. I didn’t really follow the conclusion; what should be put on the HowTo?

    • CommentRowNumber4.
    • CommentAuthorUrs
    • CommentTimeAug 1st 2017
    • (edited Aug 1st 2017)

    I don’t know about any of this, except that since many months Bas Spitters has been highlighting by email that there is no means anymore to download the nLab. Bas in turn was writing in reaction to some people he is in contact with, people with interest in web-based mathematics, who requested instructions for such downloads. Adeel has not been reacting to inquiries to this extent. I suppose he is busy with more important tasks.

    We need somebody with time and energy who could lend Adeel a hand with being system administrator.

    Some of you must be in closer contact with students of computer science than I am. Could it be an interesting thesis project for a student with interest in web technology to look into some aspects of the nnLab installation?

    • CommentRowNumber5.
    • CommentAuthorDmitri Pavlov
    • CommentTimeAug 1st 2017

    As far as I understand, Instiki stores all articles internally in an SQL database.

    At one point in the past this SQL database was periodically exported to a Bazaar repository, which could then be pulled externally, thus providing a convenient way to fetch (and backup) the entire nLab: once the database was downloaded, one simply had to do bzr pull once in a while to keep it updated. This is what I did, anyway.

    It seems to me that after the mgiration to a different server the Bazaar export feature was dropped (it is not in the Instiki code as far as I can see).

    Perhaps the SQL database could be made available via HTTP as a read-only file? This would require very little effort.

    Alternatively, if somebody gives me the SQL file, I can write a very short script that would periodically export the database to a git repository, which can then be made publicly accessible in the same manner as the bzr repository, in particular it can be updated with git pull. This requires very little effort and can be done very quickly.

    Urs, what do you think?

    • CommentRowNumber6.
    • CommentAuthorUrs
    • CommentTimeAug 1st 2017

    Urs, what do you think?

    Thanks for offering help! You should please contact Adeel Khan with this, let him know what you offer to do and what you need for that. If you don’t have his email address, let me know.

    • CommentRowNumber7.
    • CommentAuthorDmitri Pavlov
    • CommentTimeAug 1st 2017

    I emailed Adeel and you, let’s see how it works out.

    • CommentRowNumber8.
    • CommentAuthorRichard Williamson
    • CommentTimeAug 1st 2017
    • (edited Aug 1st 2017)

    To state the obvious, one should certainly not expose the SQL database itself (via HTTP or any other way). If a copy of it is made on the server every so often (via a cron job, say), and this copy is made available, that would be perfectly fine of course.

    • CommentRowNumber9.
    • CommentAuthorspitters
    • CommentTimeAug 23rd 2017

    Dmitri, did you manage to get this to work?

    • CommentRowNumber10.
    • CommentAuthorDmitri Pavlov
    • CommentTimeAug 29th 2017

    I talked to Adeel about this, he is currently traveling.

    Once he is back, we will implement it.

    • CommentRowNumber11.
    • CommentAuthoradeelkh
    • CommentTimeSep 3rd 2017
    • (edited Sep 3rd 2017)

    I apologize for the delay on this. I finally set this up again, using a git repository instead of the bazaar repository we had before. Anyone interested in using this should e-mail me their SSH public key (follow these instructions) and I’ll give you access. You will then be able to clone the git repository from nlab-git@saunders.phil.cmu.edu:nlab-content. To save server resources you can download a “bare” version of the repository with

    git clone --bare nlab-git@saunders.phil.cmu.edu:nlab-content
    

    and then run

    git clone nlab-content.git nlab-content
    rm nlab-content.git
    cd nlab-content
    git remote set-url origin nlab-git@saunders.phil.cmu.edu:nlab-content
    

    to get a “normal” repository with a working tree. Use git pull inside the working directory to sync it with the server (you can set up a https://help.ubuntu.com/community/CronHowto to do this every day, for example).

    • CommentRowNumber12.
    • CommentAuthorUrs
    • CommentTimeSep 3rd 2017
    • (edited Sep 3rd 2017)

    Thanks, Adeel!!

    I have copied your instructions over to How to download a local copy of the nLab?

    • CommentRowNumber13.
    • CommentAuthorkohlhase
    • CommentTimeSep 5th 2017
    Thanks this the git repos. This makes access much simpler.

    I am assuming that the git repository is updated periodically? Could you clarify?
    • CommentRowNumber14.
    • CommentAuthoradeelkh
    • CommentTimeSep 5th 2017

    Yes, there is a cronjob running on the server to commit any new changes every hour.

    • CommentRowNumber15.
    • CommentAuthorRodMcGuire
    • CommentTimeSep 5th 2017
    • (edited Sep 5th 2017)

    Yes, there is a cronjob running on the server to commit any new changes every hour.

    Could that be what causes the nLab to apparently hang when I try to submit an edit, or is that some other problem? Canceling and then re-editing with the text that you have hopefully saved seems to work.

    Right now I had hanging submit that eventually returned the page

    Secure Connection Failed

    • CommentRowNumber16.
    • CommentAuthoradeelkh
    • CommentTimeSep 5th 2017

    The git repository is now mirrored at GitHub: https://github.com/ncatlab/nlab-content.

    This means in particular that you can skip sending me your public key and just run

    git clone git@github.com:ncatlab/nlab-content.git
    

    It also means that there will always be a relatively reliable mirror of the nLab on GitHub, unless something breaks at some point.

    • CommentRowNumber17.
    • CommentAuthoradeelkh
    • CommentTimeSep 5th 2017

    Yes, there is a cronjob running on the server to commit any new changes every hour.

    Could that be what causes the nLab to apparently hang when I try to submit an edit, or is that some other problem?

    Were you eventually able to save your changes or is the problem still persistent? The cronjob takes very little server resources, so more likely the reason was different; around that time I was running a script on the server to replace “bad” author names from the git repository, which GitHub wasn’t accepting as valid (in this case, “bad” meant that they contained certain invalid characters).

    • CommentRowNumber18.
    • CommentAuthorDmitri Pavlov
    • CommentTimeSep 5th 2017

    Probably a better command for cloning is

    git clone https://github.com/ncatlab/nlab-content.git/

    The native git protocol is faster than ssh.

    • CommentRowNumber19.
    • CommentAuthorDmitri Pavlov
    • CommentTimeSep 5th 2017
    • (edited Sep 6th 2017)
    I cloned the github repo and it seems to work fine!

    One difference from the Bazaar repository is that there are no metadata files,
    so all metadata such as page titles, for example, must be pulled out of git history.

    I like to have an offline copy of nLab for traveling,
    which is why I mentioned this---I need to pull out all page titles to create an index to all nLab pages,
    which I can then use to quickly access the desired page.

    For the record, here is a script I currently use to create such an index:

    git -C nlab-content log --name-only --grep='Page name:' --format=format:$'\n'%B | sed -n 's/^Page name: //p;s@pages/@@;/^[0-9]*$/p' | tr \\n $'\x1c' | sed 's/\x1c\x1c/\n/g' | sed -n 's/^\x1c//;s/\x1c$//;s/^\([^\x1c]*\)\x1c\([^\x1c]*\)$/\2 \1/p' | sort -u -t' ' -k1,1 -n >nindex


    I also discovered a Unicode encoding problem: for instance, type

    git show 478b6cba2f8f00b99b326942905cd5800ac30735

    and notice how "Kähler manifold" has garbage instead of ä.

    Can we fix this? I presume this is a simple matter of adding an option to the python script.
    • CommentRowNumber20.
    • CommentAuthorkohlhase
    • CommentTimeSep 6th 2017
    I agree with Dmitri,

    it would be good to have a version of the files with all the metadata, preferrably in one file. I am having (as a newbie) problems mapping the pages to html files.

    BTW, your script gives an empty file nindex for me (on MacOS).

    Another reason to have metadata preprocessed on the server :-).
    • CommentRowNumber21.
    • CommentAuthorDmitri Pavlov
    • CommentTimeSep 7th 2017
    macOS (via FreeBSD) has its own implementations of many Unix utilities, which sometimes have a different behavior from their Linux counterparts.
    So the script might not work on macOS.
    • CommentRowNumber22.
    • CommentAuthoradeelkh
    • CommentTimeSep 16th 2017
    • (edited Sep 16th 2017)

    I’ve added metadata files to the repository. It’s only the page title, let me know if you need anything else.

    I looked into the unicode issue Dmitri mentioned. It turns out that the database encoding was set to latin1 for some reason. I’ve converted everything over to utf8 and I’m now setting up the git repository again from scratch (it will take a day or two I guess).

    After that I’ll look into adding html format files. I think it’s best to put them in a separate branch.

    • CommentRowNumber23.
    • CommentAuthorDmitri Pavlov
    • CommentTimeSep 16th 2017
    Re #22: Thanks, Adeel! We appreciate all the work you are doing for the nLab!
    • CommentRowNumber24.
    • CommentAuthorDmitri Pavlov
    • CommentTimeSep 16th 2017
    Re #22: A minor thing: it would be nice to set up proper Unix permissions on all files
    so that text files are not executable.
    • CommentRowNumber25.
    • CommentAuthoradeelkh
    • CommentTimeSep 18th 2017

    Ok, the new repository has been set up (at the same URL: https://github.com/ncatlab/nlab-content). If you’ve already downloaded it, you should delete your local copy and clone it again; git pull will give you a lot of errors.

    • CommentRowNumber26.
    • CommentAuthoradeelkh
    • CommentTimeSep 23rd 2017

    There’s now a (separate) HTML repository at https://github.com/ncatlab/nlab-content-html.

    • CommentRowNumber27.
    • CommentAuthorDmitri Pavlov
    • CommentTimeSep 23rd 2017
    The *.meta files use a single newline character (LF),
    whereas *.md files use Windows-style newline characters (CR LF).

    This causes strange effects when using git tools such as git diff on the repository.
    Can we get rid of the CR characters in the *.md files?