Want to take part in these discussions? Sign in if you have an account, or apply for one below
Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.
1 to 38 of 38
Further maintenance is planned for the server at CMU on which the nLab and nForum runs next weekend, from about 02:00 GMT to 15:00 GMT on the 11th of March. Both the nLab and nForum will be completely down during this time.
There is a google group which we use as a backup when the nForum is down, in case communication is needed during the downtime.
https://groups.google.com/forum/#!forum/nlab-talk
Ideally I’d put a message on the nLab as well. I’ll see how easy this would be later this evening.
Now added a notice about this at the top of every nLab page. Let me know if you would like me to tweak it in some way, or to remove it.
This notice is a good idea, thanks!!
It would be nice if the page shown during the downtime could state that the downtime was scheduled and due to end at 1500.
Good idea, Richard! I see that with your help things are becoming more professional.
Thanks!
Oscar: unfortunately we have everything on the same server, including the nginx config, and it is the server which will go down for maintenance, so it is not easy for us to serve any page during the downtime (we would have to change the DNS entry for the nLab to point somewhere else). Nevertheless, I will ask Adeel, just in case there is some easy solution via ’cloudflare’, which is handling the DNS stuff (I have not looked into this/do not have access to it).
in case there is some easy solution via ’cloudflare’
Each time the nLab is down the cloudflare page claims that it offers a cached mirror page, only that the link never seems to work. But from the message it is clearly meant to work, so maybe it would be easy to have cloudflare serve some stand-in material during server downtimes?
Interesting, perhaps then there will be a possibility to do something. I have now contacted Adeel about it.
Cloudflare’s “Always Online” feature only caches a few pages, see here. Customizing the error message is indeed possible but unfortunately it is a premium feature. Pointing the DNS somewhere else would still be an option though.
Thanks Adeel! As Adeel and I discussed privately, probably it is best that we do not switch the DNS. In the (very) long run, we can look to split up some things onto separate servers, to make things a bit more robust. Probably when things are clearer regarding the financing of the nLab after the HoTT grant ends would be a good time to consider options for this.
Just received notice from CMU that there will be downtime from 23:00 GMT today to 11:00 GMT tomorrow. Have updated the notice. Not clear yet whether this is in addition to the earlier announced downtime or replaces it.
Exactly how much traffic does the nLab consume?
Since it only serves HTML pages, I would expect it to be less than 1 terabyte, say. One can purchase a virtual private server with a bandwidth of several terabytes very cheaply these days, on the order of $50/year or so. For instance, see RamNode’s prices: https://ramnode.com/vps.php
If the nLab is rewritten to serve static HTML pages only, this would likely satisfy all its needs, with the added bonus of no noticeable downtime.
By “serving static HTML pages” do you just mean caching the HTML compiled from iTeX? I’d be surprised if the software isn’t already doing that.
Exactly how much traffic does the nLab consume?
I am not sure exactly. One of the things I definitely would like to do is to add some metrics/monitoring to the nLab so that we are better informed about this kind of thing. I know what I’d like to do, it’s just a question of time.
Regarding the cloud, I definitely think this could be a good option in the long run. Did you hear back from Steve, Urs?
I only have direct experience with AWS, which would I think cost quite a lot more than $50 (in the region of several hundred per year at least I would think) but I am very willing to consider other options. Of course Adeel and all other interested parties would need to share their opinion and judgements as well.
Even if we could in principle pay for it ourselves, I would think it might be good to look for a grant of some kind. This would give some kind of ’official recognition’ to those to whom it might be useful (not me, but probably other people), might open the door to future support, and might allow for an expansion of activities, e.g. maybe somebody (again, not me, but maybe a student or something) could be employed to help with maintenance; or somebody could have as a summer job to work on robustifying some aspect of the infrastructure, or improving the content of some pages, etc; or some ’workshop’ could be held to improve certain pages or a group of pages, etc.
By “serving static HTML pages” do you just mean caching the HTML compiled from iTeX? I’d be surprised if the software isn’t already doing that.
Yes, it does that. I think Dmitri may have been referring to the rewrite of the backend I am planning/working on, which would split the nLab into (at least) two webservers (’frontend’ and ’backend’), whereupon the frontend would indeed just be a simple server providing a static site, although the backend would be stlll be doing stuff requiring programming (one option, though, would be to replace the MySQL database with a file based database, which would simplify the infrastructure).
Hmm, interesting. I think I don’t quite understand the difference between “two webservers” and caching compiled HTML, since even with two webservers there would have to be something in front of both deciding based on the request (view or edit) which webserver to hand it off to (or, based on your terminology, maybe it would be the static server in front deciding to hand the edit queries off to the other in back), which seems basically the same as deciding to pull up a cached version for view requests. But I don’t have a lot of experience with webserver design.
However, I am in general very much in favor of storing web page data in filesystems. For one thing, it makes them much easier to manipulate, transfer, and back up using standard command-line tools, and they can also be versioned using a standard vcs like git. I love SQL databases, but I think they are best suited to storing highly structured and interconnected data (especially where each datum is not particularly large) of which one wants to ask complicated questions; for storing an essentially flat collection of large chunks of text I prefer filesystems.
I also agree that we should look for some kind of grant support.
I’ll see Steve next week for the MURI meeting; I’ll try to remember to talk to him about this then.
Did you hear back from Steve, Urs?
I haven’t. But I had had you in the cc, right? Maybe you could reply in that thread, to remind Steve. I imagine he might at times have other priorities than discussion of nLab matters in his Inbox.
To be fair, Steve is probably quite busy right now preparing for the MURI meeting.
But I had had you in the cc, right? Maybe you could reply in that thread, to remind Steve.
Yep, I was in the CC. I am happy to do so, but I think Mike’s suggestion to speak to Steve about it next week sounds excellent. If Mike doesn’t get the opportunity to do so, I can send an email.
Re: the second paragraph of #26: exactly, these are precisely the kind of thoughts I have. The only thing is that we have to be very careful if we adopt a file-based system to handle things like concurrent access (locking, etc) correctly.
Hmm, interesting. I think I don’t quite understand the difference between “two webservers” and caching compiled HTML, since even with two webservers there would have to be something in front of both deciding based on the request (view or edit) which webserver to hand it off to (or, based on your terminology, maybe it would be the static server in front deciding to hand the edit queries off to the other in back), which seems basically the same as deciding to pull up a cached version for view requests. But I don’t have a lot of experience with webserver design.
I’m happy to discuss this. What I would like to do here is to present some kind of example of what I have in mind, as I think this will be much more illustrative than if I try to explain it in words. My main challenge is that I would like to make incremental changes rather than an ’all or nothing’ type change, but this is a bit tricky with Instiki, because it is not really modular or API based; everything is kind of bundled together. But I’ll see if I can find a way.
I agree that in a sense there is not an enormous amount of difference between caching and a static site. The main difference is perhaps infrastructurally: in a static site, a page would literally be obtained by a link, just like in a very simple website, there would be no programming layer in between which ’looks for a cache’ and loads it if present. Nor would caches be generated dynamically, etc: when one creates a page, it would just store the page and make the link to it in the appropriate places thereafter. The way a ’frontend/backend’ server division works is that the frontend server would be the one which the user actually interacts with in the browser: when one loads an nLab page, it is the frontend server which will be called (handle the GET request), the response of which will just be HTML. But when one say saves a new page, the link underneath the ’save’ button would point (send a POST request) to the backend server, where the programming and database logic would take place. The backend server, when it is done (or before it is done, depending on the design and case in question) sends a response to the frontend server, which in turn handles it and presents something to the user in the browser. Typically, one would use Javascript to make the call from the frontend to the backend (or usually, some framework on top of Javascript), but this might not be necessary in our case. Anyhow, as I say, there are lots of possibilities here, but probably it’s most helpful if I give an example.
Urs 5, I have just a comment of taste.
Good idea, Richard! I see that with your help things are becoming more professional.
I second Urs’s delight with general quality of Richard’s volunteering and am pleased with his continuous encouragement of all good work at this site. Though Richard was undoubtfully pleased with well intended compliment, I find the ultracapitalistic ideology of “professionalism” put as an ideal discouraging for a volunteer community. I’d rather stay enthusiastic than becoming professional. People live and enjoy human action and creativity, the morality of “hard worker” is industrialized morality which has not flourished before the raise of industrial society. Professionalism includes also expertise, which is good, but also being normalized in all possible manners, including standardized language, dresscode and many other codes, timing around the clock, wish for getting to higher position, following the hierarchy and the self image as being fixed to a given vocation, expected aspiration or task spectrum. Good people who enjoyed spiritual life in recluse for example are much appreciated in previous societies, while the consumer society talks quasinotions of “laisies” and “loosers” and puts to others how hard working and normalized they should be. Of course, Urs should not take the offense of my reminder of how neat and spontaneous community the Lab is (thanks primarily to Urs’s enthusiasm),
Thanks for thoughts, Zoran. I like your point of view! I just wanted to say that I hope that I do not come across as representative of the kind of professionalism you describe; I think I am in fact much closer to ’spiritual life in recluse’ :-). If I was closer to being a representative of professionalism of the kind you describe, my ’career’ within academia would probably have been different :-).
I also wanted to emphasise something I have written elsewhere: that I have not really done much. Adeel is the person who has really done a lot of great work over the past few years under the hood; I am only doing small tweaks on top of the more fundamental work that he has done. My thanks to Urs was more of an appreciation of his kind gesture in offering encouragement than feeling that I had earned the compliment :-).
At the moment the Lab is not responding. Maybe that’s a belated effect of that scheduled downtime, but since the Forum here is active, maybe it’s something different.
It’s working for me right now.
21 Surely, I did not doubt that you are primarily an enthusiast (and visibly a modest person), nor am forgetting that Adeel quietly raised the whole system to another level when he took charge of the maintenance and problem solving.
22, 23 Lab is working for me now.
Thanks for your further words, Zoran!
Thanks very much for mentioning the downtime, Urs, please continue to do so. I see an issue that lasted about 11 minutes in the nginx logs, between 14:44 and 14:55 GMT. That would seem to be about the time of your message. Does that interval sound correct?
The first error message concerns geometry of physics – a first idea of quantum field theory. Now, as we know, this page is very large. In a way it is good that the error concerned this page, as it suggests that the problem is fixable.
It was not only this page which was affected, indeed many seem to have been affected during the downtime period. Unfortunately, our logs and metrics are very limited at the moment, so it is difficult for me to know for sure what the problem was. But since all of the pages were affected, something probably got overloaded. It could be the database or Instiki. It is difficult to imagine how Instiki could be overloaded (although if its thread use is not handled correctly, it could get overloaded by having too many open threads with a page that takes a long time to load that is requested a number of times), but the database is generally pretty sharp, even when one queries across all of it, I will try to debug further.
Hmm, one thing is noticeable is that there were a very large number of requests to load the page mentioned in #25 around the time of the problem. A lot of these seem to have come from sites crawling the web, rather than humans, and they seem to fetch all revisions and all links within the page, which exacerbates the problem.
One thing I can do is to try to stress test a bit, to see if I can reproduce the problem. Also of course I can try to speed up the page :-). Do not have time just now, but maybe a bit later this evening.
Interesting – our robots.txt is supposed to forbid bots from loading past revisions, but of course not all bots obey that. I don’t suppose you can tell what kind of bots they are?
I woudn’t claim to be an expert on this, but here’s a sample list of entries (not consecutive!) from the nginx logs.
180.76.15.25 - - [12/Mar/2018:12:41:53 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/144 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.7 - - [12/Mar/2018:12:58:17 -0400] "GET /nlab/revision/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/70 HTTP/1.1" 200 1359548 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.145 - - [12/Mar/2018:13:05:09 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/88 HTTP/1.1" 200 605887 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.138 - - [12/Mar/2018:13:05:54 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/65 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.28 - - [12/Mar/2018:13:12:48 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/160 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.144 - - [12/Mar/2018:13:15:17 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/89 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.161 - - [12/Mar/2018:13:15:52 -0400] "GET /nlab/revision/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/43 HTTP/1.1" 200 982716 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.134 - - [12/Mar/2018:13:15:54 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/135 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
One can see some 499 status codes in there, which probably means that the client aborted (because it was taking too long). More significantly, one can see different revision numbers here.
In relation to Dmitri’s question about traffic levels, one can also see here that there are a fair few bytes being sent around here just for this one page in the space of a few minutes.
There are several other sites as well, not only this one; I wouldn’t really know a systematic way to identify them. Any suggestions welcome!
Some of the pages at my nLab still have message about the downtime on last weekend on the top, e.g. https://ncatlab.org/zoranskoda/show/permutacija
I just tried to edit one of my pages, my writings (zoranskoda), from my Zadar work IP 161.53.28.4 and got the message “Your edit was blocked by spam filtering”. I tried to correct the name of the link to the arXiv from version to version. It is not the change of URL, but of the name how the URL is to be presented to Lab viewers. In the same time, I could create a new test page without a problem.
Some of the pages at my nLab still have message about the downtime on last weekend on the top
My apologies! For the removal to become visible, it is necessary for me to remove the cache. I did this only for the main nLab, forgetting that it would also affect personal webs. I have now removed the entire cache, so the messages should now be gone.
I just tried to edit one of my pages, my writings (zoranskoda), from my Zadar work IP 161.53.28.4 and got the message “Your edit was blocked by spam filtering”. I tried to correct the name of the link to the arXiv from version to version. It is not the change of URL, but of the name how the URL is to be presented to n nLab viewers. In the same time, I could create a new test page without a problem.
I believe that I know what the problem is here, and I believe the spam filter is ’working correctly’: unfortunately one of your arXiv numbers matches one of the additions to the spam filter that I made to get around the problem described here. But to be sure, could you let me know exactly which link you were trying to change? Also, when you say that you could a test page without problem, did the page include the same text as in the link you were trying to change (I assume not)?
What I will do if the problem is what I think it is is to modify the regex to allow the specific link you have in mind. For the moment, I would rather keep the spam filter as tight as possible and make exceptions than weaken it, because the problematic user has been quite persistent in trying to get around the filter.
Hi Zoran, the spam filter should now be fixed, I believe. See here. Apologies for the inconvenience.
Edit: actually, not fully fixed yet I believe. No time just now, will complete the fix later.
Edit 2: now completely fixed (for this page) I hope. Please test.
Yes, Richard, after your skillful and thoughtful handling it works fine now and I could change the things on the page as I wanted. You are right, the test page had other text (I just checked at the time that I did not have a worse kind of edit ban). Thanks a lot, and apology that I did not attend your messages, being busy last few days.
You’re welcome, glad it’s working now!
More downtime scheduled to start very shortly, 6pm GMT today (22nd) to 0am GMT tomorrow (23rd). Not much point in putting a notice out now. Also the downtime does not always seem to occur when it is supposed to.
Edit: actually, I got the times wrong, it is 03:00 GMT to 09:00 GMT tomorrow (23rd). I have now added a notice to the nLab about it.
Maintenance seemed to occur at the scheduled times, and now seems to be done, so I have removed the notice (and cleared the cache).
Coming back to the question of the server funded by the HoTT grant – I did find out about this, but then neglected to share the answer here, sorry (and thanks Richard for reminding me in the other thread). The short answer is that the physical server was purchased with the grant funds, which means that it doesn’t depend on continued grant funding for its continued existence, so it will not evaporate when the grant ends. However, a physical server has a finite lifetime, and when it reaches the end of that lifetime there are at present no funds to replace it. So we should be looking into funding and cloud solutions, albeit not with excessive urgency.
No problem, thanks very much for looking into it and for letting us know! Also, as far as I understand, we are a bit dependent on the goodwill of Joseph Ramsey at CMU to host the machine in his office and take care of anything that physically needs doing; it is very kind of him, but in the long term we should probably look for something a bit more robust. It will take some time, I imagine, to go through a grant process, so I definitely think it is good to begin with this.
Yes, that’s right too.
1 to 38 of 38