Yes, that’s right too.
]]>No problem, thanks very much for looking into it and for letting us know! Also, as far as I understand, we are a bit dependent on the goodwill of Joseph Ramsey at CMU to host the machine in his office and take care of anything that physically needs doing; it is very kind of him, but in the long term we should probably look for something a bit more robust. It will take some time, I imagine, to go through a grant process, so I definitely think it is good to begin with this.
]]>Coming back to the question of the server funded by the HoTT grant – I did find out about this, but then neglected to share the answer here, sorry (and thanks Richard for reminding me in the other thread). The short answer is that the physical server was purchased with the grant funds, which means that it doesn’t depend on continued grant funding for its continued existence, so it will not evaporate when the grant ends. However, a physical server has a finite lifetime, and when it reaches the end of that lifetime there are at present no funds to replace it. So we should be looking into funding and cloud solutions, albeit not with excessive urgency.
]]>Maintenance seemed to occur at the scheduled times, and now seems to be done, so I have removed the notice (and cleared the cache).
]]>More downtime scheduled to start very shortly, 6pm GMT today (22nd) to 0am GMT tomorrow (23rd). Not much point in putting a notice out now. Also the downtime does not always seem to occur when it is supposed to.
Edit: actually, I got the times wrong, it is 03:00 GMT to 09:00 GMT tomorrow (23rd). I have now added a notice to the nLab about it.
]]>You’re welcome, glad it’s working now!
]]>Yes, Richard, after your skillful and thoughtful handling it works fine now and I could change the things on the page as I wanted. You are right, the test page had other text (I just checked at the time that I did not have a worse kind of edit ban). Thanks a lot, and apology that I did not attend your messages, being busy last few days.
]]>Hi Zoran, the spam filter should now be fixed, I believe. See here. Apologies for the inconvenience.
Edit: actually, not fully fixed yet I believe. No time just now, will complete the fix later.
Edit 2: now completely fixed (for this page) I hope. Please test.
]]>Some of the pages at my nLab still have message about the downtime on last weekend on the top
My apologies! For the removal to become visible, it is necessary for me to remove the cache. I did this only for the main nLab, forgetting that it would also affect personal webs. I have now removed the entire cache, so the messages should now be gone.
I just tried to edit one of my pages, my writings (zoranskoda), from my Zadar work IP 161.53.28.4 and got the message “Your edit was blocked by spam filtering”. I tried to correct the name of the link to the arXiv from version to version. It is not the change of URL, but of the name how the URL is to be presented to n nLab viewers. In the same time, I could create a new test page without a problem.
I believe that I know what the problem is here, and I believe the spam filter is ’working correctly’: unfortunately one of your arXiv numbers matches one of the additions to the spam filter that I made to get around the problem described here. But to be sure, could you let me know exactly which link you were trying to change? Also, when you say that you could a test page without problem, did the page include the same text as in the link you were trying to change (I assume not)?
What I will do if the problem is what I think it is is to modify the regex to allow the specific link you have in mind. For the moment, I would rather keep the spam filter as tight as possible and make exceptions than weaken it, because the problematic user has been quite persistent in trying to get around the filter.
]]>Some of the pages at my nLab still have message about the downtime on last weekend on the top, e.g. https://ncatlab.org/zoranskoda/show/permutacija
I just tried to edit one of my pages, my writings (zoranskoda), from my Zadar work IP 161.53.28.4 and got the message “Your edit was blocked by spam filtering”. I tried to correct the name of the link to the arXiv from version to version. It is not the change of URL, but of the name how the URL is to be presented to Lab viewers. In the same time, I could create a new test page without a problem.
]]>I woudn’t claim to be an expert on this, but here’s a sample list of entries (not consecutive!) from the nginx logs.
180.76.15.25 - - [12/Mar/2018:12:41:53 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/144 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.7 - - [12/Mar/2018:12:58:17 -0400] "GET /nlab/revision/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/70 HTTP/1.1" 200 1359548 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.145 - - [12/Mar/2018:13:05:09 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/88 HTTP/1.1" 200 605887 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.138 - - [12/Mar/2018:13:05:54 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/65 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.28 - - [12/Mar/2018:13:12:48 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/160 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.144 - - [12/Mar/2018:13:15:17 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/89 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.161 - - [12/Mar/2018:13:15:52 -0400] "GET /nlab/revision/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/43 HTTP/1.1" 200 982716 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.134 - - [12/Mar/2018:13:15:54 -0400] "GET /nlab/revision/diff/geometry+of+physics+--+A+first+idea+of+quantum+field+theory/135 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
One can see some 499 status codes in there, which probably means that the client aborted (because it was taking too long). More significantly, one can see different revision numbers here.
In relation to Dmitri’s question about traffic levels, one can also see here that there are a fair few bytes being sent around here just for this one page in the space of a few minutes.
There are several other sites as well, not only this one; I wouldn’t really know a systematic way to identify them. Any suggestions welcome!
]]>Interesting – our robots.txt is supposed to forbid bots from loading past revisions, but of course not all bots obey that. I don’t suppose you can tell what kind of bots they are?
]]>Hmm, one thing is noticeable is that there were a very large number of requests to load the page mentioned in #25 around the time of the problem. A lot of these seem to have come from sites crawling the web, rather than humans, and they seem to fetch all revisions and all links within the page, which exacerbates the problem.
One thing I can do is to try to stress test a bit, to see if I can reproduce the problem. Also of course I can try to speed up the page :-). Do not have time just now, but maybe a bit later this evening.
]]>Thanks for your further words, Zoran!
Thanks very much for mentioning the downtime, Urs, please continue to do so. I see an issue that lasted about 11 minutes in the nginx logs, between 14:44 and 14:55 GMT. That would seem to be about the time of your message. Does that interval sound correct?
The first error message concerns geometry of physics – a first idea of quantum field theory. Now, as we know, this page is very large. In a way it is good that the error concerned this page, as it suggests that the problem is fixable.
It was not only this page which was affected, indeed many seem to have been affected during the downtime period. Unfortunately, our logs and metrics are very limited at the moment, so it is difficult for me to know for sure what the problem was. But since all of the pages were affected, something probably got overloaded. It could be the database or Instiki. It is difficult to imagine how Instiki could be overloaded (although if its thread use is not handled correctly, it could get overloaded by having too many open threads with a page that takes a long time to load that is requested a number of times), but the database is generally pretty sharp, even when one queries across all of it, I will try to debug further.
]]>21 Surely, I did not doubt that you are primarily an enthusiast (and visibly a modest person), nor am forgetting that Adeel quietly raised the whole system to another level when he took charge of the maintenance and problem solving.
22, 23 Lab is working for me now.
]]>It’s working for me right now.
]]>At the moment the Lab is not responding. Maybe that’s a belated effect of that scheduled downtime, but since the Forum here is active, maybe it’s something different.
]]>Thanks for thoughts, Zoran. I like your point of view! I just wanted to say that I hope that I do not come across as representative of the kind of professionalism you describe; I think I am in fact much closer to ’spiritual life in recluse’ :-). If I was closer to being a representative of professionalism of the kind you describe, my ’career’ within academia would probably have been different :-).
I also wanted to emphasise something I have written elsewhere: that I have not really done much. Adeel is the person who has really done a lot of great work over the past few years under the hood; I am only doing small tweaks on top of the more fundamental work that he has done. My thanks to Urs was more of an appreciation of his kind gesture in offering encouragement than feeling that I had earned the compliment :-).
]]>Urs 5, I have just a comment of taste.
Good idea, Richard! I see that with your help things are becoming more professional.
I second Urs’s delight with general quality of Richard’s volunteering and am pleased with his continuous encouragement of all good work at this site. Though Richard was undoubtfully pleased with well intended compliment, I find the ultracapitalistic ideology of “professionalism” put as an ideal discouraging for a volunteer community. I’d rather stay enthusiastic than becoming professional. People live and enjoy human action and creativity, the morality of “hard worker” is industrialized morality which has not flourished before the raise of industrial society. Professionalism includes also expertise, which is good, but also being normalized in all possible manners, including standardized language, dresscode and many other codes, timing around the clock, wish for getting to higher position, following the hierarchy and the self image as being fixed to a given vocation, expected aspiration or task spectrum. Good people who enjoyed spiritual life in recluse for example are much appreciated in previous societies, while the consumer society talks quasinotions of “laisies” and “loosers” and puts to others how hard working and normalized they should be. Of course, Urs should not take the offense of my reminder of how neat and spontaneous community the Lab is (thanks primarily to Urs’s enthusiasm),
]]>But I had had you in the cc, right? Maybe you could reply in that thread, to remind Steve.
Yep, I was in the CC. I am happy to do so, but I think Mike’s suggestion to speak to Steve about it next week sounds excellent. If Mike doesn’t get the opportunity to do so, I can send an email.
Re: the second paragraph of #26: exactly, these are precisely the kind of thoughts I have. The only thing is that we have to be very careful if we adopt a file-based system to handle things like concurrent access (locking, etc) correctly.
Hmm, interesting. I think I don’t quite understand the difference between “two webservers” and caching compiled HTML, since even with two webservers there would have to be something in front of both deciding based on the request (view or edit) which webserver to hand it off to (or, based on your terminology, maybe it would be the static server in front deciding to hand the edit queries off to the other in back), which seems basically the same as deciding to pull up a cached version for view requests. But I don’t have a lot of experience with webserver design.
I’m happy to discuss this. What I would like to do here is to present some kind of example of what I have in mind, as I think this will be much more illustrative than if I try to explain it in words. My main challenge is that I would like to make incremental changes rather than an ’all or nothing’ type change, but this is a bit tricky with Instiki, because it is not really modular or API based; everything is kind of bundled together. But I’ll see if I can find a way.
I agree that in a sense there is not an enormous amount of difference between caching and a static site. The main difference is perhaps infrastructurally: in a static site, a page would literally be obtained by a link, just like in a very simple website, there would be no programming layer in between which ’looks for a cache’ and loads it if present. Nor would caches be generated dynamically, etc: when one creates a page, it would just store the page and make the link to it in the appropriate places thereafter. The way a ’frontend/backend’ server division works is that the frontend server would be the one which the user actually interacts with in the browser: when one loads an nLab page, it is the frontend server which will be called (handle the GET request), the response of which will just be HTML. But when one say saves a new page, the link underneath the ’save’ button would point (send a POST request) to the backend server, where the programming and database logic would take place. The backend server, when it is done (or before it is done, depending on the design and case in question) sends a response to the frontend server, which in turn handles it and presents something to the user in the browser. Typically, one would use Javascript to make the call from the frontend to the backend (or usually, some framework on top of Javascript), but this might not be necessary in our case. Anyhow, as I say, there are lots of possibilities here, but probably it’s most helpful if I give an example.
]]>To be fair, Steve is probably quite busy right now preparing for the MURI meeting.
]]>Did you hear back from Steve, Urs?
I haven’t. But I had had you in the cc, right? Maybe you could reply in that thread, to remind Steve. I imagine he might at times have other priorities than discussion of nLab matters in his Inbox.
]]>I also agree that we should look for some kind of grant support.
I’ll see Steve next week for the MURI meeting; I’ll try to remember to talk to him about this then.
]]>Hmm, interesting. I think I don’t quite understand the difference between “two webservers” and caching compiled HTML, since even with two webservers there would have to be something in front of both deciding based on the request (view or edit) which webserver to hand it off to (or, based on your terminology, maybe it would be the static server in front deciding to hand the edit queries off to the other in back), which seems basically the same as deciding to pull up a cached version for view requests. But I don’t have a lot of experience with webserver design.
However, I am in general very much in favor of storing web page data in filesystems. For one thing, it makes them much easier to manipulate, transfer, and back up using standard command-line tools, and they can also be versioned using a standard vcs like git. I love SQL databases, but I think they are best suited to storing highly structured and interconnected data (especially where each datum is not particularly large) of which one wants to ask complicated questions; for storing an essentially flat collection of large chunks of text I prefer filesystems.
]]>