Want to take part in these discussions? Sign in if you have an account, or apply for one below
Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.
I guess that things that require Instiki to go through every page are expensive in terms of server load. I have a suspicion that these will go away (or at least, become a little less apparent) when we move to a new server since then we can migrate to MySQL as well which will optimise a few things.
In the mean time, it's worth doing a little experimenting to see if there are ways around this problem without losing too much functionality. Using 'latest changes' instead of 'recently revised' is an important one, but obviously not one that the lab elves can use.
I wonder if using the RSS is cheaper than the recently revised page? It might be worth running a test or two to find out.
Also, rather than basing this on "how long I think it took the page to load" we should take a look at the production log where it actually logs how long Instiki took to generate the page.
Migrate to MySQL?
As I understand it, the n-lab is currently using the built-in sqlite3 database. It can be configured to use a MySQL database on a MySQL server. This is better for a larger installation, which the n-lab has now become. We could do that on the current host, but although that would make things better it's unlikely to solve everything.
The point is that there are things that we could try with the current host to get the n-lab running more smoothly. However, it is not guaranteed that these will solve all of the problems and also they will be tricky to do due to not have a decent PTY when you log in. Since we have no particular loyalty to the current hosts, shifting hosts will hopefully solve any issues created by the current ones, whilst making it simpler to implement the alterations needed to migrate to MySQL.
This doesn't somehow make it that we can make a MySQL process our one persistent process (available at NFSN for $0.01 per day extra) and have it call everything else, including Rails and Instiki, as needed … does it?
Sadly, no. It's the other way around. The persistent process needs to be persistent because it's waiting for stuff from outside. It then forwards the request to the MySQL server (well, it does a bit more than just forwards it. It works out what the user meant and then translates that into a MySQL request. It then translates the results of that request into something the user would like to see.). So we need both the MySQL server and Instiki to be persistent processes.
Even were Instiki behind an apache proxy, it would still need to be persistent. I'm guessing a little here, but I would imagine that not making it persistent would be a huge slow-down as then it would need to restart itself every time it was called. By being persistent, it saves that reload time.
Darn!
I thought not, but I had to ask.
My guess, too, is that calling "recently revised" makes the server search through the content of every single page (and like of every single pdf file or otherwise that is stored on the machine) and check if there was an update. (It shouldn't be that way, should it: editing a page should simply set a flag in some file, so that calling "recently revised" should be lightning quick. But apparentlly it's not that way).
But do we want to spend effort now on fixing the nLab on its current A2 hosting server? I thought priority now is to migrate first.
And concerning that: I am back online now and have more time. I will take instructions for how I can help with migrating.
I tried to put this on the n-cafe in response to Zoran's comment about ruby, but I think my connection timed out before it got through.
We're shifting hosts not to make the n-lab run faster. We're shifting hosts to make it easier to make the n-lab run faster. Some of the changes will be implemented directly as we shift which will (hopefully) make it look like we were doing the first, but actually it's more that the current hosts are really irritating - this PTY stuff (wherein you don't get a proper login shell) is extremely irritating as it makes restarting difficult and it makes backing up the database tricky. That's reason enough to shift alone, in my extremely humble opinion.
I've blocked "Recently Revised" via the web server on ncatlab. At the moment, it blocks all requests to it, so that blocks it for all 'webs'. It may be possible that only blocking it for the nlab will have the same effect, but I'm not sure since the database is the same for all of them.
Of course, this problem should softly and silently vanish away when we get the new host sorted out....
Great. How did you do that?
I didn't think that we can easily do this, otherwise I would have suggested this long ago.
I put in a block in the web server which blocks 'recently_revised'. Jacques did the same for the 'export html'. That way, the request for 'recently revised' never gets as far as Instiki. So if someone other than me wants to reverse it, edit the file /etc/lighttpd/lighttpd.conf. I've labeled my change with my name so just search for 'ANDREW STACEY'.
I think it would be possible to turn this off within Instiki, but if we start hacking Instiki ourselves then we have to break the automatic updates.
Very good. Thanks for the explanation.
I am very happy with this. The nLab is actually "lighning quick" now, consistently, at least compared to its usual behaviour.
Calling Recently Revised for the other webs doesn't cause any problems; I've been doing it still.
On the other hand, I looking at the atom feed is perfectly sufficient for those webs, and that still works.
if we start hacking Instiki ourselves then we have to break the automatic updates
The last time that Instiki was updated, Jacques said that he couldn't get into the server to upgrade our copy. So what's automatic? And did we ever upgrade?
As I understand it (I'd have to log in and check), the n-lab "pulls" the latest updates from Jacques' source each night. Thus if there is an update available, then it gets updated. I don't know how this correlates with your statement, when did Jacques say that?
I'm not sure I understand your prior comment. Are you saying that 'recently revised' does work for the other webs or doesn't? Your first statement seems to say that it does, but your second implies that it doesn't. I would expect that it doesn't.
Sorry, my first comment was saying that calling Recently Revised for the other webs, when it was possible to do so, did not cause the site to slow down.
<div>
<blockquote>
Calling Recently Revised for the other webs doesn't cause any problems; I've been doing it still.
</blockquote>
<p>Good to know. Since the personal webs are orders of magnitudes smaller than the main one, that's consistent with the idea that the problem of Recently Revised is that it goes through each and every entry in an inefficient way each time one calls it.</p>
<p>It would seem to me that there must be an easy fix for this problem: each page should set a flag in some file when it gets revised. Recently Revised should just display that file, essentially, instead of trying to deduce its content each time.</p>
</div>
The spam filtering is done by a list of keywords; it can be pretty bad sometimes. It has nothing to do with your cookies or your IP address.
Frankly, it would be nice if the spam filtering would turn off after a while if the software can recognise one, as with cookies.
It's objecting to ‘Gucci’. I tricked it into accepting it by using ‘Gucci’ instead.
And you wouldn't believe what I had to do to get this Forum software to print that! (It kept refusing to trust me that I really meant what I said.) Human software programmers can be incredibly stupid sometimes; here we see two cases in point. (For the record, I'm pretty sure that Andrew did not program what I'm complaining about; and if he did, then it's not very stupid of him after all, since his hacks have only very little testing.)
In response to Zoran:
Another question: which LaTeX software is used by scholarpedia ? In the rendering I see LaTeX formulas between the normal text and then from top to the bottom of the page in few seconds browser gradually replaces LaTeX parts by the actual formula rendering (I do not know if this graduality is visible on faster systems than my laptop): http://www.scholarpedia.org/article/Calogero-Moser_system
I'm not sure what system it uses, but the output is pictures, not mathml. That makes it Not Good. The fact that you can see the replacement suggests also that it doesn't cache the pictures, which is also Not Good.
Toby:
Human software programmers can be incredibly stupid sometimes; here we see two cases in point.
As much as I am a 'human software programmer' then I agree completely. However, I would class myself as a mere hacker and we're not stupid, just lazy!
1 to 21 of 21