Want to take part in these discussions? Sign in if you have an account, or apply for one below
Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.
Following on from Urs' announcement, here's where to record anything regarding the migrated nlab.
As was noticed on the blog, currently the migrated site does not respond. Hopefully this doesn't mean that the migration alone doesn't solve the responsiveness problem.
If it does, would you know a way out?
I keep getting smoke at http://nlab.mathforge.org/nlab/show/diff/Sandbox.
Also check out http://nlab.mathforge.org/nlab/show/exercise+in+groupoidification+-+the+path+integral, where Urs put in a picture with raw HTML tags; it looks fine at http://ncatlab.org/nlab/show/exercise+in+groupoidification+-+the+path+integral.
Also compare http://ncatlab.org/nlab/new/3%26%238260%3B2-colimit, http://ncatlab.org/nlab/new/3%26%238260%3B2-colimit, http://ncatlab.org/nlab/new/3%2F2-colimit, and http://nlab.mathforge.org/nlab/new/3%2F2-colimit; the last of these does not work (404).
Hmm.
Regarding your first set, this is not a problem with the migration per se. The problem code, the raw diagram, is the same in both sets. Something deep within the code is parsing them differently. This may be due to different versions of ruby on the two systems. The logs are complaining about &
s in the code.
This might pertain to your second as well. Again, this is a code issue rather than a migration one. For the first two then I see no difference - what am I supposed to see? Ah, I see why I see no difference! You've put the same link twice. Nope, I still see no difference.
For the second, again it's in the code. What may be happening is that the new way of serving the pages adds another layer to the whole process. This might strip off one level of encoding, resulting in the %2F getting converted to a / before it ought to.
Maybe my dire warnings of putting / in page names were not so unrealistic after all!
But both need investigating. Are they killer problems, though? It does seem to be something about the actual version of ruby used rather than within Instiki which makes it harder to fix, and questionable as to whether they should be fixed or whether we should learn the newer way of doing things.
Right, I've tracked down the 3/2-colimit issue. That's a bug in one of apache or passenger, depending on how you read the details here:
http://code.google.com/p/phusion-passenger/issues/detail?id=113
Seems as though the fix suggested there has made it in to the main code since putting 'AllowEncodedSlashes on' in the config file made it work. I should probably read up on that to be sure that I'm happy with us having that directive ...
I don't know if these are killers or what, but they're differences between the two sites. Why is it that they work on the old site but don't work on the new site? If we know why, and we're OK with it, then that's one thing; but it seems to me that we need to know.
And as for the link that Urs put in exercise in groupoidification - the path integral, well, we haven't used that sort of thing much, but I would sure like to use it more; it's extremly convenient. So we really should have some way to make it work.
Sorry for repeating the link in (4); of course the second one should have been http://nlab.mathforge.org/nlab/new/3%26%238260%3B2-colimit. The point (which I think you got) is that the first three work, but the last one doesn't (or didn't). (They're all links from the Sandbox, by the way.)
Okay, I was expecting to see a difference between the first two links so when I got the same on both then I wasn't sure what your point was.
I agree that the link Urs put in should work, but I disagree that it should be used more. I'd much rather go down the SVG path. If for no other reason than it makes the n-lab self-contained. What happens to pictures like this when someone downloads a copy of the n-lab? Unless they're careful, they'll just get a blank box.
But nonetheless, it's important to know why it doesn't work. I'll investigate.
I would like to use phpLaTeX, but since codecogs actually works now …. You're right about being self-contained, though, which is easy enough to arrange by storing a copy of the picture locally.
The problem with the codecogs syntax is the "non-safe" characters. It appears to be an incompatibility between markdown (aka maruku) and REXML. What I think happens is that maruku ships the page off to REXML to parse but REXML now wants its input in strictest safe mode. Whether or not it's possible or even desirable to patch either of these is moot, and I'm not even sure if I'm reading the error files correctly.
The quick fix is to encode all &
s. Annoying, but if one uses as decent editor (say, via the "It's all text" plugin) then easy enough. Here's an example.
Even if people are going to use services like this, it'd be better to download the picture from codecogs and upload it again to the nLab, as you say.
I'm still confused as to why the two sites behave differently: the bad HTML works at ncatlab.org
but not at nlab.mathforge.org
.
You're quite right that the HTML that Urs put in (and that I left uncorrected) is improper. Since this was simply the HTML that codecogs offered the user to paste into a web page, I'd be inclined to report it to them as a bug. But in fact, it looks like they've already fixed that bug, as well as a related one that broke the link back to their site. So this problem should never happen again.
All the same, I still think that we should figure out why the two sites behave differently.
Incidentally, the best URIs to compare their behaviour now are probably http://nlab.mathforge.org/nlab/revision/exercise+in+groupoidification+-+the+path+integral/11 and http://ncatlab.org/nlab/revision/exercise+in+groupoidification+-+the+path+integral/11. I fixed the production version.
Incidentally, while follow a bunch of links at once on nlab.mathforge.org
, I just got two ‘Application error \ Rails application failed to start properly’ errors in a row, at 15:52 UTC, from http://nlab.mathforge.org/nlab/revision/diff/latest+changes/151 and … erm … another one. Is this the sort of reporting that helps? (Both of these were fixed by hitting reload, and this is better than long waits and timing out at ncatlab.org
.)
Yes, that's the sort of reporting that helps. I'll take a look at the logs.
The two sites behave differently because they have slightly different versions of ruby underneath. The old site uses ordinary ruby 1.8. The new site uses ruby enterprise edition. That's a version of ruby 1.8 that's been modified to use less memory (hence why we're using it). It was recommended by someone at Rails Playground. Since the issue seems to be in the libraries, rather than Instiki itself, that could well be the fault.
So the old version does some sort of HTML safety encoding? (Damn, I can't think of the right term for ‘safety encoding’ here; hopefully, you know what I mean.)
I suspect it's slightly more complicated in that's is most likely that some routine has slightly changed its expected input as to whether it was encoded or not and so the implication in "the old version does ... safety encoding" is probably wrong - the intention wasn't to make anything particularly safer, just to ensure that (to put it mathematically) the codomain of the first function was the domain of the second, rather than (as is so often the case in these languages) just something that looked like it in the right light.
But I would expect Markdown to correctly escape the URL before passing it off. That will need a little investigating to see how deep and how annoying the change is. However, in thinking about resource allocation (i.e. how much time I have) how highly do you rate it? (You've a much better global view of the 'lab than I do).
I don't think that this change affects anything else currently on the Lab (I've looked at obvious candidates, like other images), so I don't think that it's a high priority.
However, it will probably annoy Zoran whenever he posts a complicated link as <a href>
, so we should track it down eventually. (And by ‘we’, I mean ‹you›, unless you tell me specifically what I could do to help.)
But don't delay the move over this.
More important is that this sort of think seems to break diffs: http://nlab.mathforge.org/nlab/show/diff/Sandbox. That we really should fix.
(I forgot to say that you should find two hits to that in a row in the ‘I smell smoke.’ error logs, just now.)
Okay, so what's happening is that on the old version it changes all bare &
s to &
s in URLs. In the new version it doesn't do this. I think that this is actually a bug in the new version because even if you do something like:
<p markdown="1">
Here's an ampersand: &
</p>
then it ought to parse the &
inside and convert it to a &
but it doesn't.
On the other hand, if the markdown
tag isn't supplied then I'm not sure that it ought to change ampersands. What it ought to do is complain that what you've written isn't valid XHTML and refuse to accept it. It isn't doing that, though, it's complaining and then accepting it. I don't think that either behaviour is correct:
If you put in your own XHTML then it is your responsibility to ensure that it is valid. To ask the filter to interpret it is asking for trouble - the reason you put in XHTML directly is because you aren't happy with what the filter does! So the program should validate it and then accept it only if it is valid, otherwise reject it (with useful error messages, of course).
If you explicitly say that you want the filter to parse the contents, then it should do so in the same way as it does normally. However, this should apply to the contents of the tags, not their attributes. The validity of the extra syntax is still your responsibility.
That's just MHO!
(I've just been trying to use the ![alt text](URL)
syntax to get this particular diagram to work, but I can't figure it out)
Little more testing: the markdown="1"
test fails on both servers but the ampersand-in-url test only fails on the new server. It works on my course installation of Instiki as well so I find it hard to believe that it is something in the Instiki installation. That means that it must be in the version of ruby that's being used. Darn. That makes it harder to track down. Mind you, it also makes it more likely that someone else has spotted it.
Possibly you know about this, but I'm getting ‘Internal Error \ An application error occurred while processing your request.’ consistently (that is, thrice in a row) in http://nlab.mathforge.org/nlab/revision/diff/Sandbox/22. Presumably for the same reason as in my comment #17&18 above, but with a different error message.
Yes, and revision diffs 23–25 also produce this error, with the current diff producing smoke.
That's another reason why the wiki should reject bad XHTML rather than accepting it and wrapping it. I'm extremely reluctant to try it, but I would be willing to bet that if you did the <p markdown="1">&</p>
test on the original lab and then looked at the diff then you'd also see smoke. Although the parser for showing a page can cope with malformed XHTML, the diff routine seems a little less robust and croaks.
Both the original installation and the new one seem to handle that just fine: http://ncatlab.org/tobybartels/show/diff/Sandbox and http://nlab.mathforge.org/tobybartels/show/diff/Sandbox. (There's still a bug for not parsing, but it doesn't crash.)
You're a braver man than I am, Gunga Din!
That's a little odd, I must say. From the error messages it did seem to be the unescaped ampersand that was causing the smoke. It's going to take a little more investigating to figure this out. I think I might add a line to the migration script that escapes ampersands to ensure that any old stuff that relies on them getting escaped goes through okay. Then any new input will trigger the warning and (hopefully) whoever put it in will notice and fix it.
But I think that to figure out what's going on, and fix it, I really need to get some familiarity with ruby to get an idea of how all the pieces fit together. Look out for a "hello world" program any time soon!
Actually, I've got an idea on that ... but it's not to do with migration so I'll start a new discussion ...
The email address in your error message indicates that the problem is first being noticed at your end. Whether or not it's being provoked by the nlab is something that's hard to figure out without more information (time of request, for example).
One thing that might be the reason is that Urs is away at the moment. I'm not sure how widely it's known that the only reason the current nlab works at all is because Urs and Toby keep restarting it. As Urs is offline, the burden of this falls on Toby and he has to sleep sometimes.
Bah, sleep is for the weak!
But Andrew, surely you restart the Lab sometimes too?
I freely confess to my weakness! Three kids have taught me the true value of sleep!
Sometimes I stoop to such menial tasks as restarting the lab, but only when others can't find the lock ...
(I certainly don't restart it at anything like the rate you and Urs do! I think I'll grep through the logs to find out just how often the two of you have to do it.)
Had to do it.
THANKS! Andrew. For the migration!!
Ah, you noticed.
The DNS change is still propagating through the system (it hasn't changed on my machine yet) so it's a bit obvious that you get redirected from ncatlab.org to nlab.mathforge.org. Hopefully that'll be in the system before all the stateside folks wake up.
Please let me know of any glitches! I'm monitoring the error log as well, but that also gets filled with debugging stuff from Instiki (which aren't actual errors just information about what's going on), so it's useful to know when something significant has happened. Remember that a useful bug report includes the time (and timezone!), the link, the ip (or if you don't know that, something that might help me figure out what computer you were on), and a description of what happened.
I have posted an announcement of the migration
http://golem.ph.utexas.edu/category/2009/09/nlab_migration_done.html
Here is a
BUG REPORT
Could it be that the case-sesnitivity of the entry titles has been lost?
I had (maybe a bad idea, but for "historical reason") two different entries on my personal web whose title differed only in spelling
But this link
http://nlab.mathforge.org/schreiber/show/differential+nonabelian+cohomology
takes me to the capitalized-named entry. And I can't access the lower case entry at all.
(Not much harm done, as I have a backup copy of that entry on my other, "private" web).
Sorry: "differed only in capitalization", of course :-)
Another
BUG REPORT
long entry titles have apparently been truncated.
This one here for instance:
<div>
<p>Yet another</p>
<p>BUG REPORT</p>
<p>German time 11:50, I repeatedly get</p>
<blockquote>
Application error Rails application failed to start properly
</blockquote>
<p>While working on my personal web.</p>
<p>And now I get no reaction at all...</p>
</div>
Here is something that is not a bug report:
I have now worked intensively with the new Lab for a longer while and it works great. I am so glad that we finally migrated...
(I had to go and teach for a couple of hours - I was dreading getting back and finding that it had crashed again, but fortunately it seems to be okay.)
The 11:50 I think was due to a rogue process. It seems that if you change the settings and reload apache while instiki is doing something then it might go a little mad. So I should post a note for anyone who has to restart things to leave some time to monitor processes to make sure that they die properly.
Losing capitalisation is irritating. It could be an issue with the database, in which case it'll be extremely annoying to fix but given the other problem then I'll have to do it anyway so may as well fix that. If it's within the ruby code then it could take a bit to track down. However, if it's in the ruby code then I can manually go into the database and rename one of the pages for you. Actually, I can do that anyway. I suspect that there may be a few things that require me to hack the database so I might save them up for a single shot, unless there are urgent ones.
The truncation is almost certainly a database issue. I strongly suspect that it is the same issue as with the stylesheets (see above). MySQL is a bit more rigid in its enforcement of database column types than sqlite3 so whereas sqlite3 says "You want to put 80 characters in a 60 character pot? Fine, go ahead!", MySQL says "Okay, I'll take the first 60 and ditch the rest." When we came across this for the stylesheets, I looked at the values for the other fields but thought they were enough. Little did I reckon with 'Background fields in twisted differential nonabelian cohomology"!
I suspect that it is only the official title that is truncated. Redirects shouldn't be covered by this, so we could easily have a 60 character limit on titles but then put in a more suitable redirect. Jacques will have his own opinion on whether or not page titles should be any length or should have a fixed upper length. A quick search shows that a practical upper limit on URLS is about 2000 characters. Please no-one take this as a challenge! (For one, I haven't extended the limit yet).
We have 14 pages with titles longer than 60 characters (unless anyone's added some since I did the migration). The longest is a whopping 95 characters. I can "up" the limit to something long enough to accommodate these (say, 100) or we can rename them. I think that allowing completely arbitrary lengths is probably a Bad Idea, and I would say that 100 is probably long enough. However, the fact that this is now being enforced should be made plain. It's probably a good idea if I add this to the FAQ or somewhere like that.
Ah, just found a possible for the case sensitivity here. This implies that the comparisons (foreach (page) {is page equal.to page.you.want}) are case insensitive meaning that the lowercase one just happens to be first in the list. But what's most significant about this is that the data itself is case sensitive which is great as it means I don't have to manually fix that (phew!).
The bit I don't understand is that the default character set is latin1_swedish_ci. Swedish??!!??
Thanks, Andrew.
I tried to have a truancated "official" entry title with a longer redirect name. But that didn't work either. (The redirect command had no effect).
Don't worry about renaming my personal entries, I have taken care of that, or will still.
What are these very long titles? Possibly those with multiple occurence of "infinity" in them, right? In that case, since we agreed not to use the single character infinity-symbol instead I would say we shoul "up" the limit to accomodate entries about infinity-categories of infinity-somethings of infinity otherthings, since that may occur again.
The overlong titles are:
I'm surprised that the redirect didn't work. Maybe redirects are stored somewhere as well - makes sense, I suppose. Ah, yes. They are in a table called 'wiki_references' and sure enough, they have a character length of 60. Since the redirect name becomes part of the URL, it makes as much sense to put a limit on them as on page names (i.e. some, but not so it becomes excessively inconvenient).
I'll ask Jacques for a quick opinion on this as well.
I am just this minute getting consistently ‘Application error \ Rails application failed to start properly’ from http://ncatlab.org/nlab/show/reality+check, after 5 or so reloads in a row.
Yeah, I think that it just crashed.
And it's back now.
I just had to restart the web server. It's too late for me to figure out what exactly happened, but I'll try and chase it down in the morning. One 'instiki' process had gone rogue and was eating up all the memory, but why it was doing that I've no idea. Killing it brought down the memory usage, but then the web server didn't like the fact that a process had gone from underneath it so I had to restart that as well.
I've said all along that being on the new server isn't a magic bullet that will instantly speed everything up, but is rather giving us the opportunity to speed everything up, mainly by giving us better control and better access to monitor what's going on. It's not perfect, but we have a better chance of getting it closer now.
Recently Revised is back, and using it doesn't slow everything down to a crawl. That alone makes this worth while to me!
Now if we can just fix the field size restrictions, then I'll be happy.
By the way, I just noticed that the new server is in a different Time Zone??? UTC would be nice. (For the record, it is now 23:17 UTC.)
Looks like I have a few things to do that require mucking about in the database. I'm storing them up since I want to know that I have time to restore stuff if it all goes wrong. I think I know what to do, and that it should be simple to implement, but I want to be sure before I start going in under the hood.
Concerning the truncated entry names (see discussion further above):
I am going through the lab currently changing broken links due to truncation of entry titles so that they will work again, by making them point to the truncated titles.
To everybody: I won't have time and energy right now to do it for all of them (see the list that Andrew gave above). So help is appreciated.
To Andrew: what do you think now is the long-term prospect on this problem? Do you think your fiddling with the database will make the truncated long entry names come back? In that case we need to make sure now that the truncated name remains as a redirect, because, as I said, I started pointing to the truncated names, just so that the material at these entries is not lost to the Lab.
Hopefully my email reached you in time to stop you doing too much that you don't need to do.
If the truncated names are an urgent problem then I will fix the problem now. If not, I'd rather wait until my next clear patch of time so that I can be sure that I do it properly (backing up the database and stuff like that!). Then I can fix all the problems in one go - they all seem to be the same issue.
I've checked with Jacques and he thinks that there's no problem in simply raising the limit a little. Up to 100 characters will fit the current usage. However we both agree that it is silly to have no limit whatsoever. It's bad design to have long page names, and really long page names will break something somewhere. So I figured that 100 would be okay, and then we just announce that these limits will now be actually enforced.
Okay, I just replied by email:
fixing the links to the entries that I did fix was urgent to me, becase these contained material linked to from the blog which i am hoping people will come to from the blog and maybe give me feedback on. So it made me nervous that the links were broken.
I don't have a similar urge with the remaining affected entries. So I'll just wait for you to take action then.
Just let me know when you have taken action if I may need to re-adjust the adjusted links back.
<div>
<p>BUG REPORT</p>
<p>German time 20:17 Sept 2</p>
<blockquote>
Application error:
Rails application failed to start properly
</blockquote>
<p>over what is by now three minutes.</p>
<p>And I have to run and catch my train!! Argh! :-)</p>
</div>
Andrew left me some instructions to restart; I'll go dig those out and see if I can't get it back up. (I gave it a few minutes while answering email, but … nope, it's not coming back.)
OK, I got it back.
Is there any reason to think that Recently Revised is slowing things down now and causing things to go awry? It sure seems fast to me, but maybe that's only because I remember what it used to be like!
In the long run, however, Jon is probably right that it will have to be split up somehow. The same goes for All Pages.
<div>
<p>BUG REPORT</p>
<p>German time 9:12 Sept 3</p>
<blockquote>
Application error:
Rails application failed to start properly
</blockquote>
<p>and not recovering</p>
<p>happened right after submitting an entry. I was in the middle of editing a dozen of entries. Will have to log about it later.</p>
</div>
9:27: there it is again
<div>
<p>well, not quite:</p>
<p>all pages display, but hitting "edit" produces</p>
<blockquote>
access denied
</blockquote>
</div>
now it works again!
but now i need to hop on a train without inet connection...
Looking at the logs, we've hit the limit a few times again. The memory upgrade should partially fix this, and installing the monitoring program should also partially fix this. A true fix would figure out why the odd instiki process is spiralling out of control.
Incidentally, I've ordered the memory upgrade. When they do that, then there will be a downtime of approximately 30-40 minutes. I don't (yet) know when this will happen and I may not get enough notice to warning people beforehand.
They have now done the memory upgrade. Due to the way it had to be done (we had to be shifted from one machine to another as there wasn't room on the old one), we have new ip addresses. That means that it may take a short while for the domainnames to point to the right place. So if you get something funny from ncatlab.org then that'll be why. If you're on a unix machine, you can use the 'host' command to check that what you think is the nlab really is. If you type
# host ncatlab.org
Then you should get back
ncatlab.org has address 68.233.9.66
ncatlab.org mail is handled by 0 mail.ncatlab.org.
If the number is different (the old number was 74.63.3.116), it won't work right. Unfortunately, putting the right number into the browser won't help because of the way that the server is set up (it uses "virtual servers" to decide exactly what service you want - not that there are any other services but I figured I'd build in that capability from the start in case, for example, we want to shift this forum there).
It shouldn't take long for the change to propagate and it should be worth it.
No news is good news, I suppose. There haven't been any more reports of freezes after the memory upgrade so I'm hoping that that means that there haven't been any.
Actually, I can check the logs ... nope, no freezes. Yippee!
Has anyone any slowdowns or any other issues to report?
I'll do a bit of serious editing now. Will get back to you to report my experience.
Great.
Incidentally, given that I can mess around with the actual database, would anyone like me to sort out the Sandbox/Timeline historical saga? I'll have to reinsert the history for both since both are (at times) above the 65535 limit so it wouldn't be hard to sort out the historical record, or is this not considered important.
Incidentally, the pages which have, at sometime in their past, exceeded the length limit are:
(We clearly did a lot over the summer! Well done us)
I'd be in favor of sorting this out. The history information might be of interest at some point. I think it already has been here and there. In particular the timeline entry is likely to invoke the interest of many contributors, so it seems worthwhile to not confuse them by presnting them with a huge list of Sandbox history, if we can avoid it.
Is there any practical implication of the information that the entries you list once were too long? Does this mean we need to fix anything now?
No practical information (hence the word "incidentally"), beyond the fact that they aren't right at the moment so don't mess around with them until I've fixed 'em.
I have now spend some time doing some serious editing on the Lab, and it all worked very fine indeed. All reactions are quick, nothing stalled.
"Recently revised" and "All pages" usually are displayed quickly, sometimes even very quickly, but once in a while they still take a few seconds. I gaather that's whenever the cache for these pages is being rebuild, or something.
I have started implementing the database changes. So far, I have updated the meta information in the database so that:
Page titles and redirects can now be up to 100 characters long. If you try to create a page with a longer title then the page will still be created but with the name truncated to 100 characters. However, the original link to this page will not work, and also, since the first thing that happens when you edit a page is that it is shown back to you, you will get a slightly odd error message (it's logical if you think about it). That's your indication that something went wrong. But the page was still created, so your best strategy is simply to rename it to something more sensible.
The same goes for redirects. Except that you don't get any error message, it just silently truncates the redirection name.
Pages can now be up to, allegedly, 2^32 bytes. That should be enough for anyone.
String comparisons on page names and redirects are now case sensitive. That has little to do with the search function but means that 'A Page' and 'a page' are now viewed correctly as two different pages. It seems necessary to do this on a column-by-column basis so there may be other strange behaviour attributable to the fact that searches are case insensitive.
Some page titles got truncated in the migration due to the change in character length. I have changed all of the ones on the nlab back to their original names. As I did this through the nlab (rather than directly on the database), the truncated name survives as a redirect. Thus any links that anyone changed to point to the truncated name should still work.
Two of the truncated page titles were from the pre-redirect days and were simply "see XYZ". I added the proper page name (the non-truncated one) to the main page as a redirect and added these pages to 'category: delete'.
I have had a go at changing back pages on other webs. I have not touched anything on a "private web". Urs, I hope I haven't broken anything on your (public) web, but you seem to have three pages with the same name ("list of notation and constructions ...") so although I changed one of them to the correct longer name, I wasn't sure what to do with the other two, or even which is the "proper" one.
I have also uploaded the long pages that previously got truncated. Hopefully these are now back how they should be.
That should be enough for anyone.
‘640K should be enough for anybody!’
But should be enough for a good long while. (^_^)
Thanks for all of your great work, Andrew! I've been pretty useless for this whole move, just someone to bounce ideas off of mostly, but you seem to have done a good job in less than perfect circumstances.
I second that!
Urs, I fixed a few more long page titles on your (publicly available) web.
Oh, thanks, Toby. I wasn't actually aware that there was more to fix. I'll have a look.
1 to 76 of 76