Neverside

Hello Dave,

Glad to see things are still moving along, would be nice if things got active again. If I was you though I would not get caught in an endless development cycle whilst your user base continues to shrivel. I would 80% manage neverside and 20% manage your new design and ideas myself. You had a HUGE user base on tutorial forums, try to re-acquire that user base. Send out a newsletter that neverside exists still to all those emails, many of them probably use the same emails.

Just saying I would like to see the forums really active again and I know many people still vividly remember neverside, tough thing is it's becoming just a memory. I show up once every week or two to get a status and thats about it sadly.

I mean, fix the photo galleries at least ya know? I would like everything to succeed so much I have offered my time to you for free before and the offer is still out their. Last I heard their was a jittery mailing list but it has disappeared.

I want somewhere to post stuff about programming and help new comers like I was helped damn it!

-Chris

___________________

I code alot.

Hey guys -

Wanted to update you and let you know I've finally got just about everything successfully imported into Drupal, thanks very much in part to help from Jon (sparky) and Chris (erod) who are SQL geniuses This import is of course just a test, getting all the queries written out to use when it comes time to do the final import.

I discovered a roadblock the other day I wasn't expecting... NS's database is in the latin1_bin character set. Importing that into my utf_general_ci tables for Drupal produced over 600,000 mysql errors haha. A lot of search-and-replace for nasty quotation characters messing up my CSV helped, and then ran iconv -f ISO-8859-1 -t UTF-8 on my export files, a few other steps, and all the data was clean

Users
All imported (including passwords, so no reset needed). Still will import additional profile fields later.

Forum categories
yep.

Forum threads
yep.

Thread replies
yep (noticed our bbcode may not be fully compatible, will need to write some search and replace queries to bring them up to spec). There's one last tricky bit regarding making posts compatible with the threaded-comments feature, hopefully will be able to get some smarter minds than mine on that haha.

Attachments
working on that now.

Studio
will be selecting the best by hand and adding them.

PMs
not sure I'll be able to, but will try (may wait on this for later)

News
not bothering.

Anyhow, just keepin ya posted. I'm so glad this worked out. It was harder/more complicated than I expected.
Talk to you soon.

___________________

-- Dave
Neverside Admin

absolutecross.com

Thats pretty awesome to hear... hopefully all will go well.

___________________

Holy ****, its Dean fucking Venture!

___________________

It's said EE-SAW-MOO


I'll speak slowly if your on something, and faster if your not.

*pees pants*

___________________

Hmmm you might wanna get cleaned up before going out in public Chris

___________________

-- Dave
Neverside Admin

absolutecross.com

// Edit: No delete option?

Last edited by Mundo, June 4th, 2008 10:39 PM (Edited 1 times)

Coming along...

It's funny how complex this is turning out to be haha (I had always been planning on using the converter Jeremie was going to make). But it's all good - I'm becoming quite good at writing SQL queries now haha. The queries (plus notes on what they do) already fill 20 pages haha. I'm still no good writing PHP, but getting better at understanding PHP at least. Learn by doing eh? This has also been helpful in me learning about both NS's database structure as well as Drupal's.

I've started on the process for transferring Private Messages, and it looks like it will be easier than expected. The Drupal PM module is quite good also - looking forward to it I submitted one feature request to the developers tonight which I hope they'll consider adding (the PM system allows users to block other certain users from sending them messages - however it doesn't include a check to prevent users from blocking the admin and other staff). **Edit: good news, my feature request has been accepted for the Drupal 6 version of the module which is currently being developed. It's cool, the module is being worked on in part by someone on the development team for fastcompany.com (recently switched to Drupal)

I also believe I've got a good way of being more selective than I thought I'd be able to be in choosing what forum threads to bring along to the new site. I can, for instance skip on bringing the Trash forum, as well as certain other archived forums no longer in use that contain as much as tens of thousands of posts (which would just be needless cruft on the new site).

Polls won't translate quite perfectly, but I will be ensuring that at least all old polls are correctly archived and included with their respective threads (just won't be able to continue voting on them... no biggie).

Been working on solving the user closing their own account issue. There's a simple module for this in Drupal, but I've devised some plans to expand it further to ensure user authentication as well as prevent accidental deletions. I want to have this working perfectly by the time I eventually invite old members back, as I'm sure there will be a slew of users desiring the removal of their account, and there's no way I can do that by hand.

Having actually imported all the content, I've been able to pinpoint 2 queries on the Drupal core forum that perform slowly on a site as big as this. I'm looking into how to deal with it. Jeremie brilliantly stored "all" possible stats about "everything" in NS, so for NS it is light speed to look up any kind of statistic/postcount instantly instead of trying to count anything in real time. Looking to add more of that to Drupal's forum or at least to the Advanced Forum contrib module (making feature requests).

I've also been working on the attachments issue, and have been considering whether I really want to bring the attachments along after all. Bringing them is more complicated than I expected due to how they were processed and stored for NS (when uploaded, the files themselves are renamed to a sequential name/number and stripped of their file extensions... the real filename is stored in the database and PHP reassembles the files when a download request is made). They are also around 1.5GB in total, which is some cruft I'd prefer to leave out and keep the new site easier to backup and copy around while it's in its infant stages. If anyone cares to give their opinion on this issue feel free (and if your vote is to keep them, please help me write the PHP script to pull it off *wink*). I thought as a means of making archived attachments available to those few who might ever want them, I could put them in a static directory with an HTML index page of them that includes the thread and post ID (so easy to look up), and list this page in the site FAQ.

Also, I'd like to hear if anyone has any thoughts on whether we should continue to use BB code, or switch to basic (restricted) HTML. In either case there will be buttons for dynamically inserting the available tags. My personal vote is for HTML, as if you can type BB code tags you can type HTML tags (they're usually the same for the most part, with just [] instead of <>), just about everyone on a site like ours knows basic HTML, and for those who do not or prefer quick and easy buttons, the buttons do the work. Also, HTML is a standard whereas there are many different flavors of BB code from site to site throughout the internet, many subtly different enough to mess up tags you thought were fine (e.g. without a reference even "I" don't know for sure the BB code tags to use here on NS to do things I know easily how to do in HTML). Lastly, Drupal's input filters are capable of handling restricted HTML, so using BB code for security reasons is not an issue for us.

Here's an example of a possible editor:
http://ufku.com/drupal/bueditor/demo
Link to full info: http://drupal.org/project/bueditor

- The default buttons are bad, we'll have better ones.
- Note the way cool instant "in place" preview button.
- It's possible to add additional buttons for other tags, and even make certain tags available to moderators that regular members don't get access to, even by typing them.
- Even if we go with BB code, we can still use this editor as well.

Anyhow, just keeping you up to date.

___________________

-- Dave
Neverside Admin

absolutecross.com

Last edited by AbsoluteCross, June 7th, 2008 09:07 PM (Edited 4 times)

If anyone would like to help me with a tricky SQL query...

I'm hoping to get some help in possibly (if it works) rewriting one of the queries in the core Drupal forum module to make it more efficient if possible. It looks like they're going to fix this, but not as soon as I'd like haha. Tweaking this query won't solve the performance issue, but might help a little (with my current caching setup, the only delay is after a new post is made and the query runs again to update the forum listing). I haven't fully explored the plethora of other caching possibilities for Drupal, which should help as well (I'm sure it's doable, just haven't figured it out yet - e.g Gallery's forum is about our size and runs Drupal). Ideally this query should be nuked and replaced with a system more like NS (pre-storing all needed counts and "latest" info for every forum category... though evidently there's an access control issue in the way of it working as simply as that at the moment).

I'm not sure it will be possible for anyone to help by just looking at this without the rest of the code and/or an installed copy of Drupal (and understanding how Drupal works), but if the comment text gives you any hints on what needs to be improved in it, or you see the inefficiency and know how to state the query better (don't have to understand the whole functionality) then awesome
If you'd like, here is a download of the forum.module (or you can just get the whole thing to look at... forum is in the modules directory).

Here's the query, including the comment that precedes it in the code (I've reformatted it a bit for clarity):

PHP:

<?php

    // This query does not use full ANSI syntax since MySQL 3.x does not support
    // table1 INNER JOIN table2 INNER JOIN table3 ON table2_criteria ON table3_criteria
    // used to join node_comment_statistics to users.
    $sql = "

SELECT
    ncs.last_comment_timestamp,
    IF (ncs.last_comment_uid != 0, u2.name, ncs.last_comment_name) AS last_comment_name,
    ncs.last_comment_uid
FROM {node} n
INNER JOIN {users} u1
    ON n.uid = u1.uid
INNER JOIN {term_node} tn
    ON n.nid = tn.nid
INNER JOIN {node_comment_statistics} ncs
    ON n.nid = ncs.nid
INNER JOIN {users} u2
    ON ncs.last_comment_uid=u2.uid
WHERE n.status = 1
AND n.type='forum'
AND tn.tid = %d
ORDER BY ncs.last_comment_timestamp DESC";

?>

Note the {table} brackets are for table prefixing in Drupal (as I understand). The query is the same between Drupal 5 and 6. The output of this query is simply the "Last post" column (e.g. see here).

Thanks guys for your help. If this one's too crazy, that's ok - just thought I'd ask

___________________

-- Dave
Neverside Admin

absolutecross.com

My thoughts on the existing attachments: Don't even worry about it. If someone actually comes to the forums just to check out an old post and can't see the attachments, it's really no biggy. It just means they'll have to register and repost the topic. I suppose this might only apply to one scenario, but I just really don't think it's a huge issue. If it's 1.5 GB the server doesn't really need, why load it with that junk.

Just my thought.

As for bbcode or html, I don't have any preference. However I do agree with all the points you've made about most of our users understanding at least basic HTML. So go HTML, why not.

Getting exciting again

___________________

It's said EE-SAW-MOO


I'll speak slowly if your on something, and faster if your not.