New Mars Forums

Official discussion forum of The Mars Society and MarsNews.com

You are not logged in.

Announcement

Announcement: As a reader of NewMars forum, we have opportunities for you to assist with technical discussions in several initiatives underway. NewMars needs volunteers with appropriate education, skills, talent, motivation and generosity of spirit as a highly valued member. Write to newmarsmember * gmail.com to tell us about your ability's to help contribute to NewMars and become a registered member.

#76 2020-12-15 18:51:15

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

As we have closed the forum off to spammers, I wanted to see when the crash happened by looking at the members list ascending

Username   Title       Posts   Registered
target tony  Member    2      2008-05-27 profile? id=3030
Decimator   Member   39     2011-11-20 profile? id=3035

This gap means that we have other gaps such as in the topic id numbers as well...

Offline

#77 2020-12-15 19:59:18

tahanson43206
Moderator
Registered: 2018-04-27
Posts: 8,088

Re: The great crash

For SpaceNut re showing topic...

Thanks for bringing this topic back into view ...

jburk wrote:

As part of modernizing & building out Marspedia.org:

I have restored the New Mars Wiki! - wiki.newmars.com

Right now, creating new user accounts are turned off to prevent spam, but let me know if you want me to set one up for you.

Enjoy!

I finally took the opportunity to visit the Mars Wiki ...

There appears to be a lot there to study.

It might be a place we could consider for "permanent" storage of some of the wisdom that flows through this forum.

(th)

Offline

#78 2020-12-16 16:56:18

kbd512
Administrator
Registered: 2015-01-02
Posts: 4,921

Re: The great crash

SpaceNut or tahanson43206,

What is it that needs to be done to fix all the messed up topics?

Is this a simple matter of writing a JavaScript or PHP script to fix all of them?

Online

#79 2020-12-16 17:16:41

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

the invalid parts that shift plus commands are from the old conversion in the files contained in the mysql data base.
attributes to quoting of posts , color, underlining of text, making text bold, giving the text an italic appearance ect...

post 38/39 give the structure of the command where the (:post_uid##) is the ones that will cause the shifting of the posts in the topic.
post 73 has the topic reference conversion error for links within the forum
post 5 gives the third error type in its post...
post 33 last issue is the sub forums which where created in the data gap that James has recovered
This is current userid name http://newmars.com/forums/profile.php?id=###
but I have seen Pid=## somewhere and not all of the ##' are sequential...
The same sequential ## system for the topics also have gaps in the number as well as the post counters....

since that data is on the server files you need direct access from some one like the webmaster James has...

Offline

#80 2020-12-16 18:29:44

kbd512
Administrator
Registered: 2015-01-02
Posts: 4,921

Re: The great crash

SpaceNut,

Is there any reason you can't go through the posts with a script to correct / replace the offending tags?

Online

#81 2020-12-16 18:42:21

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

Since the only way to fix a post is in edit as the person that wrote it to fix the defects.
If you are the moderator you can also fix others along with your own but you can not fix any admin posts.
It takes being an admin with edit to be able to fix them.

here is a sample quote post from a defective topic with a ( before the color to remove the [ that would normally be there..
In original posts with quotes it will contain the :post_uid stuff when its shifting.

Gennaro wrote:

(color=#000000:post_uid12]I was planning to write a new message to the the thread "Commercial Viability of Space", initiated by me some weeks ago. Prior to start writing, I came upon a few minor details in need of editing in the original post (spelling errors and the like; items I thought were already cleared in fact), but as I pressed the "submit modified" button, most of the messages disappeared.
The symptoms seem identical to what happened to "Who needs a Space Station?", also started by me.

What's going on here? Can it be rescued? I'm very sorry about all this, the mess etc, and additionally so since it was probably one of the better posts I have written on this forum and there were a number of interesting comments I would have liked to pick up from.
sad(/color:post_uid12]

You would need to open every post multiple times to make the edits for all of the defect types that a single post might contain since the number can be different. The color is also something that no longer is needed to make the post distinguishable to others plus color numbers are not correct...

Offline

#82 2020-12-16 18:51:19

kbd512
Administrator
Registered: 2015-01-02
Posts: 4,921

Re: The great crash

SpaceNut,

That metadata must be stored somewhere.  Why are you trying to hand-correct something like that?  Even if we do have to do it from the front end, why can't we use a browser automation tool and a macro script that contains a list of regular expressions to search for and replace within the posts that have been messed up?

Online

#83 2020-12-16 18:54:12

tahanson43206
Moderator
Registered: 2018-04-27
Posts: 8,088

Re: The great crash

For kbd512 re post recovery ....

It is possible you may see a way to automate the repairs SpaceNut has been doing.

In any case, thanks for your interest in helping to bring some of those old topics back to life!

I was taken aback a few minutes ago, to read the post from Clark about the forum actually starting back in 1995 or so!

I thought 20 years of history was impressive!  25 years would be even more so.

In one of the topics SpaceNut recently brought back into view, I read a post from James Burk about the nature of the problem caused by a failed hard drive. As I remember the post, he was able to save the log files from the MySQL database, but could not recover the schema.

It might be possible to analyze the log files and write code to re-generate a database with a structure to match the current one, using the log files as input. That would be some heavy duty coding, but (I'm assuming) not out of the question.

I've been asked to so something similar with another space related web site, and I'm hoping to schedule time to work on that over the winter.

(th)

Offline

#84 2020-12-16 18:59:44

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

Its in the MySQL files which get ported to our display.. we do not have direct access to it....

That said a script must pull up each post to fix the defects...

here is the topic that has the post which I identified
http://newmars.com/forums/viewtopic.php?id=3526

as you see the topic progressively get shifted the further down you go....
highlight a word and use the tab key to move the screen so that the function buttons are visible for editing....

Offline

#85 2020-12-16 19:15:04

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

Here is a copy of a piece of the log file

þbinXCLI   ^   b      5.0.27-log                                        XCLI8
     K kCLI     g               @    `     std! ! ! newmars_forums UPDATE phpbb_sessions SET session_time = 1229734762, session_page = 'viewtopic.php?f=5&t=4529&start=720', session_forum_id = 5
                                WHERE session_id = '61c2c3f08d03f262b2814785b5b4eb4e'mCLI   €   ç                @    `     std! ! ! newmars_forums DELETE FROM phpbb_sessions WHERE session_user_id = 3078mCLI   +                  @    `     std! ! ! newmars_forums INSERT INTO phpbb_sessions  (session_user_id, session_start, session_last_visit, session_time, session_browser, session_forwarded_for, session_ip, session_autologin, session_admin, session_viewonline, session_page, session_forum_id, session_id) VALUES (3078, 1229734765, 1229734765, 1229734765, 'Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)', '', '67.195.37.107', 0, 0, 1, 'viewtopic.php?f=1&t=3803', 1, '09957d99a15325d180f9b0aee5f0f806')mCLI   ˜   ª                @    `     std! ! ! newmars_forums UPDATE phpbb_users
                SET user_lastvisit = 1229734765
                WHERE user_id = 3078nCLI   »   e                @    `     std! ! ! newmars_forums UPDATE phpbb_topics

I was able to pull from it posts activity and ore from them by hand but its time consuming....

Offline

#86 2020-12-16 19:20:35

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

here is a piece that contains a post
had to remove the e in the brackets for quote

WHERE session_id = 'fa14ddb7761b084a891a3369db5210cb'DLI      S*    sÜ     DLI   Ë  .    #            @    `     std! ! ! newmars_forums INSERT INTO phpbb_posts  (forum_id, poster_id, icon_id, poster_ip, post_time, post_approved, enable_bbcode, enable_smilies, enable_magic_url, enable_sig, post_username, post_subject, post_text, post_checksum, post_attachment, bbcode_bitfield, bbcode_uid, post_postcount, post_edit_locked, topic_id) VALUES (13, 6125, 0, '24.17.123.23', 1229734918, 1, 1, 1, 1, 1, '', 'Re: Obama not a US citizen?', '[quot=&quot;bobunf&quot;:1ogajlh8]S_Lajoie wrote, &quot;he had his Hawaiian birth records sealed, so no one could get them. He also had his Kenyan birth records sealed.&quot;\n\nWhat are your sources for these assertions?\n\nBob[/quot:1ogajlh8]\n<!-- m --><a class=\"postlink\" href=\"http://www.worldnetdaily.com/index.php?fa=PAGE.view&amp;pageId=79174\">http://www.worldnetdaily.com/index.php? ... geId=79174</a><!-- m -->', '571034e9d4948f29914b0568217c0781', 0, 'gA==', '1ogajlh8', 1, 0, 6565)DLI   œ   º.    #            @    `     std! ! ! newmars_forums UPDATE phpbb_config

Offline

#87 2020-12-16 22:14:43

kbd512
Administrator
Registered: 2015-01-02
Posts: 4,921

Re: The great crash

SpaceNut,

Here's a regular expression string that can strip quotes:

\[quot.+?\]|\[\/quot.+?\]

Here's what it matches:

[quot=&quot;bobunf&quot;:1ogajlh8]

[/quot:1ogajlh8]

We would obviously perform more exhaustive testing to be sure that what it matches is what we actually want to remove, but it can be done.  If there's a pattern to these offending character strings, then we can probably text match and strip them using a combination of scripts and regular expressions.  Heck, this can even be done from MySQL.  I use regular expressions to text match data stored on various databases at work, every so often.  Even if we have to do it from the front end, there are free tools to automate Firefox, so we can have the macro scan through each page of posts, sequentially, and remove the defunct tags.

Edit:

The link below has the tool I used to test that expression.

https://regex101.com

Last edited by kbd512 (2020-12-16 22:15:42)

Online

#88 2020-12-17 16:52:21

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

Here is a section that has the web page that was done with DRUPAL

@          std! !  drupal UPDATE cache_page SET data = '<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en\" xml:lang=\"en\">\r\n    \r\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\r\n\r\n  <title>FMARS 2009 | Flashline Mars Arctic Research Station</title>\r\n  \r\n  <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n<link rel=\"EditURI\" type=\"application/rsd+xml\" title=\"RSD\" href=\"http://www.fmars2009.org/blogapi/rsd\" />\n\n<link rel=\"alternate\" type=\"application/rss+xml\" title=\"FMARS 2009 RSS\" href=\"http://www.fmars2009.org/rss.xml\" />\n\n<link rel=\"shortcut icon\" href=\"/misc/favicon.ico\" type=\"image/x-icon\" />\n  <link type=\"text/css\" rel=\"stylesheet\" media=\"all\" href=\"/modules/aggregator/aggregator.css?B\" />\n<link type=\"text/css\"

This is the page content

<div class=\"node\" ><div class=\"node-inner\">\r\n\r\n      \r\n    <div class=\"teaser\">\r\n      \r\n      <div class=\"title-wrapper clear-block\">\r\n      \r\n        <div class=\"teaser-title\">\r\n          <div class=\"wrapper\">\r\n            <h2>\r\n              <a href=\"/content/flashline-mars-arctic-research-station-2009\">Flashline Mars Arctic Research Station 2009</a>\r\n            </h2>\r\n           </div>  \r\n        </div>\r\n        \r\n        <div class=\"teaser-content\">\r\n          <div class=\"wrapper\">\r\n            <h3>Project Overview</h3>\n<p>In order to help develop key knowledge needed to prepare for human Mars exploration, and to inspire the public by making real the vision of human exploration of Mars, the Mars Society maintains multiple habitats around the world, where simulation missions can take place in conditions as similar as possible to actual Mars missions. The Flashline Mars Arctic Research Station, brought online in 2000, was the first station created in support of this goal.</p>\n<h3>

This is also in the log files which were all that was in the bin after the crash.

At that time it was 90 gig of log files

Offline

#89 2020-12-17 19:19:58

kbd512
Administrator
Registered: 2015-01-02
Posts: 4,921

Re: The great crash

SpaceNut,

So, what do you need done with all of those tags?  Do they need to be removed?  If so, which ones?

Online

#90 2020-12-17 19:35:19

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

The conversion issue topics are the example in post 84 for the tags which still remain here in the posts from the beginning to about 2005 entries from what I remember.

I had been doing the fixes as I found them but one could do so much more trolling for the identifiers that cause it with a script.

Try the script repair on that first topic if it works keep going to others....

The drupal is no longer an issue as they have there own server now for the FMAR website and are no longer sharing...

Offline

#91 2020-12-17 19:38:19

kbd512
Administrator
Registered: 2015-01-02
Posts: 4,921

Re: The great crash

SpaceNut,

If we want to convert specific tags to retain formatting, then we can do a search and replace with the regular expressions.

Online

#92 2020-12-17 19:51:55

SpaceNut
Administrator
From: New Hampshire
Registered: 2004-07-22
Posts: 23,903

Re: The great crash

this topic has the on close inspection of posts the bold and italic with the tag extensions that cause normal display
http://newmars.com/forums/viewtopic.php?id=3585

of which we would want to retain the corrected appearance...

this one has the quote tag issue that causes the pane to shift
http://newmars.com/forums/viewtopic.php?id=3583

Offline

Board footer

Powered by FluxBB