You are not logged in.
Hopefully an older backup will allow for the missing posts to be put back in at some point as that requires direct access to the msql data base. I would not worry to much about them. I am studying how to push data into the data base via a command line but I am a long ways off from being able to push a missing post back into the data base.
The day started out with a discharge escape battery and freezing cold. Pull the big charger that I have which can trickle, boost and start a vehicle to put a charge onto the battery and after a few hours the battery had the ability to start the escape but the alternator has died once more. Of course when its so cold that every joint and then some of my body was in pain from the cold. Allowed the vehicle to run while connected to the charge so as to warm it up as well as for me to get a bit warmer.
Drove it down to the friends garage and got the 2007 Prius that was fully paid for back in December and feel a bit better with running wheels once more. The place where the alternator was purchased from has a 1 year warranty so I will need to remove it so that they can bench test it to prove that its bad but since the battery is holding a charge that leaves just the alternator as its barely putting out any power. I noticed that it had engine oil on it so possibly the windings are shorted after getting to hot.
Offline
For SpaceNut re #1926
Re posts .... ok ... I'll just keep reporting them and when you need the post numbers they'll be available.
As a reminder, we ** don't ** want to restore an old backup over the posts that have been repaired.
Because of the age of these posts, I'm inclined to suggest we let them rest in peace.
***
Sorry to hear of the really difficult day up there in the Granite State.
On the other hand, it sounds as though the Prius is working!
I logged in just now to report that we have another missing post.
781
782
783
784 were all missing
*** resuming at 785
Offline
We would mount the data on a remote server to pull up the post content and merge just that missing post into the repaired data base.
Thanks for the missing numbers as that will help later.
Offline
For SpaceNut re #1928
Question for you regarding missing posts ... Could those be for a topic that was deleted by an Administrator?
After the series of missing posts reported earlier, WBA has continued on without incident.
I suspect that FluxBB does not re-use post ID's, so if a post is deleted, or if an entire topic is deleted, those post ID's would remain unused forever.
Ordinarily, no one would ever discover the missing posts. The reason ** we ** are finding them is the unique way we are processing the old posts.
And we wouldn't even have discovered them at all, if I had not turned on the Stop-At-Error feature.
I need that in order to find the posts that FluxBB refused to accept the first time around.
Report on Second Repair run to 900
After that series of potholes in the post sequence, there were four posts that needed repair.
The author entered forward slash B at the start of BOLD, and apparently the old system was fine with it.
(th)
Offline
For the record, here is a summary of missing posts in the series from 601 to 900.
Step: 41 http://newmars.com/forums/edit.php?
http://newmars.com/forums/edit.php?id=00894
Bad request. The link you followed is incorrect or outdated.
717
718
720
722
723
**** restart WBA
781
782
783
784
**** restart WBA at 785
Corrected these posts:
873
877
882
894
(th)
Offline
I would think that a deleted post and for a topic that is deleted that the ques for the numbers would be resigned as available but I do not really know for sure as there are large parts of the data base that is missing due to the crash which have topic and posts numbers.
Offline
For SpaceNut re #1931 plus Post Repair process
It is possible for us to confirm how FluxBB and MySQL deal with primary keys when objects are deleted.
The standard practice (Microsoft SQL for example) is to NEVER re-use a primary key.
However, we can find out by running a simple experiment.
We can create a post that is to be sacrificed in the name of Science.
We would record the primary key (ID) of that post, and then delete it.
We would then create another post in that same topic.
I expect that FluxBB and MySQL will create the new post with a new ID, and leave the deleted one vacant.
It would be useful to have this knowledge firmed up.
If the vacancies I found yesterday are from deletions by Administrators long ago, then there would be no point in trying to restore them. The should be gone from any backup that might have been taken.
***
The current run to 17,700 has balked twice, and there were three hiccups in the first segment.
I manually edited and saved the three hiccups.
In a minute or two I'll look at the log from the second session, which ended at 17593.
Changing the script to halt-on-error is making this process more tedious, but on the ** other ** hand the work product should be clean when we reach 77,000.
(th)
Offline
For SpaceNut re Posts Repair
WBA really struggled to reach 17700 today! The change to the script is causing halts when FluxBB rejects a proposed update. However, it ** did ** finally reach the goal.
I'll start the re-scan from 901 to 1200 next.
I hope it goes better, since these posts ** should ** all be correct.
(th)
Offline
test post was http://newmars.com/forums/viewtopic.php … 33#p190633
which was deleted.
Offline
The deleted post was not recovered and the next open was used to make the post above this one.
So that is valid for that condition of a deleted post but for the crash topic posts it would not be.
Will need to think about how to find the missing posts of the past which were lost in the crash.
Offline
for SpaceNut re #1935
Thanks for running that test!
I haven't been following the weather news for your area, but I got the impression most of the activity will be East, off shore. Hoping you don't get as much snow as RobertDyck has been showing us!
(th)
Offline
For SpaceNut ... WBA found another pothole: 1148
Bad request. The link you followed is incorrect or outdated.
The rest of the posts up to that point were present and already repaired.
The run to 1200 proceeded without further incident. I ** think ** is is safe to expect that any post between 1 and 1200 is up to current standards, if it is present at all.
Edit: The Smart Wait function is a major step up from the traditional Dumb Wait.
However, I can see that it could be enhanced. The default behavior is to stop the run if it does not find the expected Address Bar value in 60 seconds, with a retry every second. The existing Smart Wait effectively handles Internet congestion, which can delay responses from FluxBB for extended periods.
However, encounter with a pothole results in a halt that could be avoided if Smart Wait were able to handle alternative scenarios.
I've considered introducing subroutines to the script design in the past, but the need was not compelling and the idea never rose to the task list.
Now, as we embark upon a verification run, I can see that there ** are ** going to be potholes, because Administrators before you have made clear that they could and did delete posts. The test post you sacrificed recently is now a pothole in the posts Index.
Later edit:
The traditional way to define a subroutine is a Begin-End block....
In this case, such a block might begin with:
SmartWaitAlternateBegin
PlaceToLook:AddressBar or Screen
StringToFind:abc
ActionToTake:Exit, Skip (something else)
SmartWaitAlternateEnd
The script parser could stack as many of these as might be needed.
The existing default is to Stop the Run (ie, Exit)
The one that would fit discovery of a pothole is to skip to the next candidate.
***
The Dell laptop gets no break today, due to the long runs yesterday. We'll start today's stretch to 18000 now.
(th)
Offline
We are expecting blizzard conditions with anywhere from 12" upward before the storm ends Sunday.
Offline
For SpaceNut re #1938
Best wishes to stay safe, dry and (hopefully) warm during the blizzard!
***
Pothole report:
http://newmars.com/forums/edit.php?id=17804
Continuing from 17805
***
Pothole report:
Starting Sequence for ID: 17807
Bad request. The link you followed is incorrect or outdated.
I wonder how many more of these there will be?
Update at 13:09 local time ... the very next halt was for the image and url combination that FluxBB rejects.
I can scan ahead to look for those.
It might even be possible to scan for potholes, but I'm not sure I can do both in the same scan.
It's certainly worth a try.
***
Starting Sequence for ID: 17864
Bad request. The link you followed is incorrect or outdated.
***
Starting Sequence for ID: 17896
Bad request. The link you followed is incorrect or outdated.
There are 103 posts to go to 18,000 ... I wonder if we'll make it!
Nope!
Starting Sequence for ID: 17909
Bad request. The link you followed is incorrect or outdated.
***
Starting Sequence for ID: 17924
Bad request. The link you followed is incorrect or outdated.
In order to try to get a handle on this situation, I collected all the scan logs done earlier, and determined that they do indeed contain the "Bad Request" text. At the time, the script was simply correcting data so the missing post was not a problem.
My plan for a possible work-around is to print the ID's that are missing, and then plan runs based upon those numbers. Ideally, WBA would be enhanced to handle interrupts like this, but that will take time. i'd like to try to stay on schedule to complete 300 posts per day.
***
Starting Sequence for ID: 17929
Bad request. The link you followed is incorrect or outdated.
At long last, the green screen of successful completion appeared!
Completed Sequence for ID: 18000
Total Command Lines found: 42
Total input Lines in script: 131Number of ID's processed: 71
Starting Number: 17930
Last Number of Run: 18000
Summary for Web Automation Report for 01-29-2022 at 18:08:49
Average time of Loop from Main form: 00:00:33Total time of Processing: 00:39:47
I'm hesitating to start tonight's re-scan of completed work. So many potholes showed up today, I'm leery of starting the run without checking to see how many potholes lie in wait.
I'll start the run because the laptop is warmed up and ready, after the struggle with the run to 18,000.
(th)
Offline
This post is dedicated to the subject of potholes in the NewMars database.
I collected all the scan files from the runs last year, which were intended to find the end of the period when bandits were part of the posts.
Today I ran a scan of those files to find any potholes. I found a number in the lower part of the index, and will post them shortly.
However, I did ** not ** find the dropouts encountered recently.
My concern is that dropouts may be occurring as we go along.
There were 19 dropouts reported by the scan last year:
485
486
698
700
702
715
716
717match
718match
720match
722match
723match
727
744
745
781match
782match
783match
784match
End of potholes reported in Scan of NewMars posts in late 2021.
The "match" text was added to show the missing posts found by the updates run, compared to the scans run last year.
It is possible the dropouts are occurring due to stress on the server, and they may not be permanent.
On the other hand, it seems likely that the posts that "match" between last fall and now are really gone.
The scan from last fall covered 75,000 posts, and at that time, only 19 were recorded as missing.
Update at 21:55 local time ....
The re-scan from 1201-1500 was flawless.
No posts needed re-work. We appear to have posts 1-1500 ready for inspection.
(th)
Offline
For SpaceNut re Posts Repair ...
The run to 18,300 hit a pothole early:
Starting Sequence for ID: 18016
Bad request. The link you followed is incorrect or outdated.
For SpaceNut... Adrian moved a topic ... that would account for deleted posts
Moved thread to Intelligent Alien Life forum.
The running count right now is:
18022
18023
18025
Update later .... missed a couple ID's
Most recent is 18227
***
Starting Sequence for ID: 18251
Bad request. The link you followed is incorrect or outdated.
Completed Sequence for ID: 18300
Total Command Lines found: 42
Total input Lines in script: 131Number of ID's processed: 49
Starting Number: 18252
Last Number of Run: 18300
Summary for Web Automation Report for 01-30-2022 at 14:35:42
Average time of Loop from Main form: 00:00:33Total time of Processing: 00:27:19
I'll start the re-scan from 1501 now. I hope it goes more smoothly.
The re-scan encountered just ** one ** faulty post, which contained a url signal at the same time as a bold request. Other than that, the run was uneventful.
We appear to have a clean slate from 1-1800...
Completed Sequence for ID: 1800
Total Command Lines found: 42
Total input Lines in script: 131Number of ID's processed: 127
Starting Number: 1674
Last Number of Run: 1800
Summary for Web Automation Report for 01-30-2022 at 18:31:29
Average time of Loop from Main form: 00:00:34Total time of Processing: 01:13:51
PS ... news reports tend to focus on Boston, so the snow there is on the screen. I'll be interested in your report from further inland. Hopefully your area did not have a power outage. The news video showed ** lots ** of those.
(th)
Offline
For SpaceNut re Post Repair process....
There are two or three situations that are occurring in the update series.
They are (presently) causing halts, because I activated the halt-on-error feature of the Smart Wait command.
1) Potholes ... posts are missing ... FluxBB sends a message to that effect, but that message is currently ignored
2) url inside img sequence ... This is present in a few posts, and WBA currently does not see it or fix it
3) italics inside url .... that is a new configuration that showed up today.
I've been thinking about how to improve performance ... The Smart Wait is working well, and I prefer not to change it
There ** is ** an option .... After a post is loaded into memory for processing, it could be examined by a new command that would look for oddball configurations (including potholes) and simply skip to the next post.
That is attractive, because it would be a clean command, and they could be stacked, as needed.
The operation I am considering is to find a fault condition, report the ID and the nature of the fault, and skip to next post.
If none of the possible fault conditions are found, then the program would continue with the existing edit process.
Because we are only at 18,300 in the main line to 77,000, I am inclined to make the time investment in a new command, because otherwise halts will continue, and what should take two hours and change is going to take multiple hours.
After the new command is activated, the operator will have to check for faults, and then fix them manually, but that is **much** better than what we have in place now.
(th)
Offline
For SpaceNut
First, I hope power is back on, if it was lost.
Second, I hope the snow is cleared, or is well on it's way to cleared.
I'm holding off on Post Repair runs until a new script command is available.
It turns out there was already a script command that had most of the functionality needed for the new command.
Installation of the new command should go much faster than would otherwise have been the case.
The elements to be updated are:
Help documentation: done
Parser - to do
Runtine process - to do
End of job reporting - to do
system testing - to do
The new command appears to be working ... it found a pothole: 18572
Completed Sequence for ID: 18600
Total Command Lines found: 44
Total input Lines in script: 138Number of ID's processed: 30
Starting Number: 18571
Last Number of Run: 18600
Summary for Web Automation Report for 01-31-2022 at 20:51:01
Average time of Loop from Main form: 00:00:33
1 Skip Exceptions were recorded.Total time of Processing: 00:17:30
Several halt occurred due to situations that were not caught in the earlier scan.
I'll start the re-scan shortly ... it will cover previously repair posts between 1801 and 2100
(th)
Offline
Power stayed on in spite of the high winds. Shoveling of 8 inches of snow is not fun but its all done until this next weekend that we get more of about the same number of inches. Not looking forward to more shoveling but it will get done.
The Prius seems to be running good in that tomorrow I will be starting to test mileage to fuel cost to see how many miles to the gallon I am getting so as to get a baseline of savings that I am hoping will be from its purchase. Once the weather is better I will get the escape down to the friends shop and begin getting it back into order.
I am not worried about the potholes we are finding and the work is getting the forum back into shape once more thanks to the efforts which you are putting forth.
Offline
For SpaceNut re #1944 and Great!!! news that power stayed on!
A friend (who lives in Cleveland) reported they got heavy snow last week and thick ice, but folks down state got very little.
Lake Erie is a source of a lot of the snow that hits that area, and I suppose the ocean was the source for your recent snowfall.
***
Glad to hear the Prius is holding up, and hope it gives you years of service! I like the idea of having a motor in the mix, even if it is small. Reports of Tesla vehicles holding up in stuck traffic are coming in, but I'd hate to be out in one in heavy snow.
***
The new pothole report-and-skip command found a pothole almost immediately after I activated it. After that, there were halts, but they were for things that are hard to predict, left over from a time when members were trying to learn how to do BBcode.
The re-scan seems to be trouble free so far.
(th)
Offline
For SpaceNut re Post Repair Re-Scan to 2100
The last leg of the run to 2100 was uneventful
I ** think ** posts from 1-2100 are ready for viewing.
The script is now set to report any updates it does, and in the last segment there were none/zero.
Likewise, the script will report if there were any potholes, but there were none.
Finally, the script will now report if there were any problems with url inside img requests, and there were none.
Completed Sequence for ID: 2100
Total Command Lines found: 44
Total input Lines in script: 138Number of ID's processed: 44
Starting Number: 2057
Last Number of Run: 2100
Summary for Web Automation Report for 02-01-2022 at 08:17:00
Average time of Loop from Main form: 00:00:36Total time of Processing: 00:27:34
Today's first scan will 18,600 to 18,900
Halts are still possible, but hopefully they will only occur when manual attention is needed.
Update at 12:54 Local time ...
There've been a series of halts recently, showing that the existing script is getting confused by some of the compound quotes that forum members have been (er, were ) generating in 2003.
The complaint by FluxBB is that the quotes are not balanced. I can easily imagine this happening when a bandit is being removed and WBA takes out a perfectly innocent quote string while pruning the bandit.
The most recent halt was at 18880 ... we have only 20 ID's to go to reach today's goal.
Update at 13:56 local time...
[q u o t e] was opened within [-i-], this is not allowed
Good Grief!
It turned out this was one of Josh Cryer's posts, and the issue was the slash was after the "eye" instead of in front of it. It would take some pretty sophisticated program to deal with ** that **.
Completed Sequence for ID: 18900
Total Command Lines found: 44
Total input Lines in script: 138Number of ID's processed: 13
Starting Number: 18888
Last Number of Run: 18900
Summary for Web Automation Report for 02-01-2022 at 14:22:03
Average time of Loop from Main form: 00:00:34Total time of Processing: 00:08:06
The rescan will start shortly ... coverage is 2101-2400
Hopefully this run will proceed without halts. It is working with the output of the initial run.
Update at 14:48 local time ... the count reached 2136 ... it should be done by 17:00 (or so) if all goes well.
Update at 19:52 local time .... There were a couple of halts along the way, so the run to 2400 ended later than it might have, but the two glitches were easy enough to fix, and the posts are now ready for inspection.
Completed Sequence for ID: 2400
Total Command Lines found: 44
Total input Lines in script: 138Number of ID's processed: 39
Starting Number: 2362
Last Number of Run: 2400
Summary for Web Automation Report for 02-01-2022 at 19:54:26
Average time of Loop from Main form: 00:00:39Total time of Processing: 00:26:10
The Dell laptop gets to take a rest over night.
(th)
Offline
For SpaceNut re Posts Repair initiative ....
This post is reserved for today's run to 19,200 in the breaking-trail phase, and to 2700 in the clean-up-validation phase.
WBA still needs a bit of work ... It should be possible to better handle conditions that are currently causing halts.
At present, if FluxBB rejects a proposed update, the field that WBA is checking for results never updates so time-out occurs and the default is to close the program. I am thinking about changing that behavior to simply note the failure and proceed to the next candidate post.
The solution ** might ** be to simply create another Smart Wait with the Continue option built in. The existing Smart Wait is working so well I prefer not to change it.
For SpaceNut ... inside baseball ...
1) FluxBB complained about italics being requested inside italics ... I changed the inside to bold and it accepted
2) FluxBB complained about mis-spelling of quote as qutote ... Not much a program can do about that.
The trail-break run has stopped so many times I lost track ... we are up to 19094 on the way to 19200.
Meanwhile, I suspect my recent update to skip to the next ID is not working as desired. It reports a skip but doesn't skip.
Update at 14:50 local time: Surprise! Green Screen
Completed Sequence for ID: 19200
Total Command Lines found: 44
Total input Lines in script: 138Number of ID's processed: 151
Starting Number: 19050
Last Number of Run: 19200
Summary for Web Automation Report for 02-02-2022 at 14:51:11
Average time of Loop from Main form: 00:00:34Total time of Processing: 01:26:49
With any luck, the validation run will complete without halts.
This run will cover 2401-2700
The snippet below is in need of debug stepping
# Edits should be complete
#
SearchClipboardZeroSkipIfMatch:[*img][*url]
#
ReportClipboard:0
Inside baseball... the new command worked fine in Loop Test mode. I forgot to add it to Step mode and to Run mode.
A new version is in process.
Meanwhile, back at the farm:
Starting Sequence for ID: 2604
Bad request. The link you followed is incorrect or outdated.
The halt occurred because the new skip feature is not yet installed in the main run loop.
A corrected version of WBA handled the final stretch of the validation series.
We reached 2700, and because FluxBB caught any (several) problems that had been missed previously, i am now fairly confident the first 2700 posts (less potholes) are in good shape.
I'm looking forward to seeing if the new version has better luck with the trail breaking series tomorrow. FluxBB is still going to object to mis-spelled words (like qutote instead of quote) but at least it won't halt if it hits a pothole, or if it finds a url command inside an image command.
(th)
Offline
From the update it would seem that the program is working better and that once not fixed posts are now all healed.
We are expecting another good amount of snow and rain in the next few days.
Offline
For SpaceNut re #1948
Thanks for noting progress !!!
Today, the program seems to be finding the img inside url problem! It reported the problem and did not halt.
Query Text Requested Was Found: Routine Reset
Skipping to Next Item
Completed Sequence for ID: 19311
I manually fixed 19311 and now it should pass review in a few weeks.
WBA halted at 19335, for the italics inside url problem. I can't program for that easily, but FluxBB is set up to find and report those incidents, so a halt is fine if it doesn't happen too often.
***
We have sleet/ice building up slowly.
Other folks further North have it much worse.
***
I'm encouraged by James Burk's outreach effort, but am holding off on saying anything until everyone else offers their ideas/suggestions.
GW Johnson is scheduled for a dress rehearsal of his Orbits talk this coming Saturday, at 2 PM Houston time.
We have one confirmed NSS person to attend to provide feedback. This rehearsal is primarily for me, because I have volunteered to serve slides on Zoom, and have never done it before.
The slides and notes are published in the GW Johnson postings topic.
***
GW Johnson could teach hundreds or thousands of young folks how to perform celestial navigation using nothing more complex than a spread sheet, if we could (somehow) find a pathway.
What I have in mind is something like the Kahn Academy, with a structure to move from novice to professional, which the Kahn Academy does not have.
***
More inside baseball .... I've mentioned this before ... I'm still trying to convince myself to invest the time to create a more flexible version of the Smart Wait.
We need the existing hardnosed version of Smart Wait to catch situations where the browser?Internet?FluxBB? goes off the rails. This happens every now and then. A worse case scenario is that the "powers that be" (whatever they are) decide to log the session out! In that situation, a swift termination is needed, and Smart Wait (version 1.0) does that.
However, investing time in building a more flexible version would (if implemented) catch many of the oddball scenarios that are causing halts at this point.
The italics inside a url is caused by one particular member who went to the trouble of creating posts with this formulation, and the bulletin board software of the time was fine with it.
FluxBB is less tolerant.
If I can talk myself into investing the time, a more flexible Smart Wait would record the situation in the log, and resume with the next ID.
Update at 12:25 local time ... Post 19499 generated a halt .... It was one by Josh Cryer ...
copy failed ... the complaint was that Josh had enclosed the letter "s" in brackets.
I changed to parentheses and FluxBB was happy. Good Grief!
The run to 19500 ended without further halts (there was only one post to deal with)
The re-scan from 2701-3000 starts now.
Because the new pothole detector is operational, I'm expecting a flawless run.
We'll see!
Update at post 2888 ... FluxBB rightly objected to the post when submitted. The original author (who is still active in this forum) somehow created a post with quotes that ended up (after parsing) with the slash in front.
I manually adjusted the slash positions,. and the post accepted.
Meanwhile, installation of a new more tolerant Smart Wait is progressing. Major components are in place, but these need adjustment. Still, I think there is a good chance the new version will be ready for live testing this evening.
Update at 19:41 local time ... the Re-Scan reached 3000.
With several posts manually repaired, I ** think ** 1-3000 are ready for use.
I noticed that a topic started by RobertDyck in 2004 is free of bogies. The Post ID was above 20,000, well inside the zone where repair is needed. I'm guessing someone must have worked on that topic. It looks clean as a whistle now.
(th)
Offline
For SpaceNut re Post Repair inititiative
An update to WBA to skip to the next ID is installed. I ran a short test to confirm it runs "normally" when FluxBB accepts a post. Tomorrow, in a normal 300 item run, there will (probably) be a chance to see if it handles a reject by FluxBB with better grace than in the past.
***
We've had a slow but steady flow of ice/snow all day, and apparently it will continue all night.
The accumulation is small, but it will be challenging for people and vehicles, without doubt.
The dogs I see out on their daily exercise runs don't seem bothered.
(th)
Offline