Housekeeping

tahanson43206 · 2021-09-12 21:08:06

For SpaceNut re repair of old posts .... as you can imagine, I'm planning to hold the samples until we've run out of work on the ID Recovery.

Speaking of ID Recovery, WBA had another flawless run to 17,700 this evening. We'll be well into the 18,000 series when we run out of the current set of verified candidates.

Meanwhile, the Lander proposal is close to ship date (ready or not) and the On Orbit system is gathering momentum.

It's good to see all the activity in other parts of the forum!

(th)

tahanson43206 · 2021-09-13 20:31:58

For SpaceNut re ID Recovery .... WBA completed a run to 17,800 ID's this evening.

We'll be trying for 17,900 tomorrow evening.

(th)

SpaceNut · 2021-09-14 19:30:00

just got my computer back up after a crash that caused the reset restart boot repeat mode where auto repair that did not repair the startup.

tahanson43206 · 2021-09-14 19:53:11

For SpaceNut re #1578

Sorry to hear of this setback! I hope things stabilize for you!

***
WBA is closing in on 17,900 ... The Internet appears to be smooth this evening.

Update at 22:09 local time ... WBA reached 17,900 ... we'll shoot for 18,000 tomorrow night!

***
GW Johnson may (I repeat ** may **) be completely re-writing the Landing Pad probe proposal. Now's a good time to think about what you would like to see in a proposal that might fly in 2020. One of the papers you found shows that ESA's exoMars is scheduled for launch in September 2022 on a Proton.

Our lander (if a miracle occurs and we actually cause it to happen) would launch in the same window.

You appeared to be concerned about shadow on the solar panels, and had an idea for a fuzzy whisker cleaning tool of some kind.

Remember to keep the functions separate and not to confuse them. The drill goes down looking for water. The rover roams for many meters around the drill, testing the soil with a pounder of some kind.

(th)

Last edited by tahanson43206 (2021-09-14 20:11:40)

tahanson43206 · 2021-09-15 06:37:13

For SpaceNut .... It is good to see Spaniard back on view in the forum.

Unfortunately, Spaniard wrote a post about electric vehicles that he accidentally put into a dedicated topic.

I'd like to find a better place for his post. Can you think of where it would be a better fit.

(th)

SpaceNut · 2021-09-15 17:15:44

Mostly in response to kbd512 but he put his in a better topic so will copy it to the tail of one of my posts.

tahanson43206 · 2021-09-15 22:14:14

For SpaceNut re ID Recovery .... WBA just polished off the last full thousand!

Our total of TestID's stands at 18,000!

There are only 344 usernames left ....

(th)

tahanson43206 · 2021-09-17 05:53:42

For SpaceNut re ID Recovery

Just 244 usernames are left in the current series ...

Text from Clipboard Row 1 in Step: 15 vinifusa
Completed Sequence for ID: 18100
Total Command Lines found: 112
Total input Lines in script: 341
Total input Lines in data: 345

Number of ID's processed: 100
Starting Number: 18001
Last Number of Run: 18100
Summary for Web Automation Report for 09-17-2021 at 07:51:57
Average time of Loop from Main form: 00:01:29
17 Date Exceptions were recorded.
Total time of Processing: 02:29:10
Total time Program was Active: 11:12:30

(th)

SpaceNut · 2021-09-17 18:05:41

SpaceNut wrote:

just got my computer back up after a crash that caused the reset restart boot repeat mode where auto repair that did not repair the startup.

Began running a malware program since the video went blank to see if there was a cause on the total drive and it had found over 80 files but it got interrupted, so started it again yesterday after supper and it finished this morning. The software found over 100 files and 5 major program executables had been infected.

It seems to be doing ok after a reboot and the hard drive indicator is not thrash so that must have been some cause to its crashing.

tahanson43206 · 2021-09-18 06:47:46

For SpaceNut re #1584

I read your report last night and let it sit overnight. It just came to me why your computer might be subject to infestation. Your skill and energy investigating links to information to display on the forum must (inevitably?) expose your system to malware.

I use Linux primarily for searches. I wouldn't trust Microsoft products to be safe in an Internet infested by those who attack Microsoft for a living.

The commercial anti-virus programs seem to work, but it's been a while since I could afford them, and meanwhile Linux has been building up resistance as volunteers all over the world contribute to the code base.

Having said that, I admit that I sometimes perform searches on a Windows machine, so am at risk along with everyone else.

It is possible to run Linux from a CD (if your computer still has a CD player).

It is also possible to run Linux from a USB stick, without changing your computer in any way. The result is slower than it would be if working from a hard drive, but all the functionality is there, so you can see for yourself if setting up a dual-boot system might be worth while.

***
WBA completed a run to 18,200 last night ... The Internet was smooth on this occasion.

Text from Clipboard Row 1 in Step: 15 WaxilliamDib
Completed Sequence for ID: 18200
Total Command Lines found: 112
Total input Lines in script: 341
Total input Lines in data: 245

Number of ID's processed: 100
Starting Number: 18101
Last Number of Run: 18200
Summary for Web Automation Report for 09-18-2021 at 08:51:20
Average time of Loop from Main form: 00:01:29
8 Date Exceptions were recorded.
Total time of Processing: 02:29:17
Total time Program was Active: 11:43:47

(th)

Last edited by tahanson43206 (2021-09-18 06:54:02)

SpaceNut · 2021-09-18 09:33:59

The hackers have not stopped there as they are finding their way into the Apple oS as well as into the Andriods as well. Its only a matter of time before they begin there hate of other OS systems.

tahanson43206 · 2021-09-18 22:03:36

For SpaceNut re ID Recovery ... WBA is having a good final run .... it is closing in on the high water mark of 18,344.

The difference between 19,632 and 18,344 is 1,288. That is a much more manageable number of potentially ** real ** members.

When kbd512 has time to polish off the TestID's that remain in the Bans table, I'll be able to get a handle on how many (if any) of the remaining Bans can be recovered.

I haven't spent any more time (yet) trying to understand how WBA might be adapted to look at the 181,112 posts that FluxBB says we have.

My first impression is that the process of looking at successive posts in a topic (for example) is more difficult than the work we've done so far.

Completed Sequence for ID: 18344

Total Command Lines found: 112
Total input Lines in script: 341
Total input Lines in data: 142

Number of ID's processed: 141
Starting Number: 18204
Last Number of Run: 18344
Summary for Web Automation Report for 09-19-2021 at 00:26:13
Average time of Loop from Main form: 00:01:29
13 Date Exceptions were recorded.
Total time of Processing: 03:31:49
Total time Program was Active: 03:32:39

That's the end of the line. It was fun while it lasted.

I'm going to miss having WBA chewing on spammer fodder for hours every night for weeks on end.

(th)

Last edited by tahanson43206 (2021-09-18 22:29:08)

tahanson43206 · 2021-09-19 08:08:40

For SpaceNut re ID Recovery

As we have reached a high water mark of 18,344 TestID's, recovered from our friends the spammers, I'd like to acknowledge the many hours you've invested in setting up ID's for me to work with, and for your follow up activities prior to kbd512 offering to help with the final polish.

In a set of notes early in the process, I had made a note:

Posts Greater than 0: 1753
Posts equal to 0: 17880

Our final results show that we have exceeded the 17,880 count. The difference is: 465

I deduce you must have quietly cleaned up 465 ghost post ID's

Under your watch, the vulnerability to uncontrolled spammer attack is massively reduced.

I'm looking forward to any guidance you may have on how the tools developed for the spammer project can be adapted to clean up the archive, where it appears that many artifacts left over from a previous time still lurk to snare the unwary forum visitor.

Later: I ran a search for posts containing the string uid0 and got over 1000 pages and over 48000 posts

For SpaceNut re possible Archive cleanup

I ran a search for post_uid0, and got 1833 pages of results ...

Some showed damage (or produced incorrect displays). Some did not ...

The result set included several posts by SpaceNut, explaining the problem(s) and how to solve them ...

One from 2015 seemed particularly helpful.

I'm still in the very early stage of trying to understand the problem, although the scope seems clear enough.

FluxBB can (and does) deliver a result set of posts with the string uid0, but (at this point) I don't see how to work wih it.

i'll keep worrying at the problem and perhaps a solution will become "visible" ...

It just occurred to me there might be a way to approach this ...

The query doesn't change .... the program might be configured to work with the first post in the set.

After cleaning up the post and replacing it (using Edit), the program could run the query again.

The result set should now show the next post with a problem.

The program can be set to run for a fixed number of iterations, like 100.

If 48,000 is the number, we'd be talking about 480 days to retire the entire result set.

It would be good to have something for WBA to do for the next 480 days!

I think the first iteration should NOT perform updates, but instead it should collect work and report what it found, without actually doing anything.

That way, you can decide if you want the program to start actual updates.

Note: It might be possible to add a "DoNotTouch" flag to the posts which you do not want updated, such as the ones from 2015 that show what needs to be done. Or! Alternatively, and perhaps less work .... simply specify that posts by SpaceNut should not be updated.

The program would have to skip over those because they would show up each time the query is run, but a check for author only takes a few seconds.

There aren't that many (most likely) ... what do you think?

If you'd like to think about updating your notes from 2015, here is the search term:

I ran a search for post_uid0, and got 1833 pages of results ...

If I pull up a post containing this string, what do you want done with it? Once the post is in memory, any number of procedures can be run.

You could specify exact actions you want taken for particular situations.

We could also approach this as an iterative (learning) process ... we could add new remediation procedures as we encounter situations that show up.

One possibility is to replace any bracketed string containing post_uid (I notice that some of these have uid <> 0 (eg, 14) with a string such as **Editedyyyymmdd**

(th)

Last edited by tahanson43206 (2021-09-19 10:54:45)

tahanson43206 · 2021-09-19 18:02:45

For SpaceNut re post tune-up ...

For SpaceNut ....

Here are some examples from a set of posts that showed up when I asked for post_uid

An option for your consideration is to remove the entire string between and including left and right square brackets

[ c o l o r =#810541:post_uid9]

:[ / c o l o r :post_uid9]

[ c o l o r=#000000:post_uid0]

Keep bold? It would be possible to detect Bold and remove the post_uid# while leaving the bold code.

[b : p o s t _uid9]

[ / b : p ost_uid9]

What do you think?

(th)

SpaceNut · 2021-09-19 18:21:29

tahanson43206 wrote:

For SpaceNut re post tune-up ...
For SpaceNut ....
Here are some examples from a set of posts that showed up when I asked for post_uid
An option for your consideration is to remove the entire string between and including left and right square brackets

[ c o l o r =#810541:post_uid9]
:[ / c o l o r :post_uid9]
[ c o l o r=#000000:post_uid0]
Keep bold? It would be possible to detect Bold and remove the post_uid# while leaving the bold code.
[b : p o s t _uid9]
[ / b : p ost_uid9]
What do you think?
(th)

Highlighted red could completely go as the color control is different than in the past

the bold. Italic and quote commands require the :post_uid# to be removed with the bracket ] to close staying
There are other commands that are used from the bbcode as well but those are the main ones.

of course this is all done within the edit post function

try this topic as it under meta and if you can not edit I will add you to the moderate list for the folder.
http://newmars.com/forums/viewtopic.php?id=3550

ok I have found how to make the change so let me know how you make out.

tahanson43206 · 2021-09-19 18:39:58

For SpaceNut re #1590

Thanks for the guidance and first-try assignment!

This will require a special function to work with the text after it is copied to memory on the client PC. That is actually my least concern.

What I have to try to figure out is how to move through the topic selected for update.

It is possible you have found the solution to simplify the problem ... The link you provided calls up a topic by topic ID.

The answer may ultimately be to go through the topics by calling them up by number.

It'll probably be a week or so before I have time to focus on this. I am definitely looking forward to giving it a try.

Todo:Topic 3550

Update at 21:53 .... Started looking at the problem .... a possible way in may be the PID display at the top of the web page when each post within a topic is selected. That line could be captured, and (if captured) it would provide a direct callup capability.

As it happens, while I do not have update privileges for this particular topic, it appears I have quote capability.

Don't change anything for now. I am perfectly happy to be unable to update anything while learning how to deal with this.

I captured (quoted) a post, and I can use that to develop the "Cleanup Procedure".

Then, after the Cleanup procedure is working, I can show you the before and after, and you can decide if you want to try an update.

I expect to be able to select and update just one post, or (eventually) a collection of them, similar to what we did with the spammer ID's.

(th)

Last edited by tahanson43206 (2021-09-19 19:59:36)

tahanson43206 · 2021-09-20 07:23:09

For SpaceNut re Old Posts Cleanup campaign ...

This morning I experimented with display of topics ....

http://newmars.com/forums/viewtopic.php?id=0001

Aside from the humorous aspect of of topic 0001, I think there is a potential to approach the problem using a simple topic-by-topic fetch and display.

The date of the first individual post seems to show up in the same place, so that post can be selected.

The post then shows up on it's own page, and it's address is available.

The address of the post might be saved and used for later processing.

According to FluxBB, there are 7,720 topics and 181,151 posts

There's enough work there to keep WBA busy for years.

What I'm not clear on (at this point) is how to advance through a topic.

For a human forum visitor, the process is simple enough ... one operates the Page Down key until the end of the current page shows up.

At that point, the human forum visitor selects Next to advance to the next page of query results.

The tool that kbd512 showed us would (apparently) be able to activate a button by specifying it's name.

I don't have that capability at the moment, but must instead depend upon tabs to reach the field, or use of mouse position.

The problem with mouse position (in this case) is that the target is not in a fixed location, because the posts vary in size.

Edit: There ** is ** a possible advantage to scanning all topics, slow as the process might be ... many topics do NOT have problems.

For example, Topic 1 is clear of problems, aside from everyone being banned.

Edit at 10:12 local time ... for SpaceNut .... here's an experiment:

Here I will present my personal work in planetary engineering. I am also very interested to hear about the work of others.

I tried a very simple command: http://newmars.com/forums/edit.php?id=1

That appears to be Post #1

it came up directly in Edit mode, which implies that if I had wanted to make changes I could have.

Working directly with posts would cut out the middle man, for sure!

This will only work where the Moderator has update privileges, but at least it shows a possible pathway.

181,151 posts at 30 seconds each: 1,509 hours ... That would be four years at one hour per day, or one year at 4 hours per day.

WBA has run successfully for as many as 14 hours, during one of the verify procedures, and a typical run for 100 ID's was just under 3 hours.

Four hours would be a reasonable run length. Most posts won't require 30 seconds, because most posts won't require adjustment.

I would envision:

1) Fetch a post by #
2) Capture the contents of the Edit window to clipboard
3) Examine the clipboard looking for bandits - in most case none will be found so skip this post

Even if adjustments are required, they don't take long in memory so...

4) Paste updated text (include date of edit)
5) Loop

Update 11:04 local time ...

Come to think of it, the defective posts must all exist below some number. Can you find what the cutoff point is:

The repair process only needs to run from 1 through whatever the last defective post might be. The entire process might take less than a year.

(th)

Last edited by tahanson43206 (2021-09-20 09:06:08)

tahanson43206 · 2021-09-20 15:01:06

For SpaceNut re Post Cleanup ...

A preliminary run of a draft script is quite encouraging ....

In the (edited) transcript below, you can see that the contents of posts I can access are available.

If a post is not available, FluxBB says so ...

Update for Web Automation Report for 09-20-2021 at 16:46:43
WebBrowserAutomation
Version 2021.9.8.1
End of file: C:\DataDell\WebBrowserAutomation\20210921ScriptScanPostsByPostNumber.txt

Text direct from Clipboard in Step: 22 Here I will present my personal work in planetary engineering. I am also very interested to hear about the work of others.
Starting Sequence for ID: 2
Text direct from Clipboard in Step: 22 It is my opinion that in order to properly understand how a mix of gasses contribute to the warming of the planet, it is important to look at the spectra. Consequently, I created a spectra extrapolator which I hope to eventually use for terreforming application. You can play with it as an applet, download it as an applications or view a brief explanation of how it works. Shortly I will be improving the user interface to make it possible to zoom in and out with mouse operations. The applet, application and source are all packed in the same jar file, so if you download the jar file you’ve got everything.
Completed Sequence for ID: 2
Starting Sequence for ID: 3
Text direct from Clipboard in Step: 22 New Mars Forums
You do not have permission to access this page.
Completed Sequence for ID: 3
Starting Sequence for ID: 4
Text direct from Clipboard in Step: 22 New Mars Forums
You do not have permission to access this page.
Completed Sequence for ID: 4
Starting Sequence for ID: 5
Text direct from Clipboard in Step: 22 New Mars Forums
You do not have permission to access this page.
Completed Sequence for ID: 5
Starting Sequence for ID: 6
Text direct from Clipboard in Step: 22 New Mars Forums
You do not have permission to access this page.
Completed Sequence for ID: 6
Starting Sequence for ID: 7
Text direct from Clipboard in Step: 22 New Mars Forums
You do not have permission to access this page.
Completed Sequence for ID: 7
Starting Sequence for ID: 8
Text direct from Clipboard in Step: 22 New Mars Forums
You do not have permission to access this page.
Completed Sequence for ID: 8
Starting Sequence for ID: 9
Text direct from Clipboard in Step: 22 I have lots of ideas for a permanent Martian colony, and I'd like to hear if anyone else has any.
Completed Sequence for ID: 9
Starting Sequence for ID: 10
Text direct from Clipboard in Step: 22 To be brief I'll say planing and engineering. For more details you can look at my mars page.
Completed Sequence for ID: 10

Total Command Lines found: 22
Total input Lines in script: 64
Number of ID's processed: 10
Starting Number: 1
Last Number of Run: 10
Summary for Web Automation Report for 09-20-2021 at 16:53:01
Average time of Loop from Main form: 00:00:24
Total time of Processing: 00:04:40
Total time Program was Active: 00:04:50
EOJ Msg#1: Successful Run. No Warning Errors.

Based on this encouraging test, I'll schedule time in the near future to code a function to perform adjustments per your specifications.

(th)

SpaceNut · 2021-09-20 17:32:25

The edit topic post and view topic are not always the same index for a post or topic as these are different tables.
Take for instance the post id which I get after its submitted.

It knows that the tid=9336

The returned id for the post is pid=185258#p185258

the edit of the post is id=185258

The edit will only be true for a post that has not gone missing in the index table

the quote function would be tid=9336&qid=185258 a 2 part command

so and so on.

A post that is not in the table will come bad with a bad command error.

this is a topic that returns and empty content but it knew that the topic existed but not the posts with in it.
http://newmars.com/forums/viewtopic.php?id=3569

I had started to map the id's for the topics some time ago but its time consuming.

tahanson43206 · 2021-09-20 17:54:31

For SpaceNut re #1594

Thanks for your interesting reply, with insights new to me (for sure!).

Do you mind if I work on posts? It turns out to be incredibly more easy than any other way of approaching the problem.

FluxBB is managing the connection between posts and topics as well as it can. There's not much I can do to help with that.

The task I'm interested in taking on is cleaning up garbage commands that are left over from the early days before FluxBB.

Please take a look at the posts I showed you that i can't reach. Could you / would you add that topic to my authorized list?

I can't even tell what topic they are from.

(th)

SpaceNut · 2021-09-20 18:07:37

I just updated all of the folders so that you can moderate them. You still will not be able to change Admin posts.

tahanson43206 · 2021-09-20 18:23:04

For SpaceNut re #1596

Thanks for adding the new capabilities! I'll run a test again tomorrow.

Question for you .... It is best for the Cleanup to work with posts directly.

However, in your #1594, you brought to my attention that there might be posts without topics.

I might be able to ask FluxBB to tell me who "owns" a post (what topic it belongs to).

At the moment I have no idea how to do that.

You should be able to experiment manually. The command is: http://newmars.com/forums/edit.php?id=####

Once you see what the command does (and why I like it) perhaps you can see a way for me to ask FluxBB what topic it is from.

If a post is an orphan, I ** should ** be able to at least report that so you can look into it.

Edit: This might be a good question to ask FluxBB.org. I'll do that.

(th)

SpaceNut · 2021-09-20 18:30:01

The topic is how I know that I have fixed it all as I will go throu each of the posts its just easier to see the completed work.

Doing individual posts will be just as acturate to the task with the edit id +# once you are up ther in count you start to get the bad command error for a post that is lostuntil we manually feed it back in from the rescued log files that James got on another server. The trouble will be in defining the overlap post areas which had been damaged.

tahanson43206 · 2021-09-20 18:34:56

For SpaceNut re #1598

Thanks for thinking about the orphan question. Just FYI ...

I asked this at FluxBB.org:

I searched for orphan in topic titles before creating this new topic ...
The Admin at our site asked me to clean up old posts, left over from a previous BB system.
I think the best way to tackle this is by using ** http://oursite/forums/edit.php?id=#### **
This command brings up just the post I want to work with, and I can run the program in sequential order from 1 to 181,000+, a few hundred at a time. We figure we should be able to complete this work in a year or less.
The Admin asked if there is a way to spot orphan posts that might exist.
I have no idea how to do that, but there might be a way.
I'll have the post number to work with.
Is there a query I can run using that number to point back to the owning topic?
Thanks!
(th)

I created a new topic for the question, since no one appears to have asked it before.

(th)

tahanson43206 · 2021-09-21 07:35:24

For SpaceNut re Posts Cleanup Campaign ...

I'll be starting another test run of 100 posts shortly.

This one includes an alert if post_uid is present.

This is a report-only test run. I'm looking to see how things go after your adjustment yesterday.

I'm thinking of adjusting posts manually for a while, accumulating examples of corrections needed so they can be consolidated in a single action function.

Update at 10:29 local time

Thanks again for adjusting the Moderator privileges ... the program ran through 100 posts and found a number of occurrences of 'post_uid' strings.

I'll be able to study those to get a good sense of what needs to be added to the fix-it-function.

Completed Sequence for ID: 100
Total Command Lines found: 24
Total input Lines in script: 75
Number of ID's processed: 100
Starting Number: 1
Last Number of Run: 100
Summary for Web Automation Report for 09-21-2021 at 10:25:36
Average time of Loop from Main form: 00:00:26
Total time of Processing: 00:44:28
Total time Program was Active: 00:44:36

I estimate that post-back will add 4 or 5 seconds, so the average will be 30 seconds per post.

Update at 11:06 local time ... The 1-100 test delivered 10 instances of post_uid but not color.

I set up a run from 101-1000 to try to catch more examples of usage. The run will take about eight hours.

My thinking right now is to code the Cleanup Function to halt the program if it encounters a post_uid format it does not recognize.

That way the repertoire will increase/improve as we go.

(th)

New Mars Forums

Announcement

#1576 2021-09-12 21:08:06

Re: Housekeeping

#1577 2021-09-13 20:31:58

Re: Housekeeping

#1578 2021-09-14 19:30:00

Re: Housekeeping

#1579 2021-09-14 19:53:11

Re: Housekeeping

#1580 2021-09-15 06:37:13

Re: Housekeeping

#1581 2021-09-15 17:15:44

Re: Housekeeping

#1582 2021-09-15 22:14:14

Re: Housekeeping

#1583 2021-09-17 05:53:42

Re: Housekeeping

#1584 2021-09-17 18:05:41

Re: Housekeeping

#1585 2021-09-18 06:47:46

Re: Housekeeping

#1586 2021-09-18 09:33:59

Re: Housekeeping

#1587 2021-09-18 22:03:36

Re: Housekeeping

#1588 2021-09-19 08:08:40

Re: Housekeeping

#1589 2021-09-19 18:02:45

Re: Housekeeping

#1590 2021-09-19 18:21:29

Re: Housekeeping

#1591 2021-09-19 18:39:58

Re: Housekeeping

#1592 2021-09-20 07:23:09

Re: Housekeeping

#1593 2021-09-20 15:01:06

Re: Housekeeping

#1594 2021-09-20 17:32:25

Re: Housekeeping

#1595 2021-09-20 17:54:31

Re: Housekeeping

#1596 2021-09-20 18:07:37

Re: Housekeeping

#1597 2021-09-20 18:23:04

Re: Housekeeping

#1598 2021-09-20 18:30:01

Re: Housekeeping

#1599 2021-09-20 18:34:56

Re: Housekeeping

#1600 2021-09-21 07:35:24

Re: Housekeeping

Board footer