Housekeeping

tahanson43206 · 2021-05-11 20:17:14

For SpaceNut re #1125

There will be approximately 200 duplicates in the course of the entire run of 3608 Usernames collected from the Bans table.

I used a feature of spreadsheet software (apparently available in all major brands) the detects duplicates. I removed the duplicates from the file of Usernames the program is reading. However, the original duplicates remain in the Bans table, so they'll be there for you to dispose of as you see fit. My impression is that a duplicate record in the Bans table can be deleted, because (as reported a few posts back) the Bans table does NOT include an ID field for the user record.

Thanks for converting/upgrading as many ID's as you can.

(th)

SpaceNut · 2021-05-11 21:01:17

Just did a search via email in the ban list to see what I would find for the *gmail.com 14 pages.

xzusaymebtqhzrc
xzusalmebmdnddm
xzusafmeynagc3f
xwusaymemmadc2c
xwusalmexnmaxsy
xwusalmecmcmxsr
xwusalme2nemczy
xwusafmeennucci
xvusayme3nlicdj
xvusalmerntgd2w
XHoortC
write my paper
WltonHall
wilsonelisa430
AnoPoush

tahanson43206 · 2021-05-12 06:18:34

For SpaceNut re ID Recovery ... WBA 1.1 reached 5100 in yesterday's run...

I'll take a look at the report you sent! I think I saved a copy of the spreadsheet showing duplicate names in the Bans list.

Edit: there was no email in NewMarsMember * gmail.com

One advantage of the current process is that we will slowly but surely be able to eliminate duplicates from the Bans table.

Total Command Lines found: 108
Total input Lines in script: 518
Number of ID's processed: 100
Starting Number: 5001
Last Number of Run: 5100
Summary for Web Automation Report for 05-12-2021 at 09:13:00
Average time of Loop from Main form: 00:03:03
19 Date Exceptions were recorded.
Total time of Run: 13:38:21
19 Date Exceptions were recorded.

(th)

Last edited by tahanson43206 (2021-05-12 07:17:59)

tahanson43206 · 2021-05-12 13:20:43

The ID Recovery process has exceeded 5,000 recovered ID's ...

Because automation was introduced to expedite the ID review process, this post is available to show some of the screens involved:

SearchTerm:WBA screens at run time
SearchTerm:WebBrowserAutomation

Opening screen WBA 2021.05.11.1
RS-232 Setup Screen
Screen with Setup complete

(th)

SpaceNut · 2021-05-12 17:32:43

Nice appearance to the front end gui.
Continued from last nights stopping point of 5013 and continued through to 5100 with out ant duplicates. Did a re-ban on 5001 to place it back into the table since it should that it was banned in the user table...

tahanson43206 · 2021-05-12 18:46:59

For SpaceNut re #1130

Thanks for taking a look at the program is it stands ...

Down the road (after the ID's are taken care of) I'd be willing to see if the program can handle some of the sophisticated tasks you've mentioned for recovery of old posts.

The technique of collecting data from fields on the screen might be adapted to the editing task.

However, it is possible you are using tools not available to moderators.

That's a way in the future.

Reminder ... I'll try for 5101 to 5200 later tonight ...

(th)

Last edited by tahanson43206 (2021-05-12 18:47:35)

tahanson43206 · 2021-05-13 07:13:43

For SpaceNut ... ID Recovery report

Completed Sequence for ID: 5200

Total Command Lines found: 108
Total input Lines in script: 418
Number of ID's processed: 100
Starting Number: 5101
Last Number of Run: 5200
Summary for Web Automation Report for 05-13-2021 at 09:10:28
Average time of Loop from Main form: 00:03:03
8 Date Exceptions were recorded.
Total time of Run: 11:48:46
8 Date Exceptions were recorded.

Tonight's run will try for 5300

(th)

SpaceNut · 2021-05-13 21:24:42

The run in the 5100 to 5200 was clean as soon as it got to 5210 we had a duplicate at which I stopped unbanning the table.

Did another but this time checking yahoo email address duplicates with ip 95.65.79.205 seems to be common to them
xvusafmehmoadqf
xvusaymecncnxom
xwusalmewnbkzjg
xwusaymectfqcua
xwusaymemtkdcjh

did the same check for banned emails for hotmail.com these are the duplicate
xwusafmedtkpd3e
xzusalmevnohzbi
xzusaymebnipddx

tahanson43206 · 2021-05-14 06:10:58

For SpaceNut re #1133

Thanks for the report on more duplicates in the Bans table ...

The program stopped at this Username overnight: AnthonyBed

That Username was present in the Bans table but NOT present in the Username table.

A possible explanation for this situation is that AnthonyBed may have been a member of a duplicate pair that was upgraded in the past, with the duplicate left behind.

In general, it is my impression that if a record exists in the Bans table without a corresponding Username in the Users table, then the Bans table entry can be removed.

The exception (as you have explained) is if you want to prevent use of an IP address or an email address.

As a suggestion ... if you want to keep the banned email addresses, you could run an experiment to see if you can create a record in the Bans table without a corresponding Username in the Users table.

It may not be possible to do that, but what appears to be possible is to rename records in the Bans table without deleting them.

I'd be interested in the results of any experiments you might run along those lines.

***
It would take a bit of programming to be able to handle missing Usernames in the Users table.

The reason is that I have tried to program WBA so it is universal, with variable operations confined to the Script.

The current date processing procedure is an exception, because the details of how that is done are unique to the FluxBB record (the Profile).

A "missing Username" situation calls for the ability to skip a Username that is found in the Bans table but which is missing in the Users table. The solution I have in mind is to be able to return to a point in the script where we last had a successful compare (in the case of Internet failure) or to return to the top of the loop to fetch the next Username from the stored list, in the case of a missing Username.

For the time being, I'll just restart the program at 5242, by manually advancing to the next Username.

(th)

Last edited by tahanson43206 (2021-05-14 06:16:15)

tahanson43206 · 2021-05-14 11:44:29

For SpaceNut .... The WBA run to 5300 stopped at 5276. The ID was in Profile, so I finished the process manually.

I suspect an Internet delay (either remote or local) might have caused the halt, because the ID itself seems fine.

I'll restart the run to 5300 shortly.

Update at 15:18 local time:

Total Command Lines found: 108
Total input Lines in script: 3112
Number of ID's processed: 24
Starting Number: 5277
Last Number of Run: 5300
Summary for Web Automation Report for 05-14-2021 at 15:17:21
Average time of Loop from Main form: 00:02:57
2 Date Exceptions were recorded.
Total time of Processing: 01:16:49
Total time Program was Active: 01:17:00
2 Date Exceptions were recorded.

I'll attempt the run from 5301-5400 tonight.

Meanwhile, I've started work on a new command for the script ...

The exact wording remains undecided, but it would be along the lines of: SearchClipboard:text

What I'm thinking about is the No Results message that appears if a Username is not present in the Users table.

A few of the names collected from the Bans table are not present in the Users table.

The most likely explanation is they were part of a duplicate pair, and when one member of the pair was upgraded, the second was left behind, due to a (possible) oversight in the version of FluxBB running at the time.

A Search command could look for the Not Found message and report the Username to the log.

It should be possible to verify all 3000+ Usernames fairly quickly, since only a few script commands are needed.

Edit at 17:13 local time .... modifications to permit use of a new command are under way ... SearchClipboardZeroReportMatch is a tentative new command.

The related commands are already present ... StoreClipboard:0 allows the contents of the just-retrieved clipboard to be stored in an array (row zero)

The text included as a parameter of the new command will be whatever text makes sense at the time.

In this case, I want to find any Usernames from the Bans table that do not a corresponding entry in the Users table.

(th)

Last edited by tahanson43206 (2021-05-14 15:16:29)

tahanson43206 · 2021-05-14 14:31:13

For SpaceNut re "Glimpse of Possible Future"

SearchTerm:Glimpse of possible future
SearchTerm:Future Glimpse of possible
SearchTerm:Automation Economic Division of Responsibility

Your initiative in encouraging me to work on the ID Recovery project provides (me at least) a glimpse of what human life might be like for at least a part of the population in future.

We already have a significant percentage of the population who live off investments.

We have another percentage of the population who derive income from running corporations which do not require their constant attention.

We also have a percentage of the population (here I'm thinking of folks in Agriculture) who start processes in motion (with a lot of effort) and then let Nature take it's course until it is time to harvest.

My experience with the WBA (as it is evolving) is (or at least seems to me) similar to Agriculture.

I put some effort into setting up a run, and then let it go, confident there are enough safeguards built in to that if something goes wrong, the program will stop and not do any harm.

A person who starts a crop on it's way may check it's progress from time to time, but otherwise (most of the time) the crop does not require any intervention during the growing period.

So **here** is the glimpse of a possible future that just came to me ...

If we humans can organize ourselves so that most of us (excluding retired and children) can assume responsibility for an automated process that has economic or social value, then the constant drumbeat of the need to work for someone else can be alleviated somewhat.

There will (no doubt) remain positions in society that require total immersion ... examples would be teachers and other educators, engineers and scientists, and no doubt other fields.

However, if the means of production of goods can be (somehow) distributed so that every citizen is responsible for some part of the automation that keeps society going, then the divisions between people can be reduced.

(th)

Last edited by tahanson43206 (2021-05-15 09:48:33)

SpaceNut · 2021-05-14 17:41:04

I see AnthonyBed with 3 other names in the ban list for the same ip 104.223.79.184 of which I am wondering if its a deleted username....

Now to check the banned table for converted Id's

edit so far TestID05238 has a duplicate
pue 5239 back into the banned as it was showing banned in the userlist…

So we got 5213 through less the couple up to 5300

tahanson43206 · 2021-05-14 20:32:57

For SpaceNut re ID Recovery .... thanks for bringing everyone up to date!

I'm about to run a test of a new command to find missing user names ...

SearchClipboardZeroReportMatch ... the script calls for a match if the query does NOT find a Username

Edit at 23:15 local time ... A test for one Username takes 38 seconds, so a set of 100 will take about an hour and 15 minutes.

That is a block of time I can run any time during the day, since no updates are performed.

The advantage of running this test is that the risk of a halt due to a missing Username in the Users table will be eliminated, so an update run should proceed without interruption.

Edit next day: 7:30 local time

Completed Sequence for ID: 5400

Total Command Lines found: 110
Total input Lines in script: 3088
Number of ID's processed: 100
Starting Number: 5301
Last Number of Run: 5400
Summary for Web Automation Report for 05-15-2021 at 07:26:58
Average time of Loop from Main form: 00:03:05
8 Date Exceptions were recorded.
Total time of Processing: 05:10:54
Total time Program was Active: 07:25:16
8 Date Exceptions were recorded.

(th)

Last edited by tahanson43206 (2021-05-15 05:31:32)

tahanson43206 · 2021-05-15 05:41:02

For SpaceNut re ID Recovery...

This is a report from the new ID Not Found detection script:

Text from Clipboard in Step: 16 http://newmars.com/forums/userlist.php
Text from Clipboard in Step: 23 http://newmars.com/forums/userlist.php? … rch=Submit
Starting Sequence for ID: 2
Text from Clipboard in Step: 16 http://newmars.com/forums/userlist.php
Text from Clipboard in Step: 23 http://newmars.com/forums/userlist.php? … rch=Submit
Query reported Not Found: fakeID
Completed Sequence for ID: 2
Starting Sequence for ID: 3
Text from Clipboard in Step: 16 http://newmars.com/forums/userlist.php
Text from Clipboard in Step: 23 http://newmars.com/forums/userlist.php? … rch=Submit
Completed Sequence for ID: 3

Total Command Lines found: 30
Total input Lines in script: 3
Number of ID's processed: 3
Starting Number: 1
Last Number of Run: 3
Summary for Web Automation Report for 05-15-2021 at 07:37:59
Average time of Loop from Main form: 00:00:26
Average time of Loop from Main form: 00:00:26
Total time of Processing: 00:02:07
Total time Program was Active: 00:02:32
Successful Run. No Warning Errors.

I'll be running this ahead of the main run, to make sure the update script does not fail due to a missing Username.

Update at 8:47 local time

Starting Sequence for ID: 38
Text from Clipboard in Step: 16 http://newmars.com/forums/userlist.php
Text from Clipboard in Step: 23 http://newmars.com/forums/userlist.php? … rch=Submit
Query reported Not Found: Assignment
Completed Sequence for ID: 38

The Username "Assignment" will be removed from the queue for tonight's run.

(th)

Last edited by tahanson43206 (2021-05-15 06:58:36)

SpaceNut · 2021-05-15 08:56:35

taking a peek at todays run

banned list emails with @yandex.com

Emeldatup
Wilsonlyday

@m.dfokamail.com
TestID05348

asowunesixe
atufuoy
azooroe
azuwasjbap
wokivhamb

tahanson43206 · 2021-05-15 09:33:47

For SpaceNut re ID Recovery....

Thank you for starting review of the 5301 series !!!

Username Email IP/IP-ranges Expires Message Banned by Actions
TestID05348 testid05348@test.com Never Upd: 2021/05/15 SpaceNut Edit | Remove
TestID05348 izecuk@m.dfokamail.com Never SpaceNut Edit | Remove

In this situation, I recommend upgrading TestID05348 to member status, and deleting the duplicate Bans table record.

There are over 200 duplicates in the Bans table. I have removed those from the list of ID's to investigate to see if they might quality for upgrade, but they are still there in the Bans table, and you will see them as we move ahead. Please note that I observed that duplicates appear to have been clustered, so we may/will have to deal with multiple occurrences in a single run.

In the Division of Responsibility that goes with this project, I think that only an Administrator should be removing records from the Bans table.

We moderators can add ID's in an emergency, and occasionally that happens, but in the present time, it is increasingly unlikely anyone in the membership will need that remedy. While there are still thousands of probable spammers left in the active membership, of late they have been bothering other forums.

(th)

Last edited by tahanson43206 (2021-05-15 09:36:27)

tahanson43206 · 2021-05-15 10:07:08

This is for SpaceNut ...

By any chance was a software upgrade to the web site (FluxBB) performed a few minutes ago? Local time is noon(ish) ...

The (possible) outage showed up as the command to open login.php was automatically converted to index.php.

That is a fascinating (to me at least) way of performing maintenance on the web site mid-day, if that is indeed what happened.

(th)

tahanson43206 · 2021-05-15 10:55:27

For SpaceNut re Recruiting ...

Just FYI ... An acquaintance has achieved significant visibility in social media in recent years .... LinkedIn sent a notice of a recent promotion, so I sent a note of congratulations, and added a brief mention of need for folks to help NewMars. To my surprise, she followed up,

We have plenty of bright, shiny (newly refurbished) User ID's ready for new members, if any take an interest in this part of the Mars Society spectrum of activities.

(th)

SpaceNut · 2021-05-15 15:55:45

The remove only changes the ban condition and does not delete the duplicate.
only one of them are real and the next is I the due website restore after it failed during the great crash.

tahanson43206 · 2021-05-15 16:43:13

For SpaceNut re #1144

Thanks for adding info about the duplicates.

I'm willing to see what powers the Moderator has.

Please designate a duplicate you want removed. I'll report what happens when I try to delete it.

How about Allenhat?

That Username does not exist in the User table.

It does not need to be kept in the Bans table unless it has an email address you want to prevent re-use, or an IP address you want to block.

It has both an email address and an IP address.

I'm curious to see what happens if I try the "Remove" option, but will wait for your confirmation you want me to try it.

There is no User record associated with that Bans Table record, so removing it should have no effect on anything else.

There are 3244 records in the Bans table ... when I delete Allenhat, there should be 3243.

I'll wait to see if you want me to try the experiment.

(th)

Last edited by tahanson43206 (2021-05-15 16:53:36)

tahanson43206 · 2021-05-15 16:59:17

For SpaceNut re return of Mars_B4_Moon

It was fun seeing a member from the early 2000's back in the mix. I went back to the beginning of his membership and read for a while ... there were a lot of interesting discussions back then, much as today ...

One thing I noticed was a recommendation we turn Registration back on ...

I think this member has no idea what you've been dealing with.

I'd like to take this opportunity to remind any formerly active members that NewMarsMember * gmail.com can help if anyone has forgotten a password.

However, there should be ** no ** problem for anyone to recover access, because the FluxBB package will send a recovery email to the email account you used when you set up the account.

If you are no longer using that account (which is certainly possible after 20 years or so) then just write to NewMarsMember * gmail.com.

I will NOT take on the responsibility of bringing an older member back into the forum. I will notify SpaceNut of the request immediately.

SpaceNut can employ the tools he has available to correct the situation speedily.

(th)

SpaceNut · 2021-05-15 18:02:01

for ip 104.223.79.184
BrianBlend valid user list

others do not show
Allenhat
AnthonyBed
Jamesecony

give it a go

tahanson43206 · 2021-05-15 18:28:30

For SpaceNut re Bans Table Remove experiment

I'm curious to see what happens if I try the "Remove" option, but will wait for your confirmation you want me to try it.
There is no User record associated with that Bans Table record, so removing it should have no effect on anything else.
There are 3244 records in the Bans table ... when I delete Allenhat, there should be 3243.
I'll wait to see if you want me to try the experiment.

I checked the User table for each of the three candidates, and they were not present.

I then used the Remove button to remove the records from the Bans table.

The count of records in the Bans table was 3244 before the experiment and it is now 3241.

The count of Registered users should NOT change, and it did not change: Total number of registered users: 19,633

However, that number is generated occasionally so is a lagging indicator.

I expect that number is stable, since the Bans table is independent of and (apparently) not linked to the Users table.

***
I'll be starting tonight's run in about an hour. The run should complete without incident, because the input table was verified this morning.

(th)

Last edited by tahanson43206 (2021-05-15 18:31:15)

SpaceNut · 2021-05-15 19:28:30

just went through the 65 pages of bans looking for duplicates here are the names

atufuoy
azooroe
azuwasjbap
DarciSage
Emeldatup
irqveizr
timivqfe
WillianHof
Willieanirl
WillMcgrat
WillyEnyea
WilmerYjb5
wilsonelisa430
Wilsonlyday
WindyDittm
WinifredCu
WinifredSu
winifredyk11
WinnieOliv
winshlwin
wljgodki
WltonHall
wokivhamb
wolfghgi
WoodrowIrrarf
WyattKiddl
xeewxuxk
XHoortC
XiomaraAsh
xiphrlWrell
xiwuroyj
xlzrxmqq
xmrtxpaa
xpdlcsWrell
xpwdkgdp
xpxwpltq
XSYGabriel
xtymprrs
xvusafmehmoadqf
xvusafmeumqudej
xvusalmerntgd2w
xvusayme3nlicdj
xvusaymecncnxom
xwtyvdqg
xwusafmedtkpd3e
xwusafmeennucci
xwusalme2nemczy
xwusalmecmcmxsr
xwusalmeenlbzez
xwusalmenmygznl
xwusalmewnbkzjg
xwusalmexnmaxsy
xwusalmeymzrxxo
xwusaymectfqcua
xwusaymemmadc2c
xwyayvqe
xwyyapdk
xzusafmeynagc3f
xzusafmeznohcuv
xzusalmebmdnddm
xzusalmeemxiztk
xzusalmegnzyzaa
xzusalmetmxzzmw
xzusalmevnohzbi
xzusalmevnohzbi
xzusaymebnipddx
xzusaymebtqhzrc

tahanson43206 · 2021-05-15 20:05:53

For SpaceNut re #1149

My count was for over 200 duplicates. I used a feature of modern spreadsheets to detect and report duplicates.

Did you use something like that?

I'm launching tonight's run now ...

(th)

New Mars Forums

Announcement

#1126 2021-05-11 20:17:14

Re: Housekeeping

#1127 2021-05-11 21:01:17

Re: Housekeeping

#1128 2021-05-12 06:18:34

Re: Housekeeping

#1129 2021-05-12 13:20:43

Re: Housekeeping

#1130 2021-05-12 17:32:43

Re: Housekeeping

#1131 2021-05-12 18:46:59

Re: Housekeeping

#1132 2021-05-13 07:13:43

Re: Housekeeping

#1133 2021-05-13 21:24:42

Re: Housekeeping

#1134 2021-05-14 06:10:58

Re: Housekeeping

#1135 2021-05-14 11:44:29

Re: Housekeeping

#1136 2021-05-14 14:31:13

Re: Housekeeping

#1137 2021-05-14 17:41:04

Re: Housekeeping

#1138 2021-05-14 20:32:57

Re: Housekeeping

#1139 2021-05-15 05:41:02

Re: Housekeeping

#1140 2021-05-15 08:56:35

Re: Housekeeping

#1141 2021-05-15 09:33:47

Re: Housekeeping

#1142 2021-05-15 10:07:08

Re: Housekeeping

#1143 2021-05-15 10:55:27

Re: Housekeeping

#1144 2021-05-15 15:55:45

Re: Housekeeping

#1145 2021-05-15 16:43:13

Re: Housekeeping

#1146 2021-05-15 16:59:17

Re: Housekeeping

#1147 2021-05-15 18:02:01

Re: Housekeeping

#1148 2021-05-15 18:28:30

Re: Housekeeping

#1149 2021-05-15 19:28:30

Re: Housekeeping

#1150 2021-05-15 20:05:53

Re: Housekeeping

Board footer