Housekeeping

tahanson43206 · 2021-05-02 11:37:00

For SpaceNut re ID Recovery ...

WBA is ready for the next run, with the first cut at stopping when we reach TestID Usernames.

I'll run that test after you have a chance to look at zmaxime.re

Meanwhile, outdoor chores await!

(th)

SpaceNut · 2021-05-02 14:11:14

I got started this morning on the banned table review only to find about 20 duplicates that had different email addresses which means that it was somehow linked on the crashed emails being duplicated so I change the groups that were in the 4500 and 4600 series that were duplicates by incrementing just the one digit to a 7 so as to keep the cycle number as it was so that you could look at the log file.
I am now back from the laundry chore so I can now move them out of the banned table to the user list.

edit
id's 4506 through to 4674 are now reset and in the userlist

zmaxime.re was a russian email address that spammed. I have reset the post count to zero....

tahanson43206 · 2021-05-02 16:40:59

For SpaceNut re #1077

First, thanks for confirming zmaxime.re is reset to 0 .... More importantly (to me for sure!), you've confirmed you can do that if it is needed << grin >>

Second ... I was taken aback by your reports of (about 20) duplicates ... What I'm not clear on is what to look for in the log. The next time that happens, please take a moment to post the ID that you found. It is possible the Internet/Intranet failures caused by the poor local wireless connection might have led to activity the program cannot see at present. I ** should ** be able to build additional checks into the script, at points where they are needed ... The email address not being updated properly in the Bans table is tricky because (if you think back) you instructed me to remove the log record of the old email address. I agree with the basic idea, that such old address should not be left in a file somewhere, but the result is I don't have it.

Third ... the only ID I see missing at the moment is 4672 ...

I'll manually recreate that one so I can start a new run at 4675.

Thanks again for your attention to detail as we proceed! It really helps!

***
Also, I was happy to see the exchange between you and Oldfart1939, and the commentary by GW Johnson about SpaceX landing planning.

It was ** good ** to see the safe landing at sea this morning (on Earth in Florida) ... the local residents kept their distance, with Coast Guard out in force.

Edit at 18:51 local time ... OK ! 4672 is created using the Russian spam ID ... hopefully we'll find a member who'll help with projects to fill that slot!

I'll start a run with the latest version of WBA shortly ... I'm looking for signs the new Stop At feature is working. The "clue" will be log reports that the ID it is working on is NOT one of those at which it should stop. That doesn't mean the program will actually ** stop ** when it hits a TestID, but at least it will be confirmation the inverse is working.

Once we hit TestID records, we can reset the query to look for records beyond the TestID group.

(th)

Last edited by tahanson43206 (2021-05-02 16:53:51)

tahanson43206 · 2021-05-02 17:33:19

For SpaceNut ... here is another legitimate stop

ZliftonFiege The creator of this ID visited once after registration day

If you want to set a rule for this situation, I can manually upgrade an account like this.

Otherwise, we can just let the program run until it hits one of these, and you can decide what you want to do.

User activity
Registered: 2019-10-20 (5.166.219.6)
Last post: Never
Last visit: 2020-02-13 08:44:11
Posts: 0 - Show all topics - Show all posts - Show all subscriptions
Admin note

***
Since a new feature is in test mode, here is a snip of the log for the run with Stop At added ...

Starting Sequence for ID: 4676
Text from Clipboard in Step: 18 Zlo88Kep
Text from Clipboard in Step: 38 http://newmars.com/forums/userlist.php? … rch=Submit
Text from Clipboard in Step: 47 http://newmars.com/forums/profile.php?id=431417
Text from Clipboard in Step: 52 Zlo88Kep
At Step: 60 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
PerformFunction: Valid ID: match in: Registered: 2019-05-15 (193.169.252.10)
Last post: Never
Last visit: 2019-05-15 03:33:32
Posts: 0 - Show al
Text from Clipboard in Step: 69 TestID04676
Text from Clipboard in Step: 78 TestID04676@test.com
Text from Clipboard in Step: 88 http://newmars.com/forums/profile.php?s … &id=431417
Completed Sequence for ID: 4676

Here's a bit of detail that (I suspect) is only interesting to someone following this really ** really ** closely ...

The log shows Record Number early, at step 47, before the name change, and at step 88 after the name change.

This shows that the name change was applied to the same record as was fetched for work.

(th)

Last edited by tahanson43206 (2021-05-02 17:39:59)

SpaceNut · 2021-05-02 17:41:26

email address is Yandex which is a known email spammer domain.
its another that was just waiting to do damage....

just did a check of the ban list

TestID04672 correct email Never Upd: 2021/05/02 RobertDyck Edit | Remove
TestID04675 correct email Never Upd: 2021/05/02 SpaceNut Edit | Remove
TestID04675 *** deleted email info**** 81.163.62.221 Never SpaceNut Edit | Remove
TestID04676 correct email Never Upd: 2021/05/02 SpaceNut Edit | Remove

tahanson43206 · 2021-05-02 18:01:29

For SpaceNut re #1080

First, thanks for the note about Yandex!

And! Thanks for the follow up regarding 4675 ... That is in the log just ahead of 4676 so I'll look at it.

My guess is I'm not picking up something as we go along, so I'll try to figure that out.

***
I'm about to post another item about Analog's Science Fact series ... this one is about 3D Printing

(th)

tahanson43206 · 2021-05-02 18:10:51

For SpaceNut re ID Recovery ...

OK ... the problem appears to be in the Bans table...

There are a number of duplicates in the section immediately ahead ...

Before i start running again, please take a look at the situation.

The Bans table obviously allows duplicate bans ...

The User table does not allow duplicate email addresses.

My recollection is it does not allow duplicate Usernames either ... it's been a while since I saw a warning about that, but (thinking back) that would have been the case.

However, the Bans table obviously does allow duplication Usernames.

Do you want to just let the program do as much as it can and clean up afterward?

(th)

SpaceNut · 2021-05-02 18:25:31

Thats correct for the current version but that may not have been so back when the accounts got created...in either case we can resolve them with my checks and moves to make the duplicate appear up stream....

tahanson43206 · 2021-05-02 19:22:24

For SpaceNut re #1083 ...

Thanks for taking a look at that situation ... If you're willing to put up with the aggravation I'll try a run up to 4700 this evening.

Halts could occur, as you have seen, so I'm not expecting clear sailing. I got spoiled by the (comparatively) easy going in the Not Verified set.

We may be dealing here with a (modest) flaw in FluxBB .... I hope the ability to create duplicate Ban records has been corrected in the version we are running now.

***
Best wishes for the work week ahead, and for an open slot for your work on the Subaru engine.

(th)

SpaceNut · 2021-05-02 19:56:31

No worries keep going and we will back fill the 4700's or remove duplicates as needed.

tahanson43206 · 2021-05-03 06:12:16

For SpaceNut re ID Recovery

Last night's run got all the way to 4699 when it halted again.

I'm getting used to the halts for legitimate reasons, and just manually deal with the minor issues.

As long as the post count is zero (which has been true since the ID you looked at), I'm evaluating the Last Visit date with respect to the Registered date, and so far I've decided to recycle the ID manually. So far the Last Visit was within a day or two of Registration, as though the spammer was confirming the login ** really ** worked.

The last two ID's needed manual upgrade. The current high water mark is 4700.

I'll try 4701-4800 tonight, but don't expect it to get far.

One thing to consider is to change the Validation procedure to not worry about the Last Visit date.

That way the Validation criteria would be:

a) The ID was already banned and
b) It has zero posts

If you authorize that change, the rate of progress should improve.

Refinement is possible ... For example, since I have the dates converted to system dates as part of the comparison procedure, I can perform difference calculations.

If the Last Visit is within one month of Registration, then we could consider the ID a spammer?

That would not be hard to do, and it would reduce the number of false stops.

Other refinements are possible ... for example, Registration can be more than a year ago.

Edit at 8:37 local time ... Moved a post by Louis from Primary to Companion

I assume Louis was checking to see if we meant business.

With your support and approval, I will continue to keep the runway clear for Oldfart1939

Edit 12:22 local time:

I'm starting to think about possible ways of implementing Validation, pending your decision on how you might want to proceed ...

A particular way of performing validation could be called for by the script, so it would be easy for the operator to increase or decrease severity of validation.

The current validation is strict, but there may be times when some relaxation of the criteria for halting the program might be desired.

An immediate option that would be easy to implement would be "PZ" ... PostZero filter ...; This command would perform the same validation as now for Post count, but would skip the dates check.

At run time, the PZ could be swapped for the PF if we hit a group of ID's which don't pass PF.

Or, as indicated earlier, more refined date checking is possible ...

I'm trying to design this program to have universal application, and everything except PF could be used for any web site.

The validation procedures may be difficult to make universal.

(th)

Last edited by tahanson43206 (2021-05-03 10:29:34)

tahanson43206 · 2021-05-03 11:57:25

For SpaceNut ... a new wrinkle

First, the good news ... the program stopped when FluxBB reported that TestID##### is already present in the Bans table.

The number reported matched the ID's in the Bans table that were given new names recently.

A possible solution is to rename the duplicate records to something other than TestID#####

For now I'll stop and wait for your decision on what you want to do.

(th)

Last edited by tahanson43206 (2021-05-03 17:14:33)

SpaceNut · 2021-05-03 19:52:39

These were the duplicates that were still in the ban table which some still had the original emails with the new names so I made them idA and IDb to do the remove from the table to see what happens.

tahanson43206 · 2021-05-03 20:37:04

For SpaceNut re #1088

Thanks for taking a look at those little critters! My guess is they were created in error by the ban procedure, and are no longer relevant because the banned Id's have been recovered.

The way looks clear for another run. I'm not expecting it to get far because the strict validation is still in place, but we could get lucky!

Nice to see Oldfart1939 still working on the 17 person landing party concept.

***
I ordered the new Million Person City competition book ... The book is also available on Kindle. I like to write in my books, so ordered the hard copy.

Update next day ... the program only ran for 4 ID's before hitting a snag.

(th)

Last edited by tahanson43206 (2021-05-04 06:15:14)

tahanson43206 · 2021-05-04 13:26:42

For SpaceNut re ID Recovery ....

First ... I read Calliban's report of difficulties faced by his wife, and would like to express sympathy and best wishes for eventual full recovery, and for freedom from side effects if he decides to take the vaccine.

Second ... I followed the link provided by Louis, and saw a notice the launch had been scrubbed.

I don't ** know ** that the launch was scrubbed ... all I can ** know ** is that I saw a display to that effect on the link Louis provided.

Now ... to the matter at hand ...

Without tolerance for for a few days between Registration and last_Visit, I am currently only able to process one or two ID's at a time. It takes longer to set up for a run than it takes to manually update ID's.

If you got back to me about date tolerance, I missed it. I'm going to test an updated version in a few minutes, with the added capability to evaluate candidates for upgrade based upon the difference in days between Registration and Last_Visit. The span of days allowed can be adjusted from the setup screen at run time, to a change to the script is not needed.

I set the default span to 30 days. Please let me know if you want the span reduced.

I told WBA to report exceptions to the log, and I'll be able to show you what it found.

Edit at 15:34 local time ... fail occurred on the first ID ... the new code did not deliver the expected result.

Back in a bit ...

Edit at 15:54 ... there ** may ** have been a switch missing .... There are two ways to halt the program, in addition to dozens of ways in the script, and I may have missed one.

Here goes try 2 ...3...4...

OK ... we seem to be able to handle a 1 day interval between Registration and Last_Visit ... 4710 is available for your review.

Edit at 19:05 local time ....

This is more like it ... we got all the way from 4711 through 4720.

The ID for you to look at is zducwf

That ID has one recorded post...

Starting Sequence for ID: 4721
Text from Clipboard in Step: 18 zducwf
Text from Clipboard in Step: 38 http://newmars.com/forums/userlist.php? … rch=Submit
Text from Clipboard in Step: 47 http://newmars.com/forums/profile.php?id=432158
Text from Clipboard in Step: 52 zducwf
At Step: 60 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:

Here's what it looks like if an ID passes all checks:

Post-Date Function Start: Reading Clipboard #4
PerformFunction: Valid ID: match in:
Registered: 2020-03-19 (118.69.77.91)
Last post: Never
Last visit: 2020-03-19 08:23:51
Posts: 0 - Show all

Here's what it looks like if an ID has a Date mismatch that is within the range allowed at run time:

Post-Date Function Start: Reading Clipboard #4
Last visit: date did not match Registered: date Registered: 2020-09-20 (185.210.217.250)
Last post: Never
Last visit: 2020-09-21 13:36:13
Posts: 0 - Show all topics - Show all posts - Show all subscriptions
Admin note

Submit When you update your profile, you will be redirected back to this page.
Jump to
Go
Powered by FluxBB

I'll wait for your decision on the ID with a post, before continuing.

(th)

Last edited by tahanson43206 (2021-05-04 17:15:28)

SpaceNut · 2021-05-04 18:02:07

Not overly concerned of the date of registration to day of last visit as meaning much of anything for what we are trying to do.

zducwf was a spammer so I set the count to zero....

tahanson43206 · 2021-05-04 18:29:01

For SpaceNut re #1091

Thanks for getting back to me about the ID recovery questions .... I'll leave the window set at 30 days between Registration and Last Visit for now. Most of those situations seem to be on the "one day" category, as the spammer created the account before midnight and then accessed it after midnight.

Thanks for taking care of zducwf ... I'll let the program loose again, to see how far it gets. I'm feeling increasingly less on edge about letting it go ... there are so many checks built in that even when I haven't thought of something, the checks seem to catch it.

***
Best wishes for you and your family as you try to figure out what to do with the vaccinations ...

In the States we seem to be mostly offering Phyzer and Moderna, which are the RNA designs. Those use a completely different mechanism for dealing with the virust ...

***
Here's an item that may be a bit of a surprise ... I went in for my annual check up today, because the Virtual method was not working on Monday. I was amazed to see how busy the practice was during my visit. I guess folks have business other than Covid to attend to ???

What I'm reporting here was that when my doctor asked routine questions, I mentioned being active with the Mars Society, and she asked a number of follow up questions that showed more interest than would be typical for these interviews. That's just a random sample of one, of course, but still, it suggests to me that perhaps awareness of Mars as a destination is beginning to seep into the body politic.

***
I wonder how far WBA will get this time << grin >>

Last time was all of 10 ID's before hitting a stop.

(th)

SpaceNut · 2021-05-04 18:34:05

The doctors office here have another problem in keeping the doctors that they have...

Cleared all of the none duplicates from the ban table for the day.

I am going to send the ones that are duplicate via email to you at the newmarsmember so you can look at them...

I am going to alter the id's with the a and b as before and fill in the corrected email in order to remove them from the ban table....

edit

I kept toggling between the user table and the banned table as I made changes to the id's that were correct versus those that appeared to be duplicates.

As I change those that were correct for username and email and did the unban the user list showed it was still banned so when I change the duplicate to the idA and made the email correct followed by the unban I was surprised to see the userlist condition change from banned to member when making the duplicate the IDA changes.... There is cross linking on the emails which has been happening but will it eventaully come out of it is the question...

tahanson43206 · 2021-05-04 19:50:44

For SpaceNut re #1093

Something to think about is the User iD of the duplicate Ban table records.

If the User ID is the same, then the duplicate Ban records should be deleted.

On the other hand, if the User ID is different then that represents a possible recoverable User ID.

The User ID is the number that you see at the right when you click on a name in the User list, and FluxBB brings up the Profile for that User.

I assume the User ID is included in the Ban table record, but I'm not sure how to see it.

Perhaps the tools you have available are able to see that level of detail.

***
The latest run got all the way to 4735 ... it hit an ID created in 2017 and accessed in 2020.

That is OK with me, to manually process an iD like that if it doesn't happen too often.

I'll start back up again shortly.

NewMarsMember will be happy to have some email ... things have been quiet there recently, except for the occasional update from Noah on his studies.

(th)

SpaceNut · 2021-05-04 20:14:54

sent another email with information and since you only did through *** its strange to see others in the banned table some how....

tahanson43206 · 2021-05-04 20:24:53

For SpaceNut re #1095

Thanks for the information to look at tomorrow!

I just came in to find the halt at a gap of (about) 50 days ...

Just as an experiment, I'll set the gap at 60 days (this was back in 2017) and see if the ID gets processed.

I'll restart the run at 4737 ...

(th)

tahanson43206 · 2021-05-04 22:13:22

For SpaceNut re ID Recovery ..

We had a decent run this time --- 4736 through 4763

The halt was for an ID with 2 posts

Please take a look at it ... I'll restart after your decision.

ZacharyAccen
User activity
Registered: 2020-06-26 (185.39.11.89)
Last post: 2020-06-26 12:45:29
Last visit: 2020-06-26 12:45:30
Posts: 2 - Show all topics - Show all posts - Show all subscriptions
Admin note

(th)

tahanson43206 · 2021-05-05 07:21:51

For Rxke,

It is good to "meet" someone who joined the forum in 2003 and has 3661 posts, ** and ** knows something about chickens.

I am happy to see support for RobertDyck's Large Ship, and if chickens are what it takes, so be it! Anything to get the Large Ship off the dime.

In a population of 7 billion people, I am glad to know that there is a productive specialization called 'manuscript photographer'.

Would you be willing to describe the occupation a bit more. Your expressed doubt that this specialization would be needed on Mars may be right, or perhaps not. It ** sounds ** similar to the work of scientists, some of whom spend their working days making images of materials they are studying, and Oldfart1939 has already identified several kinds of scientists he would recommend for an expedition.

Here is an opportunity for you to influence (or rather, ** try ** to influence) Oldfart1939 to include a specialist in documentation of scientific observations in his 17 person crew. Age of 50 would surely become 60 by the time Oldfart1939's expedition is funded, organized and on it's way to Mars. There may be good reasons to have someone in the crew of that age.

If you were willing to take on multiple responsibilities, such as assisting the scientists in donning EVA and checking the state of the machinery, and perhaps helping with maintenance of the living quarters in transit and on-planet, Oldfart1939 might be able to "see" a slot for a person with your background.

To find out, you can drop a note into the Companion topic of the 17 person expedition initiative.

(th)

Last edited by tahanson43206 (2021-05-05 07:24:00)

tahanson43206 · 2021-05-05 07:26:40

For SpaceNut ...

While we wait for you to take a look at ZacharyAccen, I'll plan to look at the email you sent to NewMarsMember, and touch up WBA a bit.

I'm thinking of improving the format of the log messages a bit ...

Edit#1 ... I ran a little experiment, and the query to fetch bans can be adjusted to pull just one letter of the alphabet ..

While waiting for you to take a look at a halt in the "Z" group, I'll ask WBA to tackle the "U" set ... there are a number of them, and there is probably a halt lurking there, but it'll be interesting to see how far we get.

This method has the distinct advantage of NOT testing the new Stop At code, which I'm not sure will work

Edit#2... The new procedure seems to be working ... a set of "U" banned ID's is in process

One refinement needed is the count of tabs to reach Edit in the fetch procedure ... The count is 22 for all ID's. The count is 18 for one page.

Reduction of query results to one page has the benefit of eliminating the possibility of going past Edit to Remove, which can happen if the results fall below 77 pages, as will happen eventually.

For that reason, I'm planning (now) to limit queries to 1 page of results.

I tested to see what happens when the page is exhausted, and the program went to the top of the Admin Bans page, where it will be caught by the existing verification process.

Edit #3: We had a (legitimate) stop in the U group: USZEliza70 has a post

I'll go on to another group pending your review

Edit#4: Here is an ID from the HA group that needs your review: HarriettSt

I'll try another similar "one page" group.

Edit#5: The next group I tried had only two members. I wanted to test the behavior of the program when it exhausts the supply of candidates from the query. The result was success. The program exhausted the supply, and proceeded as though it had a candidate. The tabs took it to a button to invoke the Ban page, and since the page was not the one expected, the program halted.

From this test, I conclude it is safe to continue forward using the "One Page" method.

Edit#6: HershelBro has one post

On to the next group ...

Edit#7: FelipaRomo has one post

(th)

Last edited by tahanson43206 (2021-05-05 14:30:12)

NewMarsMember · 2021-05-05 11:02:49

For SpaceNut re data from Bans table ...

Thank you for sending those two email with detail of the (somewhat perplexing) behavior of FluxBB ...

I'm guessing (again without access to the underlying ID field) that the problem we are seeing (thanks to your documentation) is a result of duplication of the UserId from the User table.

There must have been a time when an earlier version of FluxBB allowed an Administrator/Moderator to ban an iD multiple times.

Hopefully that behavior has been corrected in the current version.

You can test it by trying to ban an ID multiple times.

If the system allows you to create multiple Bans for a given User ID, then ** that ** is a problem that needs to be addressed.

The folks at fluxbb.org have been helpful in the past, and i am confident they will entertain a question about this if it remains a problem.

If you have the ability to see the actual ID of the records in the Bans table, please see if my theory is correct.

Again, based only on theory on my part, I am guessing that the query I run is fetching only the first copy of a duplicate record from the Bans table. I then upgrade that ID, leaving the duplicate in place. However, when I ** then ** run the Bans page to remove the email address, I would only be working on the first record that might be there. The system would only fetch one to display, and which one is delivered is (I am guessing) selected at random, or by some criteria such as physical placement in the Bans table (ie, ID order).

Meanwhile, I'm proceeding with "one page" queries for WBA.

The run just halted.

(th)

New Mars Forums

Announcement

#1076 2021-05-02 11:37:00

Re: Housekeeping

#1077 2021-05-02 14:11:14

Re: Housekeeping

#1078 2021-05-02 16:40:59

Re: Housekeeping

#1079 2021-05-02 17:33:19

Re: Housekeeping

#1080 2021-05-02 17:41:26

Re: Housekeeping

#1081 2021-05-02 18:01:29

Re: Housekeeping

#1082 2021-05-02 18:10:51

Re: Housekeeping

#1083 2021-05-02 18:25:31

Re: Housekeeping

#1084 2021-05-02 19:22:24

Re: Housekeeping

#1085 2021-05-02 19:56:31

Re: Housekeeping

#1086 2021-05-03 06:12:16

Re: Housekeeping

#1087 2021-05-03 11:57:25

Re: Housekeeping

#1088 2021-05-03 19:52:39

Re: Housekeeping

#1089 2021-05-03 20:37:04

Re: Housekeeping

#1090 2021-05-04 13:26:42

Re: Housekeeping

#1091 2021-05-04 18:02:07

Re: Housekeeping

#1092 2021-05-04 18:29:01

Re: Housekeeping

#1093 2021-05-04 18:34:05

Re: Housekeeping

#1094 2021-05-04 19:50:44

Re: Housekeeping

#1095 2021-05-04 20:14:54

Re: Housekeeping

#1096 2021-05-04 20:24:53

Re: Housekeeping

#1097 2021-05-04 22:13:22

Re: Housekeeping

#1098 2021-05-05 07:21:51

Re: Housekeeping

#1099 2021-05-05 07:26:40

Re: Housekeeping

#1100 2021-05-05 11:02:49

Re: Housekeeping

Board footer