Housekeeping

SpaceNut · 2021-05-15 20:22:51

no all by eye
sounds good on the next run

tahanson43206 · 2021-05-16 06:08:26

For SpaceNut re #1151

Pretty impressive << grin >>

If you find yourself needing to do something similar in future, the tool set is slowly increasing. In this case, the steps are:

1) Accumulate file of Usernames stored in the Bans array (something not available from FluxBB directly)
2) Find the appropriate Duplicates Detection procedure in your spreadsheet software

For Item (1), WBA was given a script able to pull 50 Usernames at a time from the Bans table, save them to disk, and continue until all are collected.
The data collected was then fed into spreadsheet as a text file with tab and space separators. This caused the names to be isolated in Column 1
Now the challenge is to remove debris left over from the data collection phase.
There might be better ways to do this, but I just sorted the column into ascending order, and then deleted the words that were not Usernames.

The cleaned up list was then passed to the Find Duplicates procedure (Help for your spreadsheet will show how to to that).

In my case, the Find Duplicates procedure changed the font from 10 to 20 bold, so they were easy to find.

Again, there may be a better way, but I just deleted one member of each pair, and the font for the remaining item returned to 10.

The final product is the file being fed into WBA for the daily run.

A refinement is that I have now added daily cleanup procedure to make sure the Usernames actually exist in the Bans table.

Since I used a manual process to remove debris from the data gathering step, I must have missed some.

***
Last night's run had Internet time outs ... The run is back in motion.

Update at 9:16 local time

Completed Sequence for ID: 5500

Total Command Lines found: 110
Total input Lines in script: 2943
Number of ID's processed: 56
Starting Number: 5445
Last Number of Run: 5500
Summary for Web Automation Report for 05-16-2021 at 09:13:57
Average time of Loop from Main form: 00:03:03
Total time of Processing: 02:53:50
Total time Program was Active: 02:54:06
3 Date Exceptions were recorded.

(th)

Last edited by tahanson43206 (2021-05-16 07:17:06)

tahanson43206 · 2021-05-16 09:02:10

For SpaceNut re ID Recovery

In preparing for tonight's run, I'm scanning ahead to find any Usernames not in the Users table

Query reported Not Found: beatssqqqkj

(th)

SpaceNut · 2021-05-16 09:38:10

completed unban through to 5500 removing duplicates by doing edits from banned table

beatssqqqkj ip and email domain shows no others

tahanson43206 · 2021-05-16 09:49:31

For SpaceNut re #1154

Thank you for investing the time to catch up with the program.

Please do not hesitate to advise if you need a break ... We have 29 days of this ahead of us, if we keep the current schedule.

I'm aiming for (US) Flag Day to finish off the last of the saved Usernames.

You may well be adding new Bans as you have time, and there may be a whole new set of ID's to process after the current set is completed.

***
The addition of new members is going slowly compared to cleaning up the debris left by our Spammer friends.

Hopefully folks who might be interested in the NewMars forum will find their way here through web searches.

Another pathway, recommendations by current members, is also available, and perhaps there are candidates considering applying for membership as we "speak".

(th)

tahanson43206 · 2021-05-16 09:55:34

For SpaceNut ... I brought a temporary malfunction of the forum login page to your attention.

Have you had a chance to follow up? The symptom was that the login page temporarily directed visitors to the Index page.

The logical explanation for this would be that someone from Mars Society had blocked logins while performing maintenance on the web site.

Would you be able to confirm this?

Another explanation is less benign ... it is possible the FluxBB package contains a back door that would allow someone to log in with a special User ID and password that are buried in the Login page.

I'd like to know if the login page contains such a vulnerability.

This comes up as a distinct possibility because I reported the success of the ID Recovery effort at FluxBB.org.

The members of FluxBB.org are First Tier Hacker grade technical people. Most (if not all) of the members are running FluxBB.org web sites, and they have full access to the source code. If there is a back door built into FluxBB, then most (if not all) members of FluxBB.org would know about it.

Thanks!

(th)

tahanson43206 · 2021-05-17 05:35:10

For SpaceNut ... run to 5600 succeeded ...

Completed Sequence for ID: 5600

Total Command Lines found: 110
Total input Lines in script: 2886
Number of ID's processed: 100
Starting Number: 5501
Last Number of Run: 5600
Summary for Web Automation Report for 05-17-2021 at 07:11:30
Average time of Loop from Main form: 00:03:05
5 Date Exceptions were recorded.
Total time of Processing: 05:11:03
Total time Program was Active: 09:06:17
Exceptions were reported. Run ended normally

The "a" set of Usernames is exhausted. Any "a" names left in the Bans Table should be those with posts.

I'll follow up on that shortly.

Edit after looking for any remaining "a" Usernames in Bans table:

ser list
Username Title Posts Registered
Assignment Helper Banned 0 2018-03-27
Assignment Helpers Banned 0 2018-02-22
Assignment Helps Banned 0 2018-02-14
Assignment Notebooks Banned 0 2018-02-15
Assignment Operators Banned 0 2018-02-18

OK ... I missed these because I did not realize the FluxBB package allows spaces in User Names.

I'll process them manually.

Edit 8:41 local time ... the "a" usernames in the Bans table are cleared.

The new high water mark is 5605.

There are 2800 or so names left in the Username candidates file ... the "b" set are in process.

Edit at 9:11 local time ... the Verify run for tonight showed now missing usernames.

I'll set the run from 5606-5700, to keep the even century milestones.

(th)

Last edited by tahanson43206 (2021-05-17 07:12:45)

SpaceNut · 2021-05-17 16:14:19

Good to see some old new blood with the Mars_B4_Moon's return.

took care of tonight's unban list 5513 through to 5606
duplicate
TestID05508 eluoaken
TestID05515 oiomowa
still investigating but will clear them soon as well as 2 others.
TestID05238 perdontas+20201022
TestID05239 testid05239

edit made changes and did the unban and all is good for these....

tahanson43206 · 2021-05-17 20:20:55

For SpaceNut re #1158

Thanks very much for taking care of those duplicates and exceptions! Glad to see the refurbished ID's up to 5605!

I'll be starting tonight's run shortly.

Thanks for intervening to put the Solar Power topic back on track!

And thanks particularly for news of further proving of the rebuilt Subaru!

Best wishes for continued success in the gradual return to full usability!

It is possible (in fact likely) you've not had a chance to follow the link provided in the Antarctic Vet's post.

I think you find it interesting when you can free up 20 minutes or so ...

(th)

SpaceNut · 2021-05-17 20:38:17

I seemed to have missed it Antarctic Vet's posted link...

edit found it
http://newmars.com/forums/viewtopic.php … 16#p180016

tahanson43206 · 2021-05-18 00:31:24

Thanks for taking a look at the Penguin site! He seems to have spent a lot of time on it.

***
This run included a Username with posts, and the program said it did what it is supposed to do..

Edit at 8:37 local time ... The skip code reported finding a post, but it did not skip properly.

Three Usernames have posts of 1 each. I'll investigate and correct the program.

Unfortunately, this part of the code cannot be tested offline with my present capability. In past years I've had web sites running where testing could be performed, but those have long expired.

Starting Sequence for ID: 5623
Text from Clipboard in Step: 18 http://newmars.com/forums/userlist.php
Text from Clipboard in Step: 25 http://newmars.com/forums/userlist.php? … rch=Submit
Text from Clipboard in Step: 34 http://newmars.com/forums/profile.php?id=430739
Text from Clipboard in Step: 39 Bolamanb
At Step: 47 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:
Exception: Skipping this User
Text from Clipboard in Step: 56 TestID05623
Text from Clipboard in Step: 65 TestID05623@test.com
Text from Clipboard in Step: 75 http://newmars.com/forums/profile.php?s … &id=430739
Completed Sequence for ID: 5623
Starting Sequence for ID: 5623

That Username will appear in the "b" set after we move on to the "c"'s

Edit at 8:37 local time ...

The three Usernames I'll be investigating are 5621, 5623 and 5633.

(th)

Last edited by tahanson43206 (2021-05-18 06:41:43)

tahanson43206 · 2021-05-18 07:29:20

For SpaceNut re ID Recovery and specifically, regarding incorrect operation of skip feature today.

I ** think ** I can modify the current Verify script to show if users have posts. That information would be useful because it is possible to run the program in debug mode if a username with posts is entered as the subject of the run.

In the mean time, I'll look at the three Usernames with (single) posts and report the situation here.

Back at 9:33 local time ... It appears I am in luck!

TestID05621 TestID05623 TestID05633
Info
There are no posts by this user in this forum.
Go back

Please reset the posts count to zero when you complete the upgrade for the current set of ID's (5606-5633)

I'll suspend further runs until the issue of skips at posts is resolved.

In the mean time, I'll see if the Verify script can be modified to report the presence of posts.

Update at 12:36 local time:

The special Verify run was fruitful ... it found 8 Usernames with posts and one missing Username.

I'll use the list of Usernames with posts to make a special input file for debugging.

Update at 14:34 local time ... the incorrect behavior turns out to be understandable ... the subroutine intended to handle skips is empty.

I must have been distracted ... it happens ... that's what testing is for.

Update at 16:19 local time ... Modifications to correctly handle Skip situations are coming along. I'm planning to finish the new procedure tomorrow.

There will be no ID Recovery attempt this evening. 5606-5633 are ready for your review.

Please note that three ID's in that set have Post counts of 1 that can be reset to zero.

(th)

Last edited by tahanson43206 (2021-05-18 14:22:41)

SpaceNut · 2021-05-18 20:10:53

The evenings group of 5606-5633 have been dealt with and are now ready with the ones reset to zero as well.
cleared the b* user list of zero post spammers that had indicated posts resetting them to zero.
All others in the b's are valid users that are not banned.

tahanson43206 · 2021-05-18 20:43:01

For SpaceNut re #1163

Thanks for taking care of that little group!

Just fYI as we move out of the current hold ... When the program is working properly, it will skip ID's it finds that have posts. That means that someone (possibly an Administrator) will need to take a look at them. We were lucky that the three processed in error last night were ones you (or someone) had already dealt with by deleting posts. I don't want to depend on luck going forward, so have 8 example ID's to work with tomorrow to step through the code to be sure it works properly.

It ** is ** fun seeing the flow of messages after Mars_B4_Moon returned, and hope that the levels will continue.

Thanks for working with Louis to (try to) help him understand what we are attempting with the special space for Oldfart1939 to work without interruption. You've suggested a Wiki as a possible outcome of his work. I've suggested an article in a major publication. Either way, as he marshals his thoughts without interruption, we all have an opportunity to contribute suggestions, references, constructive criticism or praise, as might seem appropriate at the time.

***
For Oldfart1939 re feedback on your work ...

GW Johnson is limited in access to the forum right now, but hopefully he'll be back in the mix soon. His comments about your project would be interesting for me to read (for sure) and hopefully might be helpful to you as well.

I will drop a note to Noah tomorrow to mention your request for feedback. Noah may be (and probably is) deep in studies and exam preparation, but (I'm hoping) he might have a few minutes to spare.

(th)

tahanson43206 · 2021-05-19 07:15:54

For SpaceNut re ID Recovery ...

WBA is updated with three sets of corrections, and it is waiting for step mode verification ...

The three modes, Step, Loop and Run all have slightly different procedures to handle skips correctly.

Update at 12:31 local time ... Step mode appears to be working correctly.

At least one modification is needed for Loop mode and one for Run mode.

Will return to WBA later on today.

(th)

Last edited by tahanson43206 (2021-05-19 10:32:56)

tahanson43206 · 2021-05-19 17:18:00

For SpaceNut .... I thought you might be interested in this presentation, although I'm not sure the time would work for you. You're already so skilled at using Google you could probably teach the class, but I would imagine there might be some little detail that would by worth your time, if you can attend...

Getting Better Google Search Results
Wednesday, May 19, 2021, at 7:00 p.m.
Hewie Poplock, Central Florida Computer Society,
Sarasota Technology Users Group &
Member of the APCUG Speaker’s Bureau
The presentation will show various ways to get better results quicker. You will learn many tips & tricks to help in your Google search. There will also be numerous ways demonstrated to get information quickly without having to go to websites but get the answer in your search results.
Hewie Poplock, Central Florida Computer Society / Sarasota Technology Users Group
The excerpt above was provided by APCUG.
About the Presenter:
Hewie is a former APCUG Vice President and currently serves as the Windows SIG leader and APCUG Rep for the Central Florida Computer Society. He is also is a member of the Sarasota Technology Users Club board of directors. Hewie attended his first user group meeting armed with a yellow legal pad and 21 questions. Although only four of them got addressed at that first meeting, he knew that computer user groups were an invaluable source of help, information, and friendship. That was in 1982 at an Osborne User group. He has been active in the user group movement ever since. He likes to help computer users and give back much of what he has learned over the years. You might want to check out what Hewie is currently doing via his website Hewie.net. Hewie is a member of APCUG’s Speakers Bureau.
The excerpt above was provided by APCUG.

Zoom connection information:
Meeting Link: https://us02web.zoom.us/j/87699696636?p … 9Ga080Zz09
Meeting ID: 876 9969 6636
Passcode: 186545
Mobile: +13126266799,,87699696636#,,,,,,0#,,186545#
Voice: +1 (929) 205-6099
Show original message
https://groups.google.com/d/msgid/ccs-m … 0gmail.com

Edit 2021/05/20 ... here's a link to a YouTube copy of the presentation ... it will be available for a limited time

If you missed tonight’s wonderful presentation, the link to the recording is included at the bottom of this message. It will be active until next month’s presentation.
You will also find Hewie Popluck’s handout attached to this message and posted on our website, www.ccscmh.org.
Recording:
https://us02web.zoom.us/rec/share/weoWP … 1465930000

(th)

Last edited by tahanson43206 (2021-05-20 14:16:09)

tahanson43206 · 2021-05-19 19:15:33

For SpaceNut re ID Recovery ...

The run to 5700 completed normally.

All 8 of the Banned Usernames I found that had posts of 1 (one) but no actual posts were set to zero by the time the program got to them.

If you took care of them ahead of time, thanks for doing that.

***
I'll start the Verify run for the run to 5800 shortly.

Completed Sequence for ID: 5700

Total Command Lines found: 110
Total input Lines in script: 308
Total input Lines in data: 2754
Total input Lines in data: 2754
Number of ID's processed: 67
Starting Number: 5634
Last Number of Run: 5700
Summary for Web Automation Report for 05-19-2021 at 21:18:38
Average time of Loop from Main form: 00:03:04
9 Date Exceptions were recorded.
Total time of Processing: 03:29:07
Total time Program was Active: 03:49:15
Exceptions were reported. Run ended normally

(th)

Last edited by tahanson43206 (2021-05-19 19:21:22)

SpaceNut · 2021-05-19 19:55:28

I had done a preempted strike last night to reduce the stops in the b's by checking the banned with low post count and did do the reset if they did not have any posts being valid count.

To google search in this day and age key word search some times its the order for the search engine that pulls the needle out of the haystack...

This is also true with the web pages meta data that starts out at the top of the page you view.

Several duplicates in the banned table prior to 5647 but are good through to 5700 so will get them cleared and ready...

tahanson43206 · 2021-05-19 20:51:45

For SpaceNut re #1168

Thanks for taking the time to clear the B's ahead of the program!

Thanks too for your notes about using Google search!

Finally, thanks for taking care of the recovering ID's through 5700.

I ran the Verify ahead of tonight's run and found:

Starting Sequence for ID: 69
Text from Clipboard in Step: 16 http://newmars.com/forums/userlist.php
Text from Clipboard in Step: 23 http://newmars.com/forums/userlist.php? … rch=Submit
Query reported Not Found: Cesarjfysg
Completed Sequence for ID: 69

I'll remove that Username from the run and start it up shortly.

Edit next day at 8:11 local time ... there's a new mystery to study ... I suspect the new code to skip Usernames with posts is involved.

The program completed 5781 normally, and then skipped to 5800 and reported a normal end of job.

I'll investigate to (try to) find out what happened.

(th)

Last edited by tahanson43206 (2021-05-20 06:12:54)

tahanson43206 · 2021-05-20 08:13:44

For SpaceNut ...

The new code to skip Usernames with posts is running ...

The results I've examined so far are encouraging. The run processed 100 Usernames, but only created 82 TestID's.

This implies 18 Usernames had posts ...

Here is a partial list of skipped Usernames:

5782 next. 2586 Usernames remain in file.
copcoteraelena is the first Username in the data file for tonight's run.
18 ID's were not used. 100 Usernames were processed.
CooperTrou was the last username processed as 5781
Sample of Skipped Usernames:
Starting Sequence for ID: 5731
Text from Clipboard in Step: 39 caicai2017
At Step: 47 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:
Exception: Skipping this User
Skip Request for ID: 5731
Starting Sequence for ID: 5731
Text from Clipboard in Step: 39 calmanbill432
At Step: 47 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:
Exception: Skipping this User
Text from Clipboard in Step: 39 CalvinCast
At Step: 47 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:
Exception: Skipping this User
Text from Clipboard in Step: 39 CanadafinH
At Step: 47 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:
Exception: Skipping this User
Text from Clipboard in Step: 39 cankq71B27
At Step: 47 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
User has posts greater than zero:
Exception: Skipping this User

I'm planning to investigate further.

Edit at 10:26 local time after looking at the Bans table ...

<< Big Grin >> Have you been busy? The "b" collection now has 8 pages of Usernames!

I was expecting zero! However! If you ** have ** been adding qualifying Usernames, then the program can just go back over the "b"'s later.

I'll set up a short run to try to bring the total of TestID's up to 5800.

Edit at 12:45 local time ...

The batch of Usernames in the queue for catch up to 5800 included plenty of Posts.

This turned out to be helpful, because I was able watch the Loop in particular as it processed an ID and stopped because it found posts. I am satisfied that ** that ** part of the operation is working correctly.

On the other hand, while the Run operation seems to be working, I am puzzled by the run not proceeding all the way to 5800. The code in that section seems to be skipping correctly, but the job ended before the goal was reached.

A likely explanation is that the program is relying upon a variable (a counter) that I'm forgetting. The visible display is supposed to be reflecting the runtime counter, but (apparently) it is not.

In the mean time, I'll run the Verify procedure for tonight's dash to 5900.

Edit at 13:31 local time .... Well! That was easy! The original code was written with a For-Next loop. That made sense at the time, because skipping records was not part of the vision at the start of this initiative.

A While loop would be a better fit, so I'll implement that for tonight's run. The number of Usernames will necessarily be greater than the number of ID's created, if Usernames are skipped. The While format allows for that flexibility. The run will continue until the Current ID matches the Stop number.

(th)

Last edited by tahanson43206 (2021-05-20 11:34:53)

tahanson43206 · 2021-05-20 14:18:58

For SpaceNut ... the presentation on Google Search was recorded ... I've updated the original announcement to include a link to the YouTube.

It will be available for a limited time.

(th)

SpaceNut · 2021-05-20 17:37:11

Years ago there was an advance search function that google had and I went search my old favorites from the xp machines drive and here it is.

https://www.google.com/advanced_search

it will allow for a site search to narrow down content returned

tahanson43206 · 2021-05-20 17:47:27

For SpaceNut re #1172

SearchTerm:GoogleSearch Advanced featue
SearchTerm:Search Google advanced feature

(th)

SpaceNut · 2021-05-20 18:44:19

just finished clearing the 5701 to 5800 with no duplicates....

as I finished the next to second to the last I saw 5801 pop up with a duplicate but then on the next it disappeared.

now to clear the past duplicates….

tahanson43206 · 2021-05-20 19:01:33

For SpaceNut re #1174

Thanks for telling me about being online with WBA running !!!

I watched the first several Usernames like a hawk this evening, because I new they all had posts ... Both the Loop mode and Run modes handled them smoothly. This evening the program is running with the new While loop instead of a For-Next, so I'm expecting it to finish the run all the way to 5900, although there are no guarantees ... an Internet bogdown could exceed the built-in wait times.

I have a concept in mind for a "patient retry loop", but since that would be difficult, I'm tempted to just let the existing version run if there aren't too many time out halts.

The main thing I was going for here was an increased level of trust the program can be allowed to run on it's own, because it has so many checks built in.

(th)

New Mars Forums

Announcement

#1151 2021-05-15 20:22:51

Re: Housekeeping

#1152 2021-05-16 06:08:26

Re: Housekeeping

#1153 2021-05-16 09:02:10

Re: Housekeeping

#1154 2021-05-16 09:38:10

Re: Housekeeping

#1155 2021-05-16 09:49:31

Re: Housekeeping

#1156 2021-05-16 09:55:34

Re: Housekeeping

#1157 2021-05-17 05:35:10

Re: Housekeeping

#1158 2021-05-17 16:14:19

Re: Housekeeping

#1159 2021-05-17 20:20:55

Re: Housekeeping

#1160 2021-05-17 20:38:17

Re: Housekeeping

#1161 2021-05-18 00:31:24

Re: Housekeeping

#1162 2021-05-18 07:29:20

Re: Housekeeping

#1163 2021-05-18 20:10:53

Re: Housekeeping

#1164 2021-05-18 20:43:01

Re: Housekeeping

#1165 2021-05-19 07:15:54

Re: Housekeeping

#1166 2021-05-19 17:18:00

Re: Housekeeping

#1167 2021-05-19 19:15:33

Re: Housekeeping

#1168 2021-05-19 19:55:28

Re: Housekeeping

#1169 2021-05-19 20:51:45

Re: Housekeeping

#1170 2021-05-20 08:13:44

Re: Housekeeping

#1171 2021-05-20 14:18:58

Re: Housekeeping

#1172 2021-05-20 17:37:11

Re: Housekeeping

#1173 2021-05-20 17:47:27

Re: Housekeeping

#1174 2021-05-20 18:44:19

Re: Housekeeping

#1175 2021-05-20 19:01:33

Re: Housekeeping

Board footer