Housekeeping

kbd512 · 2021-08-09 19:28:23

I'll test the script with a limited range of Test IDs first, obviously, and then I'll PM the script to you for final approval before we run it on everything.

SpaceNut · 2021-08-09 20:19:45

yes the server can set the flood control and stop responding
and yes to a sequence
th has gotten them set through just under 14,000

I see that you walked through the first id above 10000

tahanson43206 · 2021-08-09 20:35:36

For kbd512

Glad to know you have Selenium!

I read about it and decided to create my own version.

There should be no need to overwhelm the server.

Just check for the server response before going on to the next step.

My "Smart Wait" command retries up to 60 times while waiting for a desired response from the server, with each retry set at 1 second.

The only times when "Smart Wait" times out are very rarely, when a packet gets lost.

By now I have multiple thousands of interactions recorded, and only once or twice has the Smart Wait timed out.

(th)

Last edited by tahanson43206 (2021-08-09 20:36:07)

kbd512 · 2021-08-09 21:25:46

tahanson43206,

That's fantastic. I'd like to see that some time.

I only have Katalon Recorder at home, but I will purchase Katalon Studio, if need be. It's been some time since I've used one of these tools in a corporate environment, mostly because I hand off testing to other groups after I write the code.

kbd512 · 2021-08-09 21:31:19

SpaceNut,

The first part of the Katalon Recorder script is something like this:

Command | Target | Value
open | http://newmars.com/forums/admin_bans.php
pause | 3000
store | 10002 | ii
while | ii<10005
type | name=form[username] | testid${ii}
click | name=find_ban
pause | 2000
click | link=Remove
pause | 2000
storeEval | ${ii}+1 | ii
endWhile

Yes, I know it can be improved with appropriate wait commands, but this is to roughly illustrate the first part of the script.

I assume this is what you had in mind.

I'll create the second part of the script tomorrow. My brain is fried from work and I only slept a few hours last night.

tahanson43206 · 2021-08-09 21:48:35

For kbd512 re post just ahead of this one (for some reason FluxBB won't show me previous) ...

I ** really ** appreciate your showing what a script for this tool looks like....

I wrote a from-scratch tool after spending a lot of time (and buying a couple of books) investigating what is/was available. Somehow I missed the tool you've described. However, the world is big, and the Internet is huge.

The script just finished a run over 100 Userid's from 13901 through 14000.

The script uses mouse commands to click on buttons by position, since I did not have available the clever commands shown in your script!

I'm using tabs to move between fields where a mouse click is more trouble than it's worth.

I use the clipboard to capture text from the screen, and compare it to what is expected.

it looks as though the script you showed may have similar capability.

Here is a copy of the log for the script for the Update function for one user:

Starting Sequence for ID: 13902
Text direct from Clipboard in Step: 15 http://newmars.com/forums/userlist.php
Text direct from Clipboard in Step: 34 katleenbrown440
At Step: 42 Username did not match Stop At value
Post-Date Function Start: Reading Clipboard #4
PerformFunction: Valid ID: match in:
Registered: 2019-05-03 (180.211.112.67)
Last post: Never
Last visit: 2019-05-03 05:24:29
Posts: 0 - Show al
Text direct from Clipboard in Step: 54 TestID13902
Text direct from Clipboard in Step: 63 TestID13902@test.com
Completed Sequence for ID: 13902

The "stop at" test is there to prevent the script from processing a TestID that is already completed.

The PerformFunction is there to check data ranges, and if the user has zero posts.

I capture the entire screen to the clipboard to perform that analysis.

I use the same method in the Verify scripts, where I want to be sure the names collected earlier are still in the Users table, and then to be sure that the names are actually Banned by SpaceNut.

The Update script has halt commands after every test for results, so that if the Internet throws a whammy at the program it will stop immediately.

Some time ago, I wrote to SpaceNut about the difference between Level 0 automation and Level 1 automation. The current version of WBA is solidly in Level 1. I'd like to take it further but don't have a need, since Level 1 is working and I have other things I want to work on.

Update for Web Automation Report for 08-09-2021 at 20:30:49
WebBrowserAutomation
Version 2021.6.9.1
Completed Sequence for ID: 14000
Total Command Lines found: 112
Total input Lines in script: 341
Total input Lines in data: 3353
Number of ID's processed: 100
Starting Number: 13901
Last Number of Run: 14000
Summary for Web Automation Report for 08-09-2021 at 23:32:48
Average time of Loop from Main form: 00:01:29
14 Date Exceptions were recorded.
Total time of Processing: 02:29:18
Total time Program was Active: 02:59:38

Thanks again for showing that script.

If there is documentation for that program I'll try to find it tomorrow.

(th)

kbd512 · 2021-08-09 23:10:08

tahanson43206,

I never knew that particular tool existed before a few hours ago. I did a brief Google search (every bit of 10 seconds if you include the download time), because I have been aware that such tools exist for some time. I used similar tools at work while I was still in college, but nothing so nice as Katalon. I had to write the test scripts in vi for ye olde test automation suite, and then the tool executed them. Unfortunately for you two, it's also been quite some time since I had to use browser automation for work. I looked at the samples and created the script. It's not very fancy at all, nor does it need to be. I'm sure there's documentation somewhere out there, but I just downloaded the sample code. That's all I used to create that. We can make it pretty tomorrow. Anyway, it's pretty cool that you created your own test automation tools, though. Props for that.

tahanson43206 · 2021-08-10 06:19:55

For all ...

It is ** really ** beneficial for the forum to have multiple workers addressing the needs of the group, and it is certainly good to have a variety of tools of varying levels of sophistication available to address those needs.

For the record, here is a partial list of commands available using WebBrowserAutomation:

Update for Web Automation Report for 08-10-2021 at 08:12:15
MousePositionTrainingUtility
Version 2021.7.27.1
Samples of Script Commands for WebBrowserAutomation program

<esc>,click.
<esc>,combine,control,a
<esc>,combine,control,c.
<esc>,combine,control,l.
<esc>,goto,520,183.
<esc>enter.
backspace
CompareClipboardSaved:1
CompareClipboardSaved:2
Date:yyyy/mm/dd
Date:yyyymmdd
GetDataPutClipboard:1
Number:5
PasteClipboard:1
PasteClipboard:3
PerformFunction:4
ReportClipboard
ReportClipboard:1
ReportClipboard:2
ReportClipboard:3
SearchClipboardZeroReportMatch:text
StoreClipboard:2
StoreClipboard:3
StoreClipboard:4
Tab:1
Text:Updated
Wait:1000
WaitForEventAddressMatch:http:AddressBarTextHere

***
For kbd512

As you may have noted from previous conversation in this topic, the previous version of FluxBB allowed multiple Bans of users to occur. The result is that the Bans table will contain debris left behind after your complete the automated finishing process for TestID's. Rather than take your time to deal with those, I'm pretty sure WBA can deal with the lookups needed to insure that the original username for a given Ban record no longer exists and thus can be deleted.

If you want to leave that debris, and if SpaceNut agrees, I'll be willing to see if I can clean it up.

***
For SpaceNut ... Verify runs for tonight's dash to 14100 are starting now.

(th)

tahanson43206 · 2021-08-10 07:11:57

For SpaceNut ....

Just out of curiosity, would you be willing to contact Mr. Burk to find out what the backup schedule for the forum database might be?

I'm a member of a database user group, and recently the lead forum editor reminded the group of a standard practice in the industry: 1-2-3

For every 1 database, there should be backups on 2 different media stored in 3 different locations.

I understand from the discussion there are other permutations of this basic idea.

In light of the disaster that occurred in the history of the forum, I would expect there are improved backup procedures in place.

it would be good to know what backups are being made and how far back they go.

For obvious reasons this information should ** not ** be published on the open forum.

It would be sufficient if you were to confirm that you are satisifed with whatever arrangements are in place.

Edit #1: As often happens, I discover that my memory of something is transposed or is some other way different from the original...

That happened with the quotation I ** thought ** I was paraphrasing correctly ... in actual fact, the original reads:

The article talks about the 3-2-1 backup strategy and then some more modern evolutions of this. If you don't know, 3-2-1 means 3 copies of your data on 2 types of media with 1 copy offsite. That's a strategy I've followed for years in enterprise environments. We usually had a local current copy of databases on disk, a second copy on tape, and multiple tapes offsite.

The original source is available upon request.

(th)

Last edited by tahanson43206 (2021-08-10 09:55:31)

tahanson43206 · 2021-08-10 07:25:48

For SpaceNut re ID Recovery ...

Preparation for tonight's run to 14100 is underway ..,

Verify Users Table completed with no alerts ... 110 Usernames will now be check against the Bans table.

All 110 Usernames tested are in the Bans table.

All clear for tonight's update run.

Last edited by tahanson43206 (2021-08-10 08:44:12)

tahanson43206 · 2021-08-10 10:43:16

For kbd512 ...

Neat to see the company is related (somehow) to Vietnam!

Why Vietnam?
All outsourcing is not the same.
At KMS, we stand for quality and integrity. As a trusted offshore partner, we have access to Vietnam’s top talent so that we can hire the best both locally and globally.

In looking at the web site I didn't see anything hinting that free trial version of the software are available.

If I had run across a web site like that in my earlier investigation, I would not have considered it because it appears to be a commercial site.

For those who might be interested in comparing options for their projects, please try to make time to find out if what is offered on this site is first or second tier automation.

Level 0 automation is brute force, like a lawn mower. That type of automation is ** all ** that we humans had for many years. In more recent times, event driven technologies have allowed for implementation of level 1 automation in many platforms, and there are a few commercial products where level 2 automation appears to be in operation.

I rate automation based upon the ability of the software to cope with events that spoil progress.

Level 0 has to be watched like a hawk, because Administrator privileges can allow the runaway software to destroy an entire database in minutes. The WebBrowserAutomation tool has many self-checks built in to prevent departure from the desired path when the Internet throws up conditions not planned for or expected.

(th)

Last edited by tahanson43206 (2021-08-10 10:44:29)

SpaceNut · 2021-08-10 21:08:08

Just lost all of my email history and address book which really sucks....

tahanson43206 · 2021-08-11 05:39:49

for SpaceNut re #1462

Hopefully you can recover from your monthly backups.

Edit: I just realized you may be talking about your phone? Is there a way to backup data from a phone? I do understand that data can be transferred from one phone to another, because I've seen it done at the time of sale in the local wireless store. However, I've never had occasion to think about backup of phone data until now.

Edit#2: Follow up ...

How to Back Up Your Cell Phone
Connect your device to your Wi-Fi network.
Open your phone's Settings app.
Under Personal, tap “Backup & Reset.”
Tap “Back Up My Data” and switch it on.

That seems simple enough. Presumably the smart phone OS makes a storage location somewhere on the WiFi network, or perhaps the vendor provides cloud storage as a benefit. That seems more likely to me.

Here's another snippet Google found:

Settings and apps · Open your smartphone's Settings app. · Scroll down to Google and tap on it. · Tap on Backup. · Toggle on the Back up to Google Drive switch and ...

That one makes even more sense ... It does require the customer has set up Google Drive, and I would imagine it is not too hard to do.

WBA reached 14,100 overnight.

Preparations for a dash to 14,200 will start shortly.

(th)

Last edited by tahanson43206 (2021-08-11 06:01:29)

tahanson43206 · 2021-08-11 06:41:27

For SpaceNut re ID Recovery ... this post includes wrap up for run to 14,100 and preparation for 14,200

Completed Sequence for ID: 14100
Total Command Lines found: 112
Total input Lines in script: 341
Total input Lines in data: 3253

Number of ID's processed: 100
Starting Number: 14001
Last Number of Run: 14100
Summary for Web Automation Report for 08-11-2021 at 08:33:59
Average time of Loop from Main form: 00:01:29
10 Date Exceptions were recorded.
Total time of Processing: 02:30:02
Total time Program was Active: 11:57:46

Verify in users table will start shortly.

The verify run finished but results require study.

OK ... yesterday's input file was fed into the Verify run.

While that was a (modest) waste of the Server's time, no harm was done since this is a lookup-only script.

The ** good ** news is that the Verify run showed that ** every ** username updated yesterday is gone.

A rerun of the Verify script with today's input file showed all 110 usernames are ** in ** the users table.

Verify Bans table will start shortly...

The verify Bans script finished: All 110 candidates are in the Bans table.

100 of those will be fed into tonight's update run.

For the record, here is how the Bans lookup script appears in memory after input and validation:

Command Code stored at Array line: 1 W number is: 5000
Escape sequence stored at Array line: 2 <>
Command Code stored at Array line: 3 X
Command Code stored at Array line: 4 W number is: 1000
Escape sequence stored at Array line: 5 <>
Command Code stored at Array line: 6 W number is: 1000
Command Code stored at Array line: 7 W number is: 0
Command Code stored at Array line: 8 WE Text is: newmars.com/forums/userlist.php
Escape sequence stored at Array line: 9 <>
Command Code stored at Array line: 10 W number is: 1000
Escape sequence stored at Array line: 11 <>
Escape sequence stored at Array line: 12 <>
Escape sequence stored at Array line: 13 <>
Command Code stored at Array line: 14 G from Row: 1
Command Code stored at Array line: 15 RC at Row # 1
Command Code stored at Array line: 16 PC from Row: 1
Escape sequence stored at Array line: 17 <>
Command Code stored at Array line: 18 W number is: 0
Command Code stored at Array line: 19 WE Text is: http://newmars.com/forums/userlist.php?username=
Escape sequence stored at Array line: 20 <>
Command Code stored at Array line: 21 T number is: 1
Escape sequence stored at Array line: 22 <>
Command Code stored at Array line: 23 W number is: 1000
Escape sequence stored at Array line: 24 <>
Command Code stored at Array line: 25 SC at Row # 0
Command Code stored at Array line: 26 SZ
End of file: C:\DataDell\WebBrowserAutomation\20210705ScriptUsernamesAREinBansTable.txt

(th)

Last edited by tahanson43206 (2021-08-11 10:03:05)

kbd512 · 2021-08-11 09:15:47

SpaceNut and tahanson43206,

I felt too nauseated after dinner to do much of anything afterwards last night, but I'll finish the script today. We can run it this weekend.

tahanson43206 · 2021-08-11 09:58:21

For kbd512 re #1485

Sorry to hear of this setback! Sure hope whatever it was was temporary, and that you've recovered fully!

***
Looking forward to a test run this weekend! I estimate each transaction will take on the order of 23-25 seconds, if you include verify steps in the procedure.

Today's Verify Bans run (for example) took an average of 24 seconds over 110 lookups.

You'll be doing posting as well as lookups, so the time required might be greater.

it will certainly be interesting to see how the tool you have chosen performs!

(th)

SpaceNut · 2021-08-11 17:55:13

No problem kbd512 hope you are feeling better.

I got lucky for not setting up the program to delete the server location after down loading to the machine for local use.

Got all of the information back on one of my email accounts.
Now to figure out what I can do for the next one.

Have determined that the family account did not fair so well as its all gone...

I did copy post 151 back into the main topic for crew of 17 where I though it fit and it would seem that more of the same needs to be done with some of the other to get a more complete lay of thoughts.

I also if you check the newmarsmember address you will see I sent the email on back up cycle question to the web master.

tahanson43206 · 2021-08-11 21:21:11

For SpaceNut re 1467

Dlad to hear you recovered some of your data!

If your system can write DVD's (and I realize new ones cannot) you can save data on DVD.

Thanks for helping OF1939 with the 17 Person Expedition!

Thanks for your follow up regarding backups. The work we are doing is important and needed, but a backup plan is a good idea even when professionals are involved.

***
The Internet was ** really ** heaving with a combination of large swells and large waves this evening.

I've only seen this message once before:

This site can’t be reached

That means not only was a packet lost, but the entire online session was disconnected.

Good Grief!

WBA only completed 3 ID's when that happened.

I'll start up the try for 14200 shortly.

Update at 12:09 ... The swells and chop are too great for this little vessel.

I watched WBA struggle to deal with delays that were within it's capability, and then give up when swamped.

I'll try again tomorrow.

(th)

Last edited by tahanson43206 (2021-08-11 22:11:05)

kbd512 · 2021-08-12 02:06:07

SpaceNut and tahanson43206,

This Katalon Recorder script at least appears to function correctly:

Command | Target | Value
open | http://newmars.com/forums/admin_bans.php
waitForPageToLoad
store | 10002 | ii
while | ii<10005
type | name=form[username] | testid${ii}
click | name=find_ban
waitForPageToLoad
click | link=Remove
waitForPageToLoad
storeEval | ${ii}+1 | ii
endWhile | undefined

I only tested it with a literal handful of Test IDs, but it passed. I also double-checked to ensure that the Test IDs were "un-banned".

tahanson43206,

If there's more work to finalize this process, then perhaps we should see if we can use the tool to generate a more comprehensive script that performs whatever actions you're taking. I have an old machine with a fast internet connection, but don't know what you guys have to work with. On my machine it appears to take 2 to 3 seconds to "un-ban" each Test ID. If both SpaceNut and I run the script on different ID ranges over this weekend, then it should un-ban all Test IDs in less than 8 hours. I need to make and install a door frame this weekend and pick up my car from the mechanic, so I can run it and check on progress every hour or so.

We can run also create and run a check script to see that all Test IDs have been cleared. That should execute a good bit faster since the search won't yield any results if the first script worked properly.

The Katalon Recorder is free and works with Firefox and Chrome. Katalon Studio is the commercial product, but it can also be downloaded and used for free for non-commercial purposes.

tahanson43206 · 2021-08-12 05:29:48

for kbd512 re 1469

Hoping you're feeling better!

Thanks for the good news of a successful test, and best wishes for success with the full run up to 14107.

As a reminder, SpaceNut is setting the pace by manually inspecting the ID's. This cannot be automated.

The process I am carrying out is proven over many thousands of usernames, and is working well.

The finishing process is where we needed help, and it sure looks as though you've worked out a reliable solution.

I like the WaitForPageToLoad function! That is roughly equivalent to my Smart Wait.

I will follow up to see if I can find the free trial opportunity.

SpaceNut ... I've encountered a new wrinkle in Internet behavior recently.

Last night a web site offering a new product refused to accept a "free" email.

That is all I have, so I cannot get into the site. I interpret this as an effective way to cut down on spam.

kbd512 ... the web site that offers the product you found also refused my email, but it didn't explain why.

By any chance, did you use a corporate email account to download the free sample?

For all ... We will be working this batch of ID's for the next 30+ days, at the rate of 100 per day, as long as SpaceNut finds time to put them into the hopper.

For kbd512 ... the finishing script can be run at your convenience, such as once a week to finish 700 ID's.

An alternative is to see if SpaceNut can run the script.

(th)

tahanson43206 · 2021-08-12 10:02:37

For SpaceNut re ID Recovery ...

The run to 14,200 completed without incident after the Internet squalls cleared out.

I'll start verify runs for tonight now...

Update at 12:57 local time ... 110 candidates with Zero Posts were checked against the users table.

Larryvet was found NOT present. That username will be removed from the list of candidates.

Remaining candidates will now be verified to be present in the Bans table. Only those in the Bans table will be accepted for tonight's update run to 14,300.

Update at 15:34 local time ... all 110 candidates were found in the Bans table.

100 of these will be turned over to WBA for tonight's run to 14,300.

(th)

Last edited by tahanson43206 (2021-08-12 13:25:51)

SpaceNut · 2021-08-12 19:46:08

It was good to come home and see James' thoughts on the backup and for the coming conference.

Will look at bans to add to the table for conversion.

We have made great progress in the conversions thus far and taking even the same amount of time to accomplish the remaining is not going to change how we are doing with discussions or in recruitment of new members or for out reach to see if the old members would be coming back.

Placed another 100 into the banned from the M's

tahanson43206 · 2021-08-13 05:30:54

For SpaceNut re #1472

Thanks for your perspective on the state of the forum! I agree that it seems healthy, and that existing members are helping to keep discussions moving.

Thanks for taking time to set up pins for WBA to tackle! (mixing bowling with football)

Last night's run was flawless ...

Completed Sequence for ID: 14300
Total Command Lines found: 112
Total input Lines in script: 341
Total input Lines in data: 3052
Number of ID's processed: 100
Starting Number: 14201
Last Number of Run: 14300
Summary for Web Automation Report for 08-13-2021 at 07:22:18
Average time of Loop from Main form: 00:01:29
13 Date Exceptions were recorded.
Total time of Processing: 02:29:46
Total time Program was Active: 10:41:24

Verify runs for tonight's update will start shortly.

All 110 candidates were found in the users table.

Proceeding to Verify in Bans table ...

Update at 13:13 local time ... all 110 candidates showed up in the Bans table ..

100 will be fed to WBA for tonight's run to 14400.

Update at midnight local time ... tonight's run just ended.

(th)

Last edited by tahanson43206 (2021-08-13 22:04:25)

kbd512 · 2021-08-14 06:17:04

tahanson43206,

I'm going to run the script today over whatever ID range you or SpaceNut specify.

I'll monitor it for a while, maybe 15 to 30 minutes, then let it run until it completes and check back later.

If you want me to start with that ID range I was using, then I'll run the script for that ID range.

I'm not sure why Katalon's website wouldn't accept your E-mail, but yes, I think I entered my work E-mail.

Anyway, let me know what you want and I'll run the script.

tahanson43206 · 2021-08-14 07:01:27

For kbd512 re #1474

Thanks for the update on the ID finishing script.

This is SpaceNut's project, so I'll stand by to see what he decides. The only constant I can offer is that SpaceNut has been supplying input at the rate of 100 ID's per day. There is a bit of a backlog, from 10004-14400, and we have 3200+ ID's to go in the current batch.

There are several thousand more that are likely to be spammers, but they'll take a bit more inspection by SpaceNut before they can be processed.

It is really ** good ** to have another person pulling an oar.

(th)

New Mars Forums

Announcement

#1451 2021-08-09 19:28:23

Re: Housekeeping

#1452 2021-08-09 20:19:45

Re: Housekeeping

#1453 2021-08-09 20:35:36

Re: Housekeeping

#1454 2021-08-09 21:25:46

Re: Housekeeping

#1455 2021-08-09 21:31:19

Re: Housekeeping

#1456 2021-08-09 21:48:35

Re: Housekeeping

#1457 2021-08-09 23:10:08

Re: Housekeeping

#1458 2021-08-10 06:19:55

Re: Housekeeping

#1459 2021-08-10 07:11:57

Re: Housekeeping

#1460 2021-08-10 07:25:48

Re: Housekeeping

#1461 2021-08-10 10:43:16

Re: Housekeeping

#1462 2021-08-10 21:08:08

Re: Housekeeping

#1463 2021-08-11 05:39:49

Re: Housekeeping

#1464 2021-08-11 06:41:27

Re: Housekeeping

#1465 2021-08-11 09:15:47

Re: Housekeeping

#1466 2021-08-11 09:58:21

Re: Housekeeping

#1467 2021-08-11 17:55:13

Re: Housekeeping

#1468 2021-08-11 21:21:11

Re: Housekeeping

#1469 2021-08-12 02:06:07

Re: Housekeeping

#1470 2021-08-12 05:29:48

Re: Housekeeping

#1471 2021-08-12 10:02:37

Re: Housekeeping

#1472 2021-08-12 19:46:08

Re: Housekeeping

#1473 2021-08-13 05:30:54

Re: Housekeeping

#1474 2021-08-14 06:17:04

Re: Housekeeping

#1475 2021-08-14 07:01:27

Re: Housekeeping

Board footer