New Mars Forums

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Azure Test Environment for Software upgrades

Starting with assistance by kbd512, and finishing today with assistance by ChatGPT4, we now have a working Virtual Machine running at Microsoft.

You will be able to access the system  for the duration of the free trial period.

At this moment (as post was created) , there is nothing in the database, and all you can run is the default web pages.

Update: the database has been loaded with a FluxBB forum backup as of late 2023.

The database has been shown to respond to both read and write requests.

On 2024/04/11, if all goes well, a copy of phpBB 3.3.11 will be installed for testing.

On 2024/04/16, a partially functional update of FluxBB 1.5.11 is working:

http://40.75.112.55/forums/FluxBB/index.php

All existing NewMars members should be able to log into the test site. Full functionality is not yet available, but quite a bit is working.
In particular, all usernames and passwords seem to be fully usable.

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

This post is reserved for links that our Admins and members can use to test the work being done to upgrade our FluxBB 1.5.11 forum.

A test phpBB3 site is up: http://40.75.112.55/phpBB3/index.php

Update 2024/04/11: http://40.75.112.55/forums/UpdateEdited.php?id=3

The script at the link above shows that system is able to update the database.

http://40.75.112.55/index.html

http://40.75.112.55/phpinfo.php

Update 2024/03/28 ... the database is now installed and a test script is available:
http://40.75.112.55/forums/listtopic.php?id=4110

The highest topic number in the test data appears to be 4110. That number corresponds to a topic in the production database.

The index page shows the Apache2 installation running.

The PHP link shows the current PHP software running.

If you look down the page you should find: REMOTE_ADDR

That ** should ** be your IP address.

Update 20240817 - Microsoft Azure is implementing Two Factor Authentication for all users as of September, 2024.

In order to be able to comply with this requirement, it is necessary for newmarsmember@gmail.com to join/subscribe to Microsoft Extra ID services.

the username is (firstpart)@NewMarsforums.onmicrosoft.com

Next steps:

bookmark admin.microsoft.com

sign in with the username above

Manage your subscription in the admin center.

<> Add users, assign licenses and more
<> Invite your team to use Microsoft Extra ID Trial
<> Manage payment and billing details

Home > Multifactor authentication > Users

Display Name "6space6"
User principal name: (firstpart)@NewMarsforums.onmicrosoft.com
Identities: NewMarsforums.onmicrosoft.com

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

The database backup appears to have been copied from Dropbox to the Azure Virtual Machine.  I've launched the restore process.

This is going to take a while because the database backup is 540+ megabytes.

If the restore succeeds, I'll install the little listtopic.php file I made while waiting for the forum software.  It does not depend upon anything in the forum software to run, but it does exercise the topics table and the posts table, so we'll know pretty quickly if all this activity worked.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

Back in the 2008 to 2010 era Josh Cryer did the same with paying for a server and domain registration before James came to the rescue and got the forum back into the society's servers.

The index came up but the php forum did not.

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re #4

Thanks for this valuable history reminder!

I attempted to restore the database to the Azure Virtual machine but I became impatient and stopped the run before it finished.  It did restore the posts table, which is the largest by byte count, but it still had several to go, including topics.  I dropped the database, recreated it and restarted the restore to let it run all night if necessary. 

For the record, here is a link to a post with the record counts that I'll be looking for if the restore succeeds:

https://newmars.com/forums/viewtopic.ph … 63#p217263

By the way SpaceNut ... have you tested the two links I set up to point to the test system?

I'd appreciate your confirming the links work for you.  This is a lot of effort just to give you a chance to help test the new software with all of kbd512's updates.

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

I have now tried twice to insert a backup of the forum database into the MySQL system in the Azure Virtual Machine.

Each time the run stopped at the same place.  I am guessing that there is a resource limit of some kind that is preventing insertion of the remaining tables.

The solution (I gather) is to backup the individual tables that have not been inserted, and insert them individually.

Here is a list of the tables (13 out of 19) that were inserted, along with display of row counts for the last 3.

+----------------------------+
| fluxbb_bans                |
| fluxbb_categories          |
| fluxbb_censoring           |
| fluxbb_config              |
| fluxbb_forum_perms         |
| fluxbb_forum_subscriptions |
| fluxbb_forums              |
| fluxbb_groups              |
| fluxbb_online              |
| fluxbb_posts               |
| fluxbb_reports             |
| fluxbb_search_cache        |
| fluxbb_search_matches      |
+----------------------------+
13 rows in set (0.01 sec)

mysql> select count(*) from fluxbb_reports;
+----------+
| count(*) |
+----------+
|       10 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_search_cache;
+----------+
| count(*) |
+----------+
|        7 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_search_matches;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

Here is a report of the successful insertion of 12 tables of 19 that are needed.

mysql> select count(*) from categories;
ERROR 1146 (42S02): Table 'fluxbb_newmars.categories' doesn't exist
mysql> select count(*) from fluxbb_categories;
+----------+
| count(*) |
+----------+
|        7 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_censoring;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_config;
+----------+
| count(*) |
+----------+
|       81 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_forum_perms;
+----------+
| count(*) |
+----------+
|        6 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_forums;
+----------+
| count(*) |
+----------+
|       26 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_forum_subscriptions;
+----------+
| count(*) |
+----------+
|      117 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_groups;
+----------+
| count(*) |
+----------+
|        5 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_posts;
+----------+
| count(*) |
+----------+
|   210281 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_reports;
+----------+
| count(*) |
+----------+
|       10 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from fluxbb_search_cache;
+----------+
| count(*) |
+----------+
|        7 |
+----------+
1 row in set (0.00 sec)

mysql> exit
Bye
azureuser@FluxBBtest:~$ exit
logout
Connection to 40.75.112.55 closed.

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

There will be a delay before the database can be fully loaded.  The table that ended the automated process was fluxbb_search_matches.

That table has a high row count: -15,493,240 per Count(*)

I'm not even sure all those records are needed for testing code updates that kbd512 is working on.

The table that is definitely needed is topics.

My (tentative) plan is to backup the missing tables individually, and restore them on Azure individually.

Update a bit later ... the missing tables have all been backed up to individual files. 

Next steps include:
1) compression
2) delivery to Dropbox
3) Copy from Dropbox to Azure
4) Unzip at Azure
5) Restore to database

What remains unclear is whether the 15 million row table is going to fit into the Azure MySQL database.  Since this is a free development environment, there may be some limits, and 15 million rows could certainly be too many. An alternative is to simple create the table using the create function kbd512 has already provided.

The rest of the tables are all of a much more reasonable size.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

I am able to have the links now both open and am able to see the content.

Now to study.

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re #9

Thanks for taking the time to test the two links!

I am in the process of (attempting) adding the five missing tables to the MySQL database in the Azure VM.  The first step was to write the backups for the five tables, and that seemed to go well. The next step was to compress the files for delivery to Dropbox.  I ** just ** finished uploading the files to Dropbox. I collected the links to the files and saved them in a document.  The ** next ** step is to import those file files into the Azure VM, and then it will be necessary to unzip the files so they can be inserted into the database.  The two important tables are Topics and Users, but I'll import all five if the system will allow it.

Meanwhile, hopefully, kbd512 is putting the finishing touches on the forum code.  We still have 18 days or so to run tests using the Azure system at no charge, which is the right amount for my budget.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

How to import an excel file in to a MySQL database

There may also be code to do this as well when doing single post content as well to fill all of the holes back up.

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re continuing restore of individual tables to forum database on Azure VM...

There are five tables to restore as individuals ... two are finished. These are the most important.

I'll attempt to add the remaining three files tomorrow.  We have enough now to be able to test the forum software when kbd512 has it ready.

The remaining three tables can be  created as empty for testing. They are:

search_matches (already created as empty)
search_words << this table will start rebuilding the words content as soon as testing begins.
topic_subscriptions << not needed for testing - a subscription can be added and then exercised for testing

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

Offer to assist with configuration of free Azure web site....

Having just come through the process of threading the needle (or running the gauntlet depending upon point of view) to set up a temporary web site in the Azure environment, I am happy to provide guidance to anyone who would like to replicate the experience while it is still fresh in my mind, and I have the answers to the nearly infinite number of questions that a person must deal with.

I'd like to give major credit to kbd512, who coached me through the first configuration of a test Virtual Machine. That machine ended up deleted because I did not know how to fill it with software.  I took another run at the problem, with ChatGPT4 as my guide, and ** this ** time we were successful.

To set up a free demo account, you need a credit card and be willing to put it at risk.  You are given $200 of play money and so far, I have found Microsoft to be remarkably generous in how it charges for the time I've used setting up the account. After spending a ** lot ** of time setting up the web site, Microsoft was showing a balance of $196 left in the play money account.

In order to facilitate someone taking me up on this offer, I am (tentatively) planning to  collect all the questions and answers into a form that I can use to assist someone who would like to try this.

You only get one free try (ie, 30 days and $200 of play money) per email address, but I suppose you could use more than one email address.

To look at the options available, enter portal.azure.com into the browser.

I observe that Microsoft converts that URL to another one after it finds that I have set up an account with them.

The replacement page shows: login.microsoftonline.com/oauth2/v2.0/authorize?redirect_url >>>and this section points to portal.azure.com ...

The result is that I can no longer see what a person sees when first opening the portal.azure.com page.

However, my recollection is that the process of setting up an account is fairly straight forward.

I will now be able to report what happens after I select the account I've created from the list provided by the web page I'm seeing now:

so! Click and let's see what happens:

Oh! Of course ... I'm asked for the password ....

I chose a complicated password, which I can ** just barely ** remember ... >> Enter

This question pops up because I (so far) refuse to say yes: Stay signed in?  ... in this case I will say no.

Ok! after going through several web pages, we end up at:

portal.azure.com/#home

This page contains the top level links to the temporary web site I've created (with help from kbd512 and ChatGPT4)...

At the top are the many Azure services available.

I assume this page must be seen by developers working for the largest companies on Earth, and not just by a single individual building a test system.

At the bottom are buttons offering Microsoft Learning modules.  I am steering away from those, valuable as they might be, because I have only a limited amount of time and energy.

More pertinent to this post is a list of links for the little forum database test web site:

Resources:
FluxBBtest is the name of the Virtual Machine ... I chose an Ubuntu 22.04 server with 1 Gb of Ram
FluxBBtest-nsg ... Network Security Group ...
Azure_subscription_1 .... this is where the play money status may be found
FluxBB_group ... I've forgotten what this is for but apparently you need one
FluxBBtest-ip .... This contains the IP address we are using for our test

Beneath that list is a row that contains:

Navigate

Subscriptions   Resource groups   All resources    Dashboard

That last item looks interesting.  I'll have to investigate it some time.

From this page it is possible to start and stop the virtual machine, and I've had to do that once when I tried to load a huge data table and the VM went off to lala land.

If anyone decides to venture down this path, please let me know while my memories of the experience are fresh, and the notes I've taken are still available.

Once the VM is configured, the developer gains access to it using a terminal window in Linux, and I presume a Command window in Windows.

During creation of the account, there is a step where a secure file is created so that communication with the terminal window is encrypted.

Connection to the VM via this pathway is simple and reliable (once the magic string is created).  I've had to log into the VM many times, because the connection only seems to last for a brief time.  I frequently shift away from the open session, and it seems that my connection to the VM can be lost if I'm away too long.

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

The new Azure test environment is full configured and working.

Here is a URL you can run to show the numbers of posts for any topic ID you enter:
http://40.75.112.55/forums/listtopic.php?id=4

The highest topic id in the test database is 4110:
http://40.75.112.55/forums/listtopic.php?id=4110

and here is a report, prepared by ChatGPT4 showing a high level view of the process.

There are a great many details that I will attempt to save in files in case anyone decides to replicate this adventure.

Microsoft offers a free 30 day trial of their Azure system, including $200 of play money.  You ** do ** have to take the risk of entering a credit card into the system during the activation process, so I understand not everyone is going to be willing to take that risk. 

Azure Virtual Machine Configuration for Web Application Deployment

Greetings, forum members!

I've embarked on an exciting journey to set up and configure an Azure Virtual Machine (VM) for hosting a web application, specifically for a forum project. With invaluable assistance from our webmaster, kbd512, and ChatGPT4, we navigated through Azure account setup, VM creation, software installations, and troubleshooting various challenges. Below, I outline the key steps and insights gained through this adventure.

1. Setting Up an Azure Account and Creating a Virtual Machine
- The journey began with setting up an Azure account, a straightforward process that grants access to Azure's comprehensive cloud services.
- We then proceeded to create a new Virtual Machine (VM) tailored for our web hosting needs, selecting appropriate resources (CPU, memory, storage) to balance performance and cost.

2. Installing Necessary Software on the VM
- Our VM setup focused on a LAMP stack (Linux, Apache, MySQL, PHP), essential for hosting dynamic web applications.
- Apache: Installed as our web server, configured to serve our site's content.
- MySQL: Set up as our database management system, crucial for storing and managing forum data.
- PHP: Integrated to process dynamic content, enabling interactive forum functionality.

3. Configuring the Database
- We configured MySQL with a secure setup, creating specific user accounts with necessary permissions for managing the forum's database.
- Special attention was given to ensuring secure password practices, especially highlighting the importance of handling special characters in passwords correctly within PHP scripts.

4. Troubleshooting and Debugging
- A significant portion of our journey involved troubleshooting various issues, from permission errors to database connection challenges.
- Notably, we addressed an "undefined variable" PHP warning and resolved a critical "access denied" error for our database user by correctly formatting the connection string in our PHP configuration file.

5. Optimization and Security Considerations
- Throughout the process, we emphasized best practices for security and performance, including regular updates, careful management of user permissions, and the use of secure, validated inputs to protect against SQL injection and other threats.

Key Takeaways
- Persistence and detailed attention to configuration and troubleshooting are paramount.
- The collaboration between developers, knowledgeable community members, and AI-assisted guidance (like ChatGPT4) can overcome complex technical challenges.
- Sharing knowledge and experiences enriches our entire community, fostering a collaborative and supportive environment for all members.

Special Thanks
I extend my heartfelt thanks to kbd512 and ChatGPT4 for their guidance, patience, and expertise throughout this project. This journey not only enhanced our forum's infrastructure but also deepened my understanding of web application deployment and cloud-based hosting.

I hope this report inspires and aids you in your own technology endeavors. Let's continue to learn, share, and grow together as a community!

Best regards,
[tahanson43206/Junior Moderator]

ChatGPT4 went a bit over the top but I decided to leave the work as is.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

Index of /forums
[ICO]    Name    Last modified    Size    Description
[PARENTDIR]    Parent Directory         -    
[   ]    listtopic.php    2024-03-28 17:59    972    
[   ]    listtopic.php.save    2024-03-28 17:56    6    
Apache/2.4.52 (Ubuntu) Server at 40.75.112.55 Port 80

How to Install FluxBB with Apache on Ubuntu Linux

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re #15...

Thanks for the link to the very nice documentation of the procedure to install FluxBB ...

Everything seems to match what was done so far, with the exception of setting up the Azure environment.

I will attempt to document that part of the procedure.

The next step is to install the forum software as modified and updated by kbd512.

Naturally that means we won't be downloading from the repository.  That version will not run with modern MySQL or PHP.

However, the guidance for configuring Apache2 to work with the forum software looks valuable and necessary, so thank you for that.

I'm hoping we may see a preliminary test set up by this weekend.

The caveat is that this is Good Friday, and Sunday is Easter, so a lot of folks will be focused on family and related activities.

(th)

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut ... the Azure test account still has $195 credit of the $200 starting number.

Microsoft is mighty generous ... I've been running the system pretty hard over the past  several days, with data uploads and installation, and Microsoft doesn't seem phased.  However, the 30 day clock is ticking, so I'm hoping we can get good use of the system in the remaining time we have.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

I found some more code examples for the php files for fluxbb coding style of text.
https://github.com/fluxbb/fluxbb

The newmars forum home page also shows with the forum/index.php in the address bar.

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re #18

Thanks for the link to the legacy repository of FluxBB 1.5.11.

I'm pretty sure that is where kbd512 found the code that he is updating to run on modern PHP  and MySQL.

You will need to update the legacy code if you decide to build a system on your development PC.

Regarding the test I'd like you to run on the Mars Society web site...

http://newmars.com/index.html

If you click on that link, you will get a 404 error.

I'd like to change the Apache demo web page so it looks and acts like the Mars Society web page for newmars.

Do you know how to do that?

I've asked kbd512 to provide a copy of the Mars Society web page, but with work and family and other obligations, he may not be able to get to the request.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

That is the root directing how to make use of the forum address and at that point we are on the word press home page.
Much like the copy it's been modified to keep others to keep hackers out.

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re #20

I would like to learn how to modify the web page to prevent unauthorized access.  If you know how to do that, please let me know.

It was important (to me at least) to be able to show that the Apache web server is running.  We now know that so it is time to lock down the index.html file.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

A quick bing search for "php index.html locked down methods" yields lots of information and I believe. that KBD512 has server access to that directory from James.
It is password protected and maybe JoshNh4H might have it as well as there was an article published there on using co2 and water to produce energy.

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut ... thanks for doing the lookup ...

Are you able to find anything that does NOT involve PHP?

The file I'm asking about is an Apache file: index.html.

That file is located in /var/www/html

The PHP system does NOT currently have an index.html file.

(th)

SpaceNut

Administrator

From: New Hampshire

Registered: 2004-07-22

Posts: 29,431

Re: Azure Test Environment for Software upgrades

how-do-i-change-the-default-index-page-in-apache

tahanson43206

Moderator

Registered: 2018-04-27

Posts: 19,408

Re: Azure Test Environment for Software upgrades

For SpaceNut re #24

Thanks for the link to the StackExchange discussion of the several ways to approach this!

Please take a look at the modified index.html and let me know if you want additional changes.

(th)