Updating MX to new version: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
m
Text replacement - "[[MXDiags" to "[[MXdiags"
m (PASTE)
m (Text replacement - "[[MXDiags" to "[[MXdiags")
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=Who this article is for=
=Who this article is for=


Please be aware that if you want to move from Cumulus 1 to MX, you should read [[Moving_from_Cumulus_1_to_MX]] article instead.
Please be aware that if you want to move from Cumulus 1 to MX, you should read [[Migrating_from_Cumulus_1_to_MX]] article instead.


This article is for those who already use MX, and so are comfortable with basic MX installation and running.
This article is for those who already use MX, and so are comfortable with basic MX installation and running.
Line 9: Line 9:
Cumulus MX has been updated so frequently in 2020, that you may be used to updating to a new build, and for you this article is not useful.
Cumulus MX has been updated so frequently in 2020, that you may be used to updating to a new build, and for you this article is not useful.


=Introduction to updating MX=


==Installer Option==
=Where to download a release distribution=
 
==Latest release==
 
Download release distribution zip from  [[Software|the Software article in this Wiki]] for latest version.
 
Be aware that the developer needs to remember to update this link each time there is a new release.
 
An image that contains a '''Rapberry Pi lite operating system''' and the latest Cumulus MX release already included, may be downloaded from the same wiki article.


HansR on support forum is developing an installer as this is being typed, see [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=18893 this topic]. He proposes to start a new topic when his installer is ready for general use, and I hope he will update this section appropriately.
==Any release developed by Mark Crossley==


==Updating if you are running MX on a Linux computer==
Download release distribution zip from https://github.com/cumulusmx/CumulusMX/releases for earlier versions.


You might want to read galfert's post on the support forum [https://cumulus.hosiene.co.uk/viewtopic.php?p=148851#p148851 here] for the relevant Linux instructions in a concise format.


== Updating if you use the start/stop management script ==  
=Introduction to upgrading MX=


This section contributed by ''Jank'' on support forum.  Note the version on the forum might have been updated from the original included here.
<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document is 'Work In Progress' so content may not be complete or accurate!
</div>
[[Category:Cumulus MX]]


1. look on [[Software|Software download page]], find the link to latest version, and fill out the '...' below appropriately as you run these 2 commands on your device where you do downloads:
You can download and unzip in advance, but cannot replace existing files while they are being used, so how you stop MX is in next two sub-sections.
<pre>cd /tmp
wget https://github.com/cumulusmx/CumulusMX/ ... .zip</pre>


2. Once that download is complete, start cumulusmx.sh with option -u
My preference, after download, and unzip into a holding area, is to copy (it would be file transfer if you download on another device) over my existing installation all files except:
<pre>/home/pi/CumulusMX/cumulusmx.sh -u</pre>
*CumulusMX.exe
*CumulusMX.exe.config
while MX is still running. I then stop MX for minimal time needed to replace just those two files before I restart MX.  That way I minimise downtime, especially useful for a patch release when few files have changed, as it takes some time to replace all the files in the installation.


3.When asked for the zip file, enter
For simplicity, in guidance below, MX is stopped before any files are replaced.
<pre>/tmp/CumulusMXDist</pre> and hit the TAB Button


4.Choose the zip file with the CumulusMX update and hit return.
==Upgrading if you run MX as a service==


5. Follow the on screen instructions
Don't forget to stop the service, before you do the upgrade.


6. With each update component .....you can choose: [y]es, [n]o, [A]ll, [N]one, [r]ename
For  a Linux Operating system: <code>sudo systemctl stop cumulusmx</code>.


I would recommend select '''A''' as that will simply replace all files without further action.
For Microsoft Windows Operating System:


==Upgrading if you run MX interactively==


CumulusMX will be restarted after update completes.
If you do NOT run as a service, you will want to use '''Control and C''' in the terminal (or command) window running MX to make the software close tidily.  


You can check if the update was successful by using option -s:
<pre> /home/pi/CumulusMX/cumulusmx.sh -s</pre>


== Updating to the next MX release if you have not updated before ==
== Updating to the next MX release if you have not updated before ==
Line 51: Line 58:
The simplest update is from the immediate preceding build, and the steps required are summarised as follows:
The simplest update is from the immediate preceding build, and the steps required are summarised as follows:
# Download the release distribution zip for the next build (see later for where from)
# Download the release distribution zip for the next build (see later for where from)
#* My advice is to have a separate download location away from the location where you are installing/running the software; it can retain older releases to make regression simple if the new release proves to have a bug.
# Use '''Control-C''' to stop Cumulus MX (see later if running as a service)
# Use '''Control-C''' to stop Cumulus MX (see later if running as a service)
# Take a backup of your complete existing MX installation (as it is not running, no files will be locked)
# Take a backup of your complete existing MX installation (as it is not running, no files will be locked)
# Unzip the new distribution overwriting the previous installation (the release announcement might ask you to delete obsolete files)
# Unzip the new distribution, overwriting the previous installation (the release announcement might ask you to delete obsolete files)
# Run any one-off batch scripts needed to prepare for upgrade (see later for examples)
# Run any one-off batch scripts needed to prepare for upgrade
#* These might be needed if log files in the new release contain fields that were not in earlier release
#* These might be needed if you use optional database functionality (see later for examples)
# Do any actions that require you to use [[MX Administrative Interface|Admin Interface]] to change settings
# Do any one-off actions in relation to any files provided by MX for you to run a web site with the default pages
# Restart your Cumulus MX (consider running with '''-debug''' parameter if you are not sure the new build is bug free)
# Restart your Cumulus MX (consider running with '''-debug''' parameter if you are not sure the new build is bug free)


The remainder of this article explains all options for updating, and is thus less simple than above.
The remainder of this article explains all options for updating, and is thus less simple than above.


==Considerations that determine when to update==
==Installer Option==


Cumulus MX will nag you, in various places, to make you aware if you are not running the latest build. Some people will choose to update as soon after a new release as they can. However, each upgrade does involve a period when MX is not running, and that causes some loss of data:
[https://cumulus.hosiene.co.uk/memberlist.php?mode=viewprofile&u=9016 HansR] on support forum has developed an multi-platform installer, see [https://cumulus.hosiene.co.uk/viewtopic.php?f=44&t=18916 Multiplatform Installer for CumulusMX] which makes life a lot easier if you do not wish to dive into the (file level contents) internals of CumulusMX.
*for some weather station types readings taken every minute, or more frequently, are replaced by whatever period you station data logger records at;
*for other stations, without their own logging, all data is lost for the period when MX is not running.


#Copy the '''InstallCMX.exe''' to any directory you want, on the drive where you wish to install (or have previously installed) MX.
#Copy the CumulusMX release distribution zip(s) to that same directory, if you have not already done so. You may have more than one distribution in the same directory.
#*The install procedure gives you the possibility to select, or define, the Archive to install, and the location where to install.
#Stop CumulusMX
#Run '''InstallCMX''' and confirm / fill in (on the console) where you wish to install (or update) CMX. The default for Windows is C:\CumulusMX\ and for Linux it is : /home/CumulusMX. The Installation directory can be modified.
#* You can give the '''build number''', for the release distribution zip, to install as command line '''argument'''.
#Start CumulusMX


Other, more cautious, people (like the present writer) will not update each time a new release becomes available (and there are a lot of new releases in 2020), here are some of the reasons:
'''NOTE:''' On Windows you run the installer as any other command line executable and it is best to open a command window in which you start the installer. On Linux you run it on the command line as "mono ./InstallCMX.exe", the mono command can be omitted if mono is already active (e.g. if you run CumulusMX as a service, and stop it, mono remains active).
* you will see in the Cumulus Support Forum that many builds have bugs, and so you realise that it is better to stick with your fully working release, than install one with bugs;
'''NOTE:''' In an existing installation with modified files, make sure those are in a different, either filename or folder (a safe place). If they have the same name as files in the distribution, they will be overwritten.
* you may wish to avoid the loss of data mentioned above, by minimising the number of times that you stop MX;
'''NOTE:''' Check in the <code>webfiles</code> directory to see if any files there have been modified, since your last upload to your web server. If so, move those new files to the website.
* you may run MX on a computer that you rarely visit, perhaps even in a remote location not often visited, so you prefer to leave it untouched, rather than risk possibility of being unable to restart remotely;
* you may just have more important ways to use your time than updating your MX software, and some new builds might not give you any benefits that make it worthwhile to change your priorities.


It is perfectly possible to update from old versions of MX to the latest, skipping intermediate versions, but there are some key versions that you should not skip over. This article also includes suggestions for which releases to install and get running before moving onto newer releases. This will be especially useful for those people who do not immediately update to new releases, as per above suggestions.
After the installation, there is a '''log file'''. Check the log file to see everything has gone well.
There is an '''ini file''' where you can control:
* NormalMessageToConsole=true or (default) NormalMessageToConsole=false
* TraceInfoLevel=Warning (out of: Error, Warning, Info, Verbose, None)


===Advice about skipping versions===
Any reactions (thank you, issues and questions) please post in the [https://cumulus.hosiene.co.uk/viewtopic.php?f=44&t=18916 download thread of the installer].


Early builds of MX were all assigned to the same version number, recently the version number seems to change with every new build. Thus the following simple advice about skipping versions might not be applicable in 2021 onwards.
Modifications and additions on user request can be discussed (e.g. think about automated start/stop, upload of webfiles to the directory etc....), use the talk for this page.


One would assume that, skipping minor releases in an update (from 3.x.y to 3.x.z, i.e. only final digit of version number changing is a minor release) would always be simple, even if several intermediate builds are skipped. Most developers would not introduce a new feature in a minor release that requires a one-off extra action for a successful upgrade. Unfortunately, some minor releases of MX have required you to do special actions.
==Updating if you are running MX on a Linux computer==


One might assume that the ease of doing a major update (i.e. where middle part of version number changes) involves greater difficulty. Most developers would only classify a new release as a major update if there was new functionality being introduced that, depending on what functionality you use, might involve a one-off extra action to prepare for upgrade to that version. It is harder to understand why some MX releases are classified as major version number change, and some as minor version number change.
You might want to read galfert's post on the support forum [https://cumulus.hosiene.co.uk/viewtopic.php?p=148851#p148851 here] for the relevant Linux instructions in a concise format.


You may be using an old version of MX that is actually several versions behind, or even still using a beta version released by Steve Loft, don't worry because this article covers upgrading however old a version you are using. You can update skipping some major updates, but as that can be more complex, there is a separate section later with advice on how to update in stages from an early build to a recent build.
== Updating if you use the start/stop management script ==


= Knowing when a new release is available =
This section contributed by ''Jank'' on support forum.  Note the version on the forum might have been updated from the original included here.


== CREDIT ==
1. look on [[Software|Software download page]], find the link to latest version, and fill out the '...' below appropriately as you run these 2 commands on your device where you do downloads:
<pre>cd /tmp
wget https://github.com/cumulusmx/CumulusMX/ ... .zip</pre>


Thanks to ''Billy'' on support forum for suggesting text for this section.
2. Once that download is complete, start cumulusmx.sh with option -u
<pre>/home/pi/CumulusMX/cumulusmx.sh -u</pre>


== Using forum notifications ==
3.When asked for the zip file, enter
<pre>/tmp/CumulusMXDist</pre> and hit the TAB Button


As new releases are announced in [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=17887 Cumulus MX Announcements and Download - PLEASE READ FIRST] topic, you can use the spanner tool to '''subscribe''' to this topic to receive notifications of a new post announcing a new release (or any other release-related announcement).
4.Choose the zip file with the CumulusMX update and hit return.


== MX terminal message ==
5. Follow the on screen instructions


If ...  
6. With each update component .....you can choose: [y]es, [n]o, [A]ll, [N]one, [r]ename
#...you have a monitor to see the terminal output from the Cumulus MX engine (Windows calls this either a command window, powershell window, or a terminal window depending on how you invoke it, for Unix-based implementations this is the output window when using the terminal functionality), AND
#...your device running MX is connected to internet, AND
#...your MONO (if not Windows) is not obsolete (SSL certificate out of date), AND
#...you restart MX
... then you will see a prompt when a new version of MX is available.


It is worth stressing that if you leave MX running, then this feature will leave you blissfully unaware that an update is available; it only checks when MX is restarted.
I would recommend select '''A''' as that will simply replace all files without further action.


== MXDiags ==


In addition ...
CumulusMX will be restarted after update completes.
#...if you can view the MXdiags file for the current session of MX, AND
#... MX is running with connection to the internet, AND
#...you restart MX
... if a new version of MX is available, the MXDiags file will say so (the message is not easy to spot as there is a lot of output before it, and variation in what output appears before it). Anyway, here is one example, just one example as in my experience '''the message has appeared at different places for each of the recent updates'''):
<pre>
2020-05-27 04:18:48.326 Calculating sunrise and sunset times
2020-05-27 04:18:48.326 Sunrise: 04:58:11
2020-05-27 04:18:48.326 Sunset : 21:19:54
2020-05-27 04:18:48.326 Tomorrow sunrise: 04:57:08
2020-05-27 04:18:48.326 Tomorrow sunset : 21:21:11
2020-05-27 04:18:48.388 You are not running the latest version of CumulusMX, build 3080 is available.
2020-05-27 04:18:48.763 Station type:</pre>


There is no message in whatever place it can appear ...
You can check if the update was successful by using option -s:
*... if you are using latest version, OR
<pre> /home/pi/CumulusMX/cumulusmx.sh -s</pre>
* ... if you are not connected to the internet, OR
*... if you keep MX running all the time!


== Start/Stop script ==
=Considerations that determine when to update=


When the '''Status''' option of this script is used, whenever a new release of MX is available, it will output a message.
Cumulus MX will nag you, in various places, to make you aware if you are not running the latest build. Some people will choose to update as soon after a new release as they can. However, each upgrade does involve a period when MX is not running, and that causes some loss of data:
 
*for some weather station types readings taken every minute, or more frequently, are replaced by whatever period you station data logger records at;
== Dashboard in MX Admin Interface ==
*for other stations, without their own logging, all data is lost for the period when MX is not running.


In recent releases, one of the alarm indicators shown at the bottom of the dashboard page of the administrative interface, will flash red when you are not running latest build, and glow green when there is no newer release compared to the one you are running.


== Web Tags ==
Other, more cautious, people (like the present writer) will not update each time a new release becomes available (and there are a lot of new releases in 2020), here are some of the reasons:
 
* you will see in the Cumulus Support Forum that many builds have bugs, and so you realise that it is better to stick with your fully working release, than install one with bugs;
Since 3.7.0 release of MX, two relevant web tags have become available:
* you may wish to avoid the loss of data mentioned above, by minimising the number of times that you stop MX;
* <#NewBuildAvailable> Returns a boolean value
* you may run MX on a computer that you rarely visit, perhaps even in a remote location not often visited, so you prefer to leave it untouched, rather than risk possibility of being unable to restart remotely;
**  0 - MX running the latest build available
* you may just have more important ways to use your time than updating your MX software, and some new builds might not give you any benefits that make it worthwhile to change your priorities.
**  1 - MX is running an earlier build than the latest public release
* <#NewBuildNumber>  Displays the latest public release build number
 
=Where to download a release distribution=


==Latest release==
It is perfectly possible to '''upgrade from rather old versions of MX, to the latest''', skipping intermediate versions, '''but there are some key versions that you should not skip over'''. This page, in subsequent sections, includes suggestions for which releases to install (and get running) before moving onto newer releases. This will be especially useful for those people who do not immediately update to new releases, as per above suggestions.


Download release distribution zip from  [[Software|the Software article in this Wiki]] for latest version.
===Advice about skipping versions===
 
Be aware that the developer needs to remember to update this link each time there is a new release.
 
An image that contains a '''Rapberry Pi lite operating system''' and the latest Cumulus MX release already included, may be downloaded from the same wiki article.


==Any release developed by Mark Crossley==
Please see [[Updating_MX_to_new_version#Updating_from_a_very_old_version]] sub-section later on this page (and preceding sub-sections where relevant)


Download release distribution zip from https://github.com/cumulusmx/CumulusMX/releases for earlier versions.
The important point to make here is that when you do an update, you should copy in '''ALL''' files from the release distribution you want to run next. This is because there are dependencies between files within a distribution, so missing out any particular file may stop other files from working. However, if you are skipping versions, read the in-between release announcements to see if there are any one-off actions.


= What to read (and when) before updating =
= What to read (and when) before upgrading =


== Updating from immediately preceding build ==
== Updating from immediately preceding build ==
Line 305: Line 298:
====if using either 3.8.x or 3.9.y release====
====if using either 3.8.x or 3.9.y release====


See previous entry, upgrade directly to 3.9.6, or any later build without any major change.
See previous entry, upgrade directly to 3.9.6, or any later build, within 3.9.x, without any major change.
 
====From 3.9.y to 3.10.x====
 
The major aspects that changes at 3.10.1 (3.10.0 was withdrawn) are:
* Changes to settings
** You need to open the [[MX Administrative Interface]] and work through all pages in the '''Settings''' menu.
*Changes to web files in <code>CumulusMX/web</code>, <code>CumulusMX/webfiles</code> and introduction of <code>CumulusMX/webfiles-legacy</code>
**If you used the provided templates to produce web pages in earlier releases, or you had directly customised provided [[Customised templates|web templates]], these will no longer work
**Please see [[New Default Web Site Information|New Default Web Site Information page]] for further advice
 
====From 3.10.x to 3.11.y====
 
Treat like any other major version upgrade as described above.
 
 
= Knowing when a new release is available =
 
== CREDIT ==
 
Thanks to ''Billy'' on support forum for suggesting text for this section.
 
== Using forum notifications ==
 
As new releases are announced in [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=17887 Cumulus MX Announcements and Download - PLEASE READ FIRST] topic, you can use the spanner tool to '''subscribe''' to this topic to receive notifications of a new post announcing a new release (or any other release-related announcement).
 
== MX terminal message ==
 
If ...
#...you have a monitor to see the terminal output from the Cumulus MX engine (Windows calls this either a command window, powershell window, or a terminal window depending on how you invoke it, for Unix-based implementations this is the output window when using the terminal functionality), AND
#...your device running MX is connected to internet, AND
#...your MONO (if not Windows) is not obsolete (SSL certificate out of date), AND
#...you restart MX
... then you will see a prompt when a new version of MX is available.
 
It is worth stressing that if you leave MX running, then this feature will leave you blissfully unaware that an update is available; it only checks when MX is restarted.
 
== MXDiags ==
 
In addition ...
#...if you can view the MXdiags file for the current session of MX, AND
#... MX is running with connection to the internet, AND
#...you restart MX
... if a new version of MX is available, the MXDiags file will say so (the message is not easy to spot as there is a lot of output before it, and variation in what output appears before it). Anyway, here is one example, just one example as in my experience '''the message has appeared at different places for each of the recent updates'''):
<pre>
2020-05-27 04:18:48.326 Calculating sunrise and sunset times
2020-05-27 04:18:48.326 Sunrise: 04:58:11
2020-05-27 04:18:48.326 Sunset : 21:19:54
2020-05-27 04:18:48.326 Tomorrow sunrise: 04:57:08
2020-05-27 04:18:48.326 Tomorrow sunset : 21:21:11
2020-05-27 04:18:48.388 You are not running the latest version of CumulusMX, build 3080 is available.
2020-05-27 04:18:48.763 Station type:</pre>
 
There is no message in whatever place it can appear ...
*... if you are using latest version, OR
* ... if you are not connected to the internet, OR
*... if you keep MX running all the time!
 
== Start/Stop script ==
 
When the '''Status''' option of this script is used, whenever a new release of MX is available, it will output a message.
 
== Dashboard in MX Admin Interface ==
 
In recent releases, one of the alarm indicators shown at the bottom of the dashboard page of the administrative interface, will flash red when you are not running latest build, and glow green when there is no newer release compared to the one you are running.
 
== Web Tags ==
 
Since 3.7.0 release of MX, two relevant web tags have become available:
* <#NewBuildAvailable> Returns a boolean value
**  0 - MX running the latest build available
**  1 - MX is running an earlier build than the latest public release
* <#NewBuildNumber>  Displays the latest public release build number
 


= Recommended Actions when Updating =
= Recommended Actions when Updating =
Line 387: Line 453:


== After an update if you use the standard web template files ==
== After an update if you use the standard web template files ==
If you do not use the example web template files, provided as standard, with a MX release, the remainder of this section can be ignored.
If you use the third party [[CumulusMX and Cumulus1 UI style Multilingual Websites]] web pages, as maintained by BCJKiwi, follow any instructions in his package as to which of the files in this section are required and any announcements in his forum topic about when such files need to be upgraded.


=== web folder from zip ===
=== web folder from zip ===


The new web template files will be in your '''web''' folder and MX will process the new files and upload the web pages produced as before with no further action by you.
If you do use the example web templates to produce web pages for your web server, and you have copied all files from the release distribution ....
 
....MX will find any new web template files in your '''web''' folderMX will process those template files, and upload the web pages produced, ''as before with no further action by you''.


=== webfiles folder from zip ===
=== webfiles folder from zip ===
<br/>
<big>PLEASE COULD SOMEONE USING LATEST MX RELEASE ENSURE THIS IS KEPT UP TO DATE</big>
<br/>


You do need a further action, if any files in this folder have been updated.  The updated files must be uploaded to your web site.
You do need a further action, if any files in this folder have been updated.  The updated files must be uploaded to your web site.
Line 403: Line 480:
*'''weatherstyle.css''' has not been updated since its last update by Steve Loft in 2009.
*'''weatherstyle.css''' has not been updated since its last update by Steve Loft in 2009.
*subfolder '''images''' contains 2 pictures that have stayed same since MX launch
*subfolder '''images''' contains 2 pictures that have stayed same since MX launch
*subfolder '''js''' contains one file. The file '''\CumulusMX\webfiles\js\cumuluscharts.js''' has changed in a lot of releases.  The standard web page '''trends.htm''' will only work if the correct version of that file has been uploaded to your web site and placed in a sub-folder "js" of where "trends.htm" is installed. BCJKiwi's user interface '''charts.php''' also requires the same JavaScript file, so upload the file anytime it is updated in a MX release.
*subfolder '''js''' contains one file or two files (depending on which MX release you are using:
**The standard web page '''trends.htm''' will only work if the correct version of any file here has been uploaded to your web site, and placed in a sub-folder "js" of where "trends.htm" is installed. .
*** BCJKiwi's user interface '''charts.php''' also requires the same JavaScript file (or files?), so upload the file(s0 anytime there is a change in a MX release.
** The file '''\CumulusMX\webfiles\js\cumuluscharts.js''' has changed in a lot of releases, it is updated whenever there is a change in the JSON files used for providing data to the charts, and you must use the right upgrade.
** The file '''historiccharts.js'' (it was introduced from release 3.8.7) is similar to previous script, but drives the historic chart functionality on your web server
*subfolder '''lib''' currently contains 3 folders:
*subfolder '''lib''' currently contains 3 folders:
**'''highstock''' - this folder is no longer needed on your web site
**'''highstock''' - this folder is no longer needed on your web site (and the most recent MX releases have emptied it)
**'''jquery''' - this folder is needed for "trends,htm" to work, but it has not been updated since 2014. Both the files included are obsolete packages, no longer officially available, and there are no plans to replace them.
**'''jquery''' - this folder is needed for "trends,htm" to work, but it has not been updated since 2014. Both the files included are obsolete packages, no longer officially available, and there are no plans to replace them.
**'''steelseries''' - this folder has 3 sub-folders, and I can only think of 2 MX releases when anything was changed here, and I don't anticipate any further changes.
**'''steelseries''' - this folder has 3 sub-folders, I can only think of 2 MX releases when anything was changed here, and I don't anticipate any further changes.
 
= After update - checking for bugs in MX or mistakes in your installation =


== After update - checking for bugs in MX or mistakes in your installation ==
Start the new installation of MX and watch out for any errors - If the device you run MX on has a monitor, then look in the terminal/command window. In all cases look at the latest file in the [[MXdiags_folder|MXdiags folder]] to see if any errors are reported.


Start the new installation of MX and watch out for any errors - If the device you run MX on has a monitor, then look in the terminal/command window. In all cases look at the latest file in the MXdiags folder to see if any errors are reported.
In newer releases of MX, also see [[ServiceConsoleLog.txt]]


Also keep an eye on the support forum for first few days, some releases do have bugs, as developer cannot test all possible configurations.
Also keep an eye on the support forum for first few days, some releases do have bugs, as developer cannot test all possible configurations.
5,838

edits

Navigation menu