Migrating from Cumulus 1 to MX: Difference between revisions

m
Line 74: Line 74:
=Migrating existing data files from legacy Cumulus to a recent build of MX=
=Migrating existing data files from legacy Cumulus to a recent build of MX=


If you want to continue using the same weather station, and you are not moving to a different home, you will want to maintain the data you collected using Cumulus 1 in your MX installation. This is what is meant by migration, and is the subject of the rest of this article.
If you want to continue using the same weather station, and you are not moving to a different home, you will want to maintain the data you collected using Cumulus 1 in your MX installation. This is what is meant by migration, and there is quite a bit to read, taking up most of the rest of this Wiki page.  The Cumulus configuration files for Cumulus 1 and for MX share the same name, but their content is very different, so these are discussed at length in subsections below.
 
Obviously, you will want to copy ALL (except weather diary which uses a different file) the files in the [[#"data" folder]], from any old installation into the new installation, but migrating these is not always easy, here is a quick summary of the potential issues (all discussed further in various subsections later):
* '''When you first run MX, the [[#Start date|Start Date]] is recorded in the configuration file'''
** In the legacy software, this date is purely used as something to appear on one of the [[Customised_templates#The_Standard_Templates_for_Cumulus_1|sample web templates]]
** In Cumulus MX, this date controls what data is read from [[Monthly log files]], such as [[Standard log files]], and [[Extra Sensor Files]], any lines in those files with earlier dates are ignored by CumulusMX.exe, ExportToMySQL.exe, and other utilities.
**  You can edit this [[Cumulus.ini#Data_Logging|'''StartDate=xxxxx''']] parameter, in the admin interface select ''Station Settings → Common Options → Advanced Options → Records Began date''
* Be aware your new installation has to use the same "locale" as the old installation, or MX will struggle as the locale affects how new lines are stored, and how MX expects old lines to have been stored the same way.  
* If your old installation, is on a different operating system to the new installation, remember that Microsoft Windows uses different line terminators to all other operating systems, although MX should cope with mixed line terminators, any third party routines reading your data files will probably not accept a line terminator change.  See [[MX on Linux]] page for more information on this.
* Cumulus MX is much more fussy than Cumulus 1 about every line in any data file using exactly the same locale formatting:
** Cumulus 1 is able to accept any character (other than the list separator, and space) being used to separate hour and minutes in time-stamps, MX expects a colon ":"
** [[Amending dayfile]] tells you about how MX is far more fussy about the content in [[dayfile.txt]]
** [[:Category:Ini Files|.ini files]] explains how time-stamps are formatted differently in the extreme tracking files, and how MX prefers decimal points to decimal commas


==File names==
==File names==


Note that if you run MX on a UNIX based operating system (e.g. Linux or Raspberry Pi OS) all file names are case-sensitive, please read documentation to see where capital letters are required in those file names. Be aware that Wiki pages change first letter to a capital even when a file that must be all lower-case is being described.
Note that if you run MX on a UNIX based operating system (e.g. Linux or Raspberry Pi OS) all file names are case-sensitive, please read documentation to see where capital letters are required in those file names. Be aware that Wiki pages always change first letter to a capital, so do check carefully if that wiki page is describing a file name that must be all lower-case.


==Line terminators==
==Line terminators==


If you are not running MX on a Microsoft Windows computer, ideally you should change the characters appearing at the end of every line in any file you move from Cumulus 1.
If you are not running MX on a Microsoft Windows computer, ideally you should change the characters appearing at the end of every line in every file you move from Cumulus 1.


I say ideally, because although Microsoft is fussy, and determined to be different by insisting files match its choice, most other operating systems can tolerate different line terminators.   
I say ideally, because although Microsoft is fussy, and determined to be different by insisting files match its choice, most other operating systems can tolerate different line terminators.  This means that MX will generally tolerate files using a mixture of line terminators. However, if you are using third-party routines, please be aware that these are often written to expect a particular line terminator, and they may give unexpected results when used with files that do not match that expectation!


With Cumulus 1 (or MX) on Windows, each line in every file ends with both carriage return and line feed.
With Cumulus 1 (or MX) on Windows, each line in every file ends with both carriage return and line feed.
Line 92: Line 104:
For all Unix-based operating systems (Linux, Raspberry Pi OS, and other variants), each line should end with just a Line Feed.
For all Unix-based operating systems (Linux, Raspberry Pi OS, and other variants), each line should end with just a Line Feed.


To change end of line characters, run each file through an editor designed for programmers. Various editors are available but "Notepad ++" is one that is popular on Windows, but can run on other operating systems. In its '''Edit''' menu, choose '''EOL conversion'''.  
To change end of line characters, run each file through an editor designed for programmers. Various editors are available but "Notepad ++" is one that is popular on Windows, but can run on other operating systems. In its '''Edit''' menu, choose '''EOL conversion'''.   On a Linux system like a Raspberry Pi computer, please see [[MX_on_Linux#Changing_line_terminators]].


If you use PHP Hypertest Pre-processor anywhere on your installation, normally that is written for "Line-feed" as line terminator and any "carriage return" in your files may mess up the content unless you code in a "trim" to remove it. Hopefully, if you use PHP, you are technical enough to realise you may need to edit the code depending on which device it is being run on!
If you use PHP Hypertext Pre-processor anywhere on your installation, normally that is written for "Line-feed" as line terminator and any "carriage return" in your files may mess up the content unless you code in a "trim" function to remove it. Hopefully, if you use PHP, you are technical enough to realise you may need to edit the code depending on which device it is being run on!


==Configuration file==
==Configuration file==
Line 145: Line 157:
When you first run any Cumulus software (whatever flavour) a parameter is added to the configuration file that documents the date Cumulus was first used.  
When you first run any Cumulus software (whatever flavour) a parameter is added to the configuration file that documents the date Cumulus was first used.  


For Cumulus 1, this parameter appeared in two places on the example web template for all-time records, but was otherwise ignored.  Thus if you decided to import into your data logs readings from before you first ran Cumulus, the legacy software would be able to find those earlier records.
For Cumulus 1, this parameter appeared in two places on the example web template for all-time records, but was otherwise ignored.  Thus if you decided to import into your data logs readings from before you first ran Cumulus, the legacy software would be able to find those earlier records, without you needing to change the date that appeared on the web page.


'''CumulusMX.exe''' uses this parameter to determine the first standard log file to start reading from, it will ignore any log files for earlier dates. Thus if you were to migrate your Cumulus 1 configuration file into MX, you should check the "StartDate=" line in the '[Station]' section is correct for your earliest data before you let MX read this configuration file for first time.  If you need to make any edit, ensure you stick to exactly the same date format.
'''CumulusMX.exe''' uses this parameter to determine the first standard data file to start reading from, it will ignore any data files for earlier dates, it will also ignore any lines with an earlier date in the first data file. Thus if you were to migrate your Cumulus 1 configuration file into MX, you should check the [[Cumulus.ini#Data_Logging|'''StartDate=xxxxx''']] parameter in the '[Station]' section is correct for your earliest data before you let MX read this configuration file for first time.  If you need to make any edit, ensure you stick to exactly the same date format.


From release 3.10.1, MX allows you to edit this start date within the admin interface. It is "hidden" as an "advanced" setting, with a strict danger warning, but it is there!
From release 3.10.1, MX allows you to edit this start date within the admin interface, just select ''Station Settings → Common Options → Advanced Options → Records Began date''. It is "hidden" as an "advanced" setting, with a strict danger warning, because of the importance of sticking to exactly the right date format when you edit it!


=== Station connections===
=== Station connections===


You can skip this subsection if your weather station connects to MX either by USB or by wireless connection.  It is relevant if your station connects by a serial connection and MX needs to be told which port.  It also applies if the serial connection is converted to USB.
''You can skip this subsection if your weather station connects to MX either by USB or by wireless connection''.   
 
This subsection is relevant if your station connects by a serial connection and MX needs to be told which port.  It also applies if the serial connection is converted to USB.


If your weather station used a port to connect to Cumulus 1, that port was set on the settings screen as a number and stored in [[Cumulus.ini_(Cumulus_1)]] in the station section as a parameter in the format '''Port=n'''.  
If your weather station used a port to connect to Cumulus 1, that port was set on the settings screen as a number and stored in [[Cumulus.ini_(Cumulus_1)]] in the station section as a parameter in the format '''Port=n'''.  
Line 161: Line 175:
If your old parameter had a value of '''3''', and you are still using Windows, the new setting would have value of '''COM3''', i.e. serial port 3 now requires a "COM" prefix.   
If your old parameter had a value of '''3''', and you are still using Windows, the new setting would have value of '''COM3''', i.e. serial port 3 now requires a "COM" prefix.   


A typical parameter value for other serial connecting devices might be "/dev/ttyUSB3" where the final digit will change depending on the new connection.
A typical parameter value for other serial connecting devices might be "/dev/ttyUSB3" where the final digit will change depending on the new connection.  You can search for posts on the support forum that talk about how to find out what connection is being used, depending on what hardware you are using.


===RG11 Rain gauge===
===RG11 Rain gauge===


If you use a RG11 rain gauge:
If you use a RG11 rain gauge:
*Replace: '''RG11port=n''' and '''RG11port2 =n''' (Cumulus 1) where n is a number, by '''RG11portName=xxxx''' and '''RG11portName2=yyyy'''  (Cumulus MX on Windows) where the value is a string with values as per previous paragraph depending on device on which Cumulus MX is running.
*Replace: '''RG11port=n''' and '''RG11port2 =n''' (Cumulus 1) where n is a number,  
* With: '''RG11portName=xxxx''' and '''RG11portName2=yyyy'''  (Cumulus MX on Windows)
 
The new value is a string with values as per previous paragraph depending on device on which Cumulus MX is running.
 
===Other Cumulus.ini parameters===


From release 3.10.1, there appear to be other new parameters, not yet documented.
Lots of parameters in [[Cumulus.ini]] are being changed as MX is developed.  So much had changed by the time that MX releases reached 3.12.0, the first run of that particular release created a brand new '''Cumulus.ini''' discarding the old file. If you migrate from Cumulus 1 to MX without stepping through MX releases, you may have several issues, but without knowing which MX release you have selected, all I can suggest is that you work through all settings found in the Settings page of the MX release you choose. Although you could compare [[Cumulus.ini (Cumulus 1)]] and [[Cumulus.ini]] pages, there is no guarantee that either is totally accurate!


==Strings.ini==
==Strings.ini==
Line 180: Line 199:
The contents of the [[Samplestring.ini|samplestring.ini]] file you get in your MX release distribution varies depending on the release you have downloaded.   
The contents of the [[Samplestring.ini|samplestring.ini]] file you get in your MX release distribution varies depending on the release you have downloaded.   


Check your existing '''strings.ini''' file against the ''samplestring.ini'' file in the MX distribution you have.  If the attribute names (left hand side of the equals sign) match for the parameters you selected to include in your '''strings.ini''', then you can reuse your existing file. If your file includes attributes that are no longer in the MX ''samplesting.ini'' file, then you will need to edit your '''strings.ini''' file that is placed in the folder containing CumulusMX.exe.
Check your existing '''strings.ini''' file against the ''samplestring.ini'' file in the MX distribution you have.  If the attribute names (left hand side of the equals sign) match for the parameters you selected to include in your '''strings.ini''', then you can reuse your existing file. If your file includes attributes that are no longer in the MX ''samplesting.ini'' file, then you will need to edit your '''strings.ini''' file that is placed in the folder containing CumulusMX.exe.  You may also need to add new items into your '''strings.ini''' based on new content in ''samplestring.ini''.


==NOAA style reports==
==NOAA style reports==
Line 186: Line 205:
The generation of reports is an optional feature, if you have never used it your (legacy) Cumulus Reports folder will be empty, then you have no files to move to your MX installation, and you should skip the rest of this sub-section.
The generation of reports is an optional feature, if you have never used it your (legacy) Cumulus Reports folder will be empty, then you have no files to move to your MX installation, and you should skip the rest of this sub-section.


Please see [[Reports_folder]] for full information.  Cumulus software creates reports, it does not edit existing reports, so migration is simple. Just copy the contents of the '''Reports''' folder in your original Cumulus installation into the folder in the new installation.
Please see [[Reports_folder]] for full information.  Cumulus software creates reports, it does not edit existing reports, so migration is fairly simple. Just copy the contents of the '''Reports''' folder in your original Cumulus installation into the folder in the new installation. Of course, nothing is totally simple, the encoding default changes between Cumulus 1 and MX, and once again we might need to consider end of line characters!


For those of you who are more technical:
For those of you who are more technical:
* files created in Microsoft's Windows Operating System use two characters (carriage return and line feed) to end each line, while all other operating systems use a single character (line feed in most Unix derived systems). Apple Mac are again different in using just Carriage Return. This should not cause any problems.
* files created in Microsoft's Windows Operating System use two characters (carriage return and line feed) to end each line, while all other operating systems use a single character (line feed in most Unix derived systems). Apple Mac are again different in using just Carriage Return. This should not cause any problems.
* files can be encoded (how individual characters are represented by binary codes) in different ways. There is more about encoding at [[Reports_folder#Encoding]], the relevance here is that if your MX settings and Cumulus 1 settings use different encodings you may find some characters (e.g. degree symbol) do not appear correctly when viewing some of your reports.
* files can be encoded (how individual characters are represented by binary codes) by Cumulus in two different ways. There is more about encoding at [[Reports_folder#Encoding]], the relevance here is that if your MX settings and Cumulus 1 settings use different encodings (as they will if you let this default) you may find some characters (e.g. degree symbol) do not appear correctly when viewing some of your reports.


=="data" folder==
=="data" folder==


You should copy all files in the [[data folder]] from your legacy Cumulus installation to your MX installation if you want to be able to see past data.
You should copy all files in the [[data folder]] from your legacy Cumulus installation to your MX installation if you want to be able to see past data, but as mentioned earlier there are some complications:
 
There are some complications:
* See [[#Start date]] earlier on this page for a change that you may need to make to ensure MX accepts data in any old [[Standard log files]] and [[Extra Sensor Files]]
* See [[#Start date]] earlier on this page for a change that you may need to make to ensure MX accepts data in any old [[Standard log files]] and [[Extra Sensor Files]]
* See [[#Weather Diary]] later on this page, and [[Weather Diary]] page; Cumulus does not provide any utility to convert the old legacy format to the new MX format.
* See [[#Weather Diary]] later on this page, and [[Weather Diary]] page; Cumulus does not provide any utility to convert the old legacy format to the new MX format.
Line 229: Line 246:
===the .ini files===
===the .ini files===


Your Cumulus 1 '''.ini''' files can be read in all release versions from 3.0.0 to 3.5.y. Steve Loft did make two changes in his beta MX, these mean your .ini files may look strange while they have some entries made by Cumulus 1 and some made by MX:
Once again, the simple instruction is to copy these from your old installation to your new installation, to ensure your extreme record history is maintained. Once again, differences between Cumulus 1 and MX mean that the process may not be that simple, especially if you try to migrate straight to the latest MX release.
 
====MX releases up to 3.5.4====
 
'''Your Cumulus 1 '''.ini''' files can be read in all release versions from 3.0.0 to 3.5.y.''' Steve Loft did make two changes in his beta MX, these mean your .ini files may look strange while they have some entries made by Cumulus 1 and some made by MX:
# all newly stored values will use decimal points in these files, (i.e. any decimal commas valid in the legacy software, are not used by MX).   
# all newly stored values will use decimal points in these files, (i.e. any decimal commas valid in the legacy software, are not used by MX).   
# all newly stored time-stamps will use the ISO specification (ISO 8601 Data elements and interchange formats – Information interchange – Representation of dates and times) so new dates have year first, the parts of the date are separated by hyphens, and new times use colon (:) between hour and minutes (i.e. the regional settings that the legacy software adopted have been abandoned by MX)
# all newly stored time-stamps will use the ISO specification (ISO 8601 Data elements and interchange formats – Information interchange – Representation of dates and times) so new dates have year first, the parts of the date are separated by hyphens, and new times use colon (:) between hour and minutes (i.e. the regional settings that the legacy software adopted have been abandoned by MX)
Line 254: Line 275:
====Historical background to dates/times====
====Historical background to dates/times====


When Steve Loft designed his original Cumulus (1), he had no experience to draw upon as to the best way to treat items like dates. He wrote the software originally just for his own benefit and did not need to worry about time zones. Subsequently as he enhanced his software to make it usable by others, he faced many issues on how to cope with different time zones, and different weather stations having different sensors. In Cumulus 1, he basically focussed on compatibility by keeping to his original design for the data log files (both those ending in '''.ini''' and those ending in '''.txt''', and only adding extra fields to the end.
When Steve Loft designed his original Cumulus (1), he had no experience to draw upon as to the best way to treat items like dates. He wrote the software originally just for his own benefit and did not need to worry about time zones. Subsequently as he enhanced his software to make it usable by others, he faced many issues on how to cope with different time zones, and different weather stations having different sensors. In Cumulus 1, he basically focussed on compatibility by keeping to his original design for the data logging files (both those ending in '''.ini''' and those ending in '''.txt''', and only adding extra fields to the end.


When Steve Loft took a new look at the data log files for Cumulus 2, he started with a new design, the principal change was that he decided to use UTC for all fields in them that reference dates and times. Steve struggled with Cumulus 2 largely because he was (at that time) not familiar with the C# language he was later to use for MX. It is fair to say that conversions between local time and UTC did contribute to his failure to get Cumulus 2 to provide the functionality he had offered in the original Cumulus.
When Steve Loft took a new look at the data log files for Cumulus 2, he started with a new design, the principal change was that he decided to use UTC for all fields in them that reference dates and times. Steve struggled with Cumulus 2 largely because he was (at that time) not familiar with the C# language he was later to use for MX. It is fair to say that conversions between local time and UTC did contribute to his failure to get Cumulus 2 to provide the functionality he had offered in the original Cumulus.
Line 295: Line 316:


If you moved your Cumulus 1 installation from one windows pc to another, it is just possible that you might have a mix of "decimal comma" and "decimal point" in your values, or you might have changed the field separator (normally ";" or ",").  Again, these must be consistent in all dayfile.txt lines for MX, and must match what is defined in the locale used.
If you moved your Cumulus 1 installation from one windows pc to another, it is just possible that you might have a mix of "decimal comma" and "decimal point" in your values, or you might have changed the field separator (normally ";" or ",").  Again, these must be consistent in all dayfile.txt lines for MX, and must match what is defined in the locale used.


=web server=
=web server=
5,838

edits