Calculate Missing Values: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
mNo edit summary
No edit summary
Line 1: Line 1:
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both flavours.[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:MX txt Files]][[Category:Cumulus 1]]
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both flavours.[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:MX txt Files]][[Category:Cumulus 1]]


=How Cumulus Works=
=Introduction=


The way that Cumulus works is that:
If you started using Cumulus software before June 2020, you will be aware that the list of fields in both [[Standard_log_files#List_of_fields_in_the_file|data logs]] and [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]] will be different depending on when an individual line was first created (or last edited).
# It reads what we can call '''source values''' (defined below) from your weather station (based on measurements from sensors being transmitted in some way by the station)

# It calculates what we can call '''derived values''' which may be either the source re-expressed in different units, or calculated by combining two or three source values to get a new derivative
If you are using the [[Highcharts_-_Historic|'''Historic Charts''']] feature introduced from release 3.9.2 - b3097 (7 December 2020), you may notice that these new charts have gaps in available data, and the dates with/without data vary depending on what is being plotted.
# It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various [[:Category:Ini_Files|period tracking files]]

#* You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.
For these, and other reasons, this page explains various ways in which you can edit log files, (and for MX users, database tables) so all lines contain all the fields that can be calculated for the weather station you are using.
# It periodically stores the spot (current) derived values in a collection of [[Monthly_log_files]],
#* for MX these include [[Standard log files]], [[Extra Sensor Files]], and [[Air_Link_Log.txt]], at time of writing
#* for Cumulus 1 these are [[Speciallog.txt]], [[Standard log files]], and [[Extra Sensor Files]], for all version 1.9.3 and 1.9.4 releases
# At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]]


{{TOCright}}
{{TOCright}}
To make sense of explanations on this page, you need to understand the terminology used here.




=Some definitions=
=Some definitions=

To make sense of explanations on this page, you need to understand the terminology used here.


==Source value==
==Source value==
Line 23: Line 28:
However, Cumulus does include code that expects a weather station to provide ''a defined minimum set of source values'':
However, Cumulus does include code that expects a weather station to provide ''a defined minimum set of source values'':
# Current air temperature
# Current air temperature
# Relative Humidity
# Current Relative Humidity
# At least one wind speed
# At least one wind speed
# Current air pressure (absolute or sea-level)
# Current air pressure (absolute or sea-level)
Cumulus will stop processing any information from a weather station unless the above 4 source values are being supplied and reveal they are being updated (failure is set is after a total of 6 unsuccessful consecutive attempts to read each of these).


This requirement is a default, but it can be changed:
By default [[Cumulus.ini_(Cumulus_1)#Read-only_parameters_in_the_Station_section|see here]], Cumulus 1 will stop processing any readings from a weather station unless the above 4 source values are being supplied and reveal they are being updated (failure is set is after a total of 6 unsuccessful consecutive attempts to read each of these).
*For recent releases of MX, this is classified as an advance setting (see [[Cumulus.ini]])
*For earlier releases of MX, this is "No sensor check" (see [[Cumulus.ini(MX_3.0.0_to_3.9.7)#Read-only_parameters_in_the_Station_section_.28releases_3.0.0_to_3.6.12.29|here for how to change default]]
*For legacy Cumulus 1, [[Cumulus.ini_(Cumulus_1)#Read-only_parameters_in_the_Station_section|see here for how to change default]]


Cumulus also expects that your weather station can provide:
Cumulus also expects that your weather station can provide:
* A rainfall counter (this could be annual rainfall, or count of rocker bucket gauge tips)
* A rainfall counter (this could be annual rainfall, or count of rocker bucket gauge tips)

Although the lack of that rainfall counter source value will affect functionality, Cumulus will continue to process other source values that are available.
Although the lack of that rainfall counter source value will affect functionality, Cumulus will continue to process other source values that are available.


Some weather stations may also provide one, or more, of these other source values (not a complete definitive list):
Some weather stations may also provide one, or more, of these optional source values (not a complete definitive list):
* Dew-point Temperature
* Dew-point Temperature
* Wind Chill Temperature
* Wind Chill Temperature
Line 43: Line 51:
* Air pollution measurement
* Air pollution measurement


There are some configuration settings [[Cumulus.ini_(Cumulus_1)#Read-Write_parameters_in_the_Station_section|here]] where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.


==Derived value==
===How Cumulus might transform a Source Value===


A dictionary will define '''derived''' as "obtained from a source", and that is the meaning adopted on this page. Steve Loft (in the Cumulus Support Forum) used the terminology "derived" for two purposes.
*For some weather data processed by Cumulus the value sent by the weather station and that used by Cumulus will be different:
# One type of derived value takes a source value, applies any multiplier (may be both first order and second order multipliers) and/or constant that has been defined in calibration settings, and converts the output to the units selected by the Cumulus user.
** This can be because the units used by the weather station, and the units required by the Cumulus user are different (and Cumulus allows the user to select the units in which values will be stored)
# The other type of derived value takes more than one source value, applies a standard calculation, and ouputs a new derivative
** This can be because the user has entered some calibration settings. The exact settings available vary for different weather data, but in some cases a second order multiplier is available (the square of the input value is multiplied by the second order multiplier), a first order multiplier is available (the input value is multiplied by the first order multiplier), and an offset is available (added to the input value). If you want to understand how this works, please see [[Cumulus.ini_(Cumulus_1)#Offsets]] for the details
#* Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file
**A weather station might use a rainfall counter, Cumulus converts it to a rainfall rate (based on difference between two counter readings scaled up to represent amount per hour)


==Derived value==
=="Create Missing"==


This also has two meanings in a Cumulus context:
A dictionary will define '''derived''' as "obtained from a source", and that is the meaning adopted on this page. Cumulus software will use the source values it obtains from your weather station to output not just those source values (as described above), but also values that are "obtained from the source" in two ways:
# If a particular standard log file line has fewer fields than the latest line;
#By using a combination of two (or three) source values, and in some cases adding a time element, Cumulus software is able to calculate the various derived values that this Cumulus Wiki page will be explaining later.
#* '''Create Missing''' is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
#*As Cumulus has developed more and more of these derived spot values are being reported, and that explains why an old [[Standard_log_files#List_of_fields_in_the_file|data log line]] might have fewer fields than newer lines.
# If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
#*If you have some of these derived value fields where the value is missing because it was not calculated by the Cumulus software release you were running at the time the line was created, you will find out below how to calculate those missing values and have them added to the data logs.
#* Please see [[Amending dayfile]] page for full details.
#Cumulus software will track, both the source values, and the values derived from them, and compare against existing extreme records, to report the highest and/or lowest, in various periods.
#* '''Create Missing''' is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.
#*Although, Steve Loft (in the Cumulus Support Forum) used the terminology "derived" for this process in his description of what his software does, this Wiki page is not about those derived values.
#*You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.


If you are using Cumulus MX, there is a download [[Software#Create_Missing| linked from here]] that does both of these. There are also editors within the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|admin interface]] for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.


If you are using the legacy Cumulus 1 software:
===Derived spot values===
# For the standard log file meaning above, provided you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] includes a script that produces a HTML form where you specify [[Standard_log_files#Introduction|the log file name]] you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated. Please note that script was written to run on an old version of PHP that was current at the time the script was written, it will need some editing to work on latest PHP.
# For the daily summary log meaning above, go to the '''Edit''' menu, and select ''Dayfile.txt''. This brings up an editor with a button labelled "Create Missing", that will not affect any existing line, but can insert missing lines, see [[Amending_dayfile#Create_Missing]].


===Accurate or Not?===
Now we have established what we mean by a derived spot value, we can understand that Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.

If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1. That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
#Only use the editor when Cumulus is '''not''' doing catch-up or end-of-day, or any other action that could create an editing conflict.
#With the contents of the file loaded into the editing table, click the '''Create Missing''' button
#Cumulus 1 will now work through every line from start to end
#If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
#It will read from the [[Standard log files]] all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
#*Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)

=Derived spot values=

Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.


Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):
Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):
* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity
* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity
** The original legacy Cumulus, and MX, use different formulae to calculate dew point, so there is a continuity break if some of your data logs were created by the original Cumulus software and some by MX.
** The original legacy Cumulus, and MX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by MX.
* '''Wet Bulb''', is only calculated by Cumulus 1, not MX
*[[Wind_chill|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
*[[Humidex|'''Canadian Humidity Index (Humidex)''']], [[Heat_index|'''USA Heat Index''']], and [[Apparent_temperature|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
* [[Wind_chill|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
* [[Humidex|'''Canadian Humidity Index (Humidex)''']], [[Heat_index|'''USA Heat Index''']], and [[Apparent_temperature|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
**(The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#Rest_of_this_article|here]]).
** (The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like|here]]).
** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
*[[Feels_Like|'''Feels Like Temperature''']] is calculated by the Cumulus MX flavour only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like|has varied]] in different releases, but use a variation on
* [[Feels_Like|'''Feels Like Temperature''']] is calculated by the Cumulus MX flavour only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like|has varied]] in different releases, but use a variation on
*[[Heat/cold_degree_days_and_Chill_hours|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
* [[Heat/cold_degree_days_and_Chill_hours|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
**A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.
**A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.


The links about will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.
The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.


There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.
* For MX, please see [[MX_Administrative_Interface#Changing_Settings]] and [[Cumulus.ini]] pages for how to find the settings
* For legacy Cumulus 1, please see [[Cumulus_Screenshots#Station|Configuration -->> Station]] menu and [[Cumulus.ini_(Cumulus_1)]] pages

=Field Count Variations=

When the [[Standard_log_files#List_of_fields_in_the_file|standard data logging]] file was introduced it only had 16 (or fewer?) fields. As time has gone by, extra fields have been added to the file. At time of writing, 29 fields have been in the file since release 3.6.12 (build 3088), and currently the "To Do" database does not include any suggestions that would add more fields.

When the [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]] file was introduced it had 15 fields. As time has gone by, extra fields have been added to the file. At release 3.6.12 there were 54 fields, but at earlier and later releases there are fewer fields. At the last update of this page (release 3.7.0) there were 52 fields. The number of fields in a line of the file might be changed in a future release.

When you use Cumulus to edit any of these files, it expects the file to have the number of fields defined in the release you are using. If an existing line in the file has fewer fields, Cumulus can still read it, but Cumulus will add trailing field separators if the file line is edited.

Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.

=Why do "Create Missing"?=

Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the [[Correcting_Extremes]] page. You are unlikely to worry about missing past values.

However, if you want to be sure that your all-time extremes, or monthly-all-time extremes, are correct, then [[Correcting_Extremes#All-time_extreme_functionality|this table]] shows how the start date for these extremes varies. You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".

If you are using the [[Highcharts_-_Historic|'''Historic Charts''']] feature introduced from release 3.9.2 - b3097 (7 December 2020), you may notice that these new charts have gaps in available data, and the dates with/without data vary depending on what is being plotted. You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".


=How to do "Create Missing"=

As mentioned earlier, there are a number of options, here are the detailed instructions for each option.

==CreateMissing.exe==

This is only available if you already have MX installed and have been using it.

Download link at [[Software#Create_Missing]] page, unzip, and install in same folder as CumulusMX.exe. The read me for this utility program is at https://github.com/cumulusmx/CreateMissing/blob/master/README.md.

Check you do not have a file called '''dayfile.txt.sav''' in your [[Data Folder|data sub-folder]]. If such a file exists, the utility program will not run.

Run the utility by changing directory to the folder where you installed it:
* On Linux operating systems, you need execute rights in that folder (prefix with '''sudo''' if you don't have rights) type <code>mono CreateMissing.exe</code>.
* On Microsoft Windows operating systems, type <code>CreateMissing</code> in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.

The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days), but as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.

This utility program looks in [[Cumulus.ini]] for:
# The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (although it can be edited to another date, such as when you imported earlier data or moved to a new home after you first used Cumulus). That will be the earliest date the utility program processes. However, if a dayfile.txt file exists and has an earlier date, then a prompt will ask you if you want to use the earlier date or the "StartDate=" date.
# The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters. This identifies which standard log lines belong to each day by checking against date and time of that line.

The utility program looks in the [[Data Folder|data sub-folder]] to see if dayfile.txt exists (if it does, its file name has a '''.sav''' suffix added after the ''.txt''). If a file with .sav suffix already exists, the utility program will stop, as it cannot create a file if it already exists.

The utility program will output to any terminal session open and to a file saved in [[MXdiags_folder|MXdiags sub-folder]].

Here is a short section of typical output (there will also be messages when each log file is opened or finished with):
<pre>
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.719 Date: 04/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 05/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 06/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 07/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 08/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 09/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 10/07/2020 : Entry is OK
2021-06-08 19:35:44.721 Date: 11/07/2020 : Adding missing data
2021-06-08 19:35:44.777 Date: 12/07/2020 : Adding missing data
2021-06-08 19:35:44.791 Date: 13/07/2020 : Adding missing data
2021-06-08 19:35:44.805 Date: 14/07/2020 : Adding missing data
2021-06-08 19:35:44.819 Date: 15/07/2020 : Adding missing data
2021-06-08 19:35:44.834 Date: 16/07/2020 : Adding missing data
2021-06-08 19:35:44.848 Date: 17/07/2020 : Adding missing data
2021-06-08 19:35:44.863 Date: 18/07/2020 : Adding missing data
2021-06-08 19:35:44.877 Date: 19/07/2020 : Adding missing data
2021-06-08 19:35:44.892 Date: 20/07/2020 : Adding missing data
2021-06-08 19:35:44.905 Date: 21/07/2020 : Adding missing data
2021-06-08 19:35:44.919 Date: 22/07/2020 : Adding missing data
2021-06-08 19:35:44.933 Date: 23/07/2020 : Adding missing data
2021-06-08 19:35:44.948 Date: 24/07/2020 : Adding missing data
2021-06-08 19:35:44.962 Date: 25/07/2020 : Adding missing data
2021-06-08 19:35:44.977 Date: 26/07/2020 : Adding missing data
2021-06-08 19:35:44.992 Date: 27/07/2020 : Adding missing data
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.007 Date: 30/07/2020 : Entry is OK
</pre>


=Calculating Missing Derived Spot Values for fields in Standard Data Logs=


==Using a PHP script on your web server==
==Using a PHP script on your web server==
Line 101: Line 209:
<div style="background: LemonChiffon;padding:5px; margin:2px;">
<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!

<big>Mark Crossley has now written a utility (download at https://github.com/cumulusmx/CreateMissing) that can update [[Standard log files]] to add missing fields.</big>
</div>
</div>
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
Line 112: Line 218:
It is a long-winded way to edit, and the MX editor does not even validate what you have entered. An alternative is to edit each log file externally, and you can read how to do that in the "Work around for standard log files" section below.
It is a long-winded way to edit, and the MX editor does not even validate what you have entered. An alternative is to edit each log file externally, and you can read how to do that in the "Work around for standard log files" section below.


Some readers of the Cumulus support log will know that a third-party replacement for the MX editor was worked on, but never got incorporated into MX. The idea was to replace the alt_editor software used by Mark Crossley, with a standard HTML form script. This allowed in-line editing, it allowed the derived values to be calculated and displayed (so you simply decided whether to accept the replacements as suggested for the various fields), and finally it applied some validation to each field to ensure any manual edit inserted a value that was within the allowed range. The main reason for its rejection from the public MX was the complex way in which different files included in the admin interface interact, and the consequent issue that changes made for this replacement had a knock on effect on other pages in the admin interface. The author could not afford the time to redesign the whole admin interface so the proposed replacement could be integrated.
<small>Some readers of the Cumulus support forum will know that a third-party replacement for the MX editor was worked on, but never got incorporated into MX. The idea was to replace the alt_editor software used by Mark Crossley, with a standard HTML form script. This allowed in-line editing, it allowed the derived values to be calculated and displayed (so you simply decided whether to accept the replacements as suggested for the various fields), and finally it applied some validation to each field to ensure any manual edit inserted a value that was within the allowed range. The main reason for its rejection from the public MX was the complex way in which different files included in the admin interface interact, and the consequent issue that changes made for this replacement had a knock on effect on other pages in the admin interface. The author could not afford the time to redesign the whole admin interface so the proposed replacement could be integrated.</small>




Line 123: Line 229:
Cumulus 1 does not provide any functionality to edit the standard data logs, whether to correct a rogue value, or to add a missing derivative.
Cumulus 1 does not provide any functionality to edit the standard data logs, whether to correct a rogue value, or to add a missing derivative.


===Work around for standard log files===
==Work around for standard log files==


As Cumulus 1 does not provide an editor, the only option is to edit the file outside Cumulus using a '''Comma Separated File''' editor, a plain text editor, or a spreadsheet program (like the free open source '''Libre Office Calc''' or the commercially charged for ''Microsoft Excel'').
An option is to edit the file outside Cumulus using a '''Comma Separated File''' editor, a plain text editor, or a spreadsheet program (like the free open source '''Libre Office Calc''' or the commercially charged for ''Microsoft Excel'').


As the log file for the current month will be updated frequently by the original Cumulus software (when it is running), Cumulus 1 applies an exclusive lock, and conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while the original Cumulus software is running. A full discussion of the problems with conflicts of access to the standard log file can be found [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 in this support forum topic].
Note: Cumulus 1 applies an exclusive lock to current standard log file, and conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while the original Cumulus software is running. A full discussion of the problems with conflicts of access to the standard log file can be found [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 in this support forum topic].


If you decide to edit the current log outside Cumulus, then remember that, if you leave Cumulus running it will continue to append new lines. Therefore, you either need to close Cumulus while you are doing the edit; or (if you are able to merge two files) close Cumulus while you replace its file with a merge of what you have edited and the extra lines added since you took away a copy to edit.
If you decide to edit the current log outside Cumulus, then remember that, if you leave Cumulus running it will continue to append new lines. Therefore, you either need to close Cumulus while you are doing the edit; or (if you are able to merge two files) close Cumulus while you replace its file with a merge of what you have edited and the extra lines added since you took away a copy to edit.
Line 145: Line 251:




===Other logs===
=Other logs=


Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if your [[dayfile.txt|daily summary log]] is missing some fields, they cannot be calculated from other fields in the same log line. The same applies to any missing extreme records for this month, this year, monthly-all-time, or all-time.
Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your [[:Category:Ini Files|.ini]] files is missing some fields, they cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.


However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.


=Missing fields (or missing lines) in dayfile.txt=
Please see [[Amending dayfile]] page for full details.


<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!

<big>Mark Crossley has now written a utility (download at https://github.com/cumulusmx/CreateMissing) that can update dayfile.txt using [[Standard log files]] as source.</big>
</div>

==Missing or Corrupted lines==

If you accidentally corrupt a line in the [[dayfile.txt|daily summary log]], then both the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|original Cumulus software]] and [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX's admin interface]] have editors for viewing and modifying content.

If you have one, or more, dates missing in your dayfile.txt file, then the next question is, '''has the line been deleted by accident?'''

If a line for a particular date was present before, but is now corrupted or missing:
#See if you have a back-up of dayfile.txt with the line present and correct
#*If it is a recent date, then Cumulus makes a backup of dayfile.txt every time it is restarted and after every end-of-day rollover
#*If it is an older date, then maybe you took a back-up onto a separate drive or separate device
#If you have a suitable backup available, take a copy of that file
#Append onto the copy, any dates after when that copy ends, taking the extra lines from the current dayfile.txt
#Rename the current dayfile.txt to say dayfile.old
#Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
#Cumulus will now use the file with all days correct

If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1. That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
#Only use the editor when Cumulus is '''not''' doing catch-up or end-of-day, or any other action that could create an editing conflict.
#With the contents of the file loaded into the editing table, click the '''Create Missing''' button
#Cumulus 1 will now work through every line from start to end
#If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
#It will read from the [[Standard log files]] all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
#*Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)

''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''

'''Another lesson here, is to periodically take a backup, stored away from your Cumulus running environment in case you ever corrupt an old line'''

Although there exists a tentative plan for an executable, to be added to the MX package, that can read the standard data logs and calculate all the fields needed for a missing date in MX's dayfile.txt, this is only a gleam in the developer's eye! At the moment, the 'Create Missing' functionality is not available to MX users. Furthermore, since Cumulus 1 can only handle dayfile.txt with 15 to 45 fields, and MX expects its dayfile.txt to have exactly 52 fields, you cannot swap files from MX back to Cumulus 1 to create the missing dates, and then back to MX again.

==Correcting individual fields in existing lines==

The "Create Missing" functionality, described in previous sub-section, is designed for when an entire line is missing for a prior date. Often people want to correct an individual field, in all lines within a particular period. This might be because you have suddenly discovered your pressure high and low are wrong by a certain offset, because you used the wrong setting for converting absolute pressure to the sea level equivalent you want to store in the file. Or it might be that you realise that MX stores 52 fields while you have old lines (perhaps from Cumulus 1) with fewer fields.

Neither Cumulus 1 nor MX provide any functionality to make such multi-line editing easy. However, the first possibility can be resolved by opening dayfile.txt in a spreadsheet (carefully ensuring each column is treated as either text (date and time fields) or number (the value fields) and that your field separator is set in the filter conditions to be a column separator. Spreadsheets have a "Paste Special" function that allows you to add/subtract a constant to every cell in a column. The extra fields in MX all store derived values, i.e. they are calculated from source values (see [[Calculate_Missing_Values#Some_definitions|Some definitions]] section earlier. If you have the correct spot derived values for the equivalent source fields in your standard data logs, then there is a work-around that can read standard log and calculate what is required to store in our daily summary file. The next two sub-sections describe how for Cumulus 1 and MX respectively.

===Correcting individual fields in multiple lines of Daily summary Log in Cumulus 1===

In the '''Edit''' menu, select '''Dayfile.txt''' screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines. If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor.

For the daily summary log, the original Cumulus software includes functionality to '''Create Missing''' in its daily summary log editor, which was described earlier. This functionality creates an entire missing log line and will insert highs and lows whether they are extremes of source values, or extremes of derived values.

This functionality is not designed to work out individual missing extreme figures, only to insert complete missing lines. However, we can workaround that constraint:

'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''

There are two common reasons for needing to use this, Cumulus 1 only, workaround:
# To add extra derived fields now calculated by Cumulus, to past lines created by an earlier version of Cumulus that did not calculate those derived fields at the time
# If you are importing into Cumulus records from other software, and that other software does not provide derived extremes (e.g. dew point, apparent temperature, heat index, rain rate, wind run) so your insert in the daily summary log file has only populated a sub-set of the fields.

The steps you need to take, to implement this Cumulus 1 only workaround:
#Select a time to do this work that is at least 10 minutes, preferably a whole hour, after Cumulus has performed a daily rollover, and ensure you finish this work long before the next rollover time. This will ensure your work cannot conflict with when Cumulus 1 is updating dayfile.txt.
# Take a copy of dayfile.txt original as backup,
# Use the Cumulus 1 editor (in '''Edit''' menu, select '''Dayfile.txt''' screen), and let it edit the original file
# Delete any days with partial information (e.g. from Cumulus versions that created fewer fields)
# Click the ''Create Missing'' button
# Wait while Cumulus scans all the standard log files, and for each source or derived field in those files, tracks the highest and lowest for each meteorological day that is now missing in dayfile.txt.
#* As an aside, be aware that when the contents of [[today.ini]] was used to create a new line in dayfile.txt originally, the high and low that were copied across were derived from all source values that Cumulus obtained from your weather station (depending on your weather station type this might be around every minute; some weather stations provide new readings less frequently, some provide them more frequently).
#*Continuing this aside, the high and low that Cumulus is now deriving are based purely on those values stored in the standard data log, by default that is just every 10 minutes.
#The new lines that Cumulus 1 adds to your dayfile.txt are also added to another log file '''dayfileeditlog.txt''', so we can access that to find the new values.
#For simplicity, click '''OK''' to save the edited '''dayfile.txt''' file, and exit the editor.
#Rename the amended dayfile.txt as "dayfile(generated).txt".
#Create an empty text file in a suitable temporary directory, and name it "dayfile.txt",
#Use an external editor to open your empty file.
#*You can use a Comma Separated Value file editor, although you will find this difficult starting with an empty file
#*You can use an editor designed for computer programmers or developers that can edit text file
#*You can use a spreadsheet application, such as "Libre Office Calc" or "Microsoft Excel", providing you obey the editing rules defined later
#Now merge in (as read-only text) the previous contents of the original dayfile.txt from the copy taken in step 2
#Now merge in (where there are fields missing) just those fields from the respective lines in "dayfile(generated).txt".
#Finally, copy your temporary "dayfile.txt" into the "Cumulus\data" folder, so from now on Cumulus accesses the file with the maximum number of fields present.

One note of caution:
*If a required source or derived value is not present in the standard data log lines, Cumulus will write into related dayfile.txt fields zero for values, and "00:00" for time stamps, because it does not understand the concept of "Null".
*Cumulus 1 does not have the functionality to retrospectively calculate derived values when it is reading source values in the standard data log line.

===Correcting individual fields in multiple lines of Daily Summary Log in Cumulus MX===



'''This section was written before Mark Crossley released the utility that works with MX log files, and updates dayfile.txt, that can be downloaded from https://github.com/cumulusmx/CreateMissing.'''


Without that utility, the only method, involves using database functionality and multiple steps:
#The first step depends on whether you use the option in MX to create and update database tables...
#*...If you don't use that option yet, use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table,
#*...Or '''if you already have the table''' and have MX updating the standard logs into that database table
#Once you have the table available, you can fairly easily update the table to add any missing values into existing columns in this 'monthly' table, and from it update columns in a daily summary table.


See [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] where there are two PHP scripts designed for this type of task.

==Rules when editing daily summary log==


===General External Editing Rules===
=General External Editing Rules=


* Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
* Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
Line 285: Line 287:
**Problems with terminating characters are normally intercepted by operating system, before the contents of a line reaches any software like Cumulus, but if partial editing or merging has produced a file with mixed line terminators, there is a high possibility this will stop any software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.
**Problems with terminating characters are normally intercepted by operating system, before the contents of a line reaches any software like Cumulus, but if partial editing or merging has produced a file with mixed line terminators, there is a high possibility this will stop any software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.
**Finally, if you are going to use a script (such as JavaScript or PHP) to attempt to read a Cumulus file, that script might only recognise a different line terminator to that your device operating system recognises (most likely with processing on a windows device, the script will treat one of the terminating characters (CR) as part of the adjacent field's text, and only treat the LF as a line terminator).
**Finally, if you are going to use a script (such as JavaScript or PHP) to attempt to read a Cumulus file, that script might only recognise a different line terminator to that your device operating system recognises (most likely with processing on a windows device, the script will treat one of the terminating characters (CR) as part of the adjacent field's text, and only treat the LF as a line terminator).


====Specific rules for dayfile.txt edits====

* The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
* All rows must start with date and include at least 14 further fields in correct sequence.
* The (meteorological) date format uses two digits for the year.
*In each line, the individual fields must be separated from each other by whatever is the list separator in your locale. Although Cumulus uses "Comma Separated Value" format, the separator does not have to a comma (,), in many locales it is a semicolon (;).
*For fields where a decimal value is permitted (see [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log page]], the separator between integer and decimal parts must consistently be either a decimal point or (only if the field separator is not a comma) a decimal comma. Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
*Rows can vary in length but only by missing off fields at the end. The minimum number of fields after the date is 14, the maximum varies between different versions.
*Each field has a pre-defined format, and the same format must always be used in that field position.
*No fields will accept letters.
* Some fields (e.g. bearings, solar, humidity) are integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
* Cumulus will only accept highest/lowest figures if each value has a related time-stamp.
** Except for wind gust (start of line) where an extra field is fitted in between value and time-stamp, each time-stamp field will immediately follow the value field for that parameter.
* Time stamp fields must always be in format HH:mm i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
** Be aware you will have problems if you, or your editing software, add seconds.
** If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
* Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
** Nulls (2 field separators without something between them, e.g.';;') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps.

[[File:Open office (editing cumulus log files).png]]
These reasons are why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed not to recognise special field types).

For spreadsheet applications (e.g. Calc in Libre Office, or on Microsoft Windows Excel) avoid using default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file. If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your application uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).

Revision as of 15:27, 9 June 2021

Crystal Clear info.png This document is 'Work In Progress' so content may not be complete.

Request for help from Wiki Readers

  • Do you understand how MX works?
  • Do you use hardware, or MX functionality, that is not yet documented? Can you begin that documenting?
  • Can you contribute simple text for novice users, examples of what you have done, correction of typing or factual errors, or supply missing details?
  • Will you make this page more useful by bringing content up-to-date as new releases change some information written for older releases?
  • Does any page need a section for novices, so they don't need to read more technical information further down that page?
  • Is there some information on this page, that should be on a separate page? Can you create the new page and move the less relevant information off this page, don't forget this page needs a link to the new page so people who expect to find it here know where it has moved to?

If you plan on contributing to the Wiki, then you will need an account.

  • Please use the Request Account form to apply for an account. Note that the Wiki is currently undergoing restructuring and is largely locked for editing, but please apply for an account if you wish to contribute in the future.
  • You will find help on how to contribute to this wiki at How to Edit.
  • If you need to consult others, please use the Cumulus Wiki suggestions forum.

Please be aware that information on this page may be incorrect.

Cumulus Version MX SpecificCumulus Version 1 SpecificThis page applies to both flavours.

How Cumulus Works

The way that Cumulus works is that:

  1. It reads what we can call source values (defined below) from your weather station (based on measurements from sensors being transmitted in some way by the station)
  2. It calculates what we can call derived values which may be either the source re-expressed in different units, or calculated by combining two or three source values to get a new derivative
  3. It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various period tracking files
    • You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the Correcting_Extremes page.
  4. It periodically stores the spot (current) derived values in a collection of Monthly_log_files,
  5. At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into daily summary log


Some definitions

To make sense of explanations on this page, you need to understand the terminology used here.

Source value

A weather station sends values based on its sensors to Cumulus. If Cumulus reports that value without changing it (an offset and/or multiplier might be applied to convert it to the unit wanted by the Cumulus User), the value is described as a source value because Cumulus is reporting something that has its source elsewhere.

There is not a single list of what weather values are called "source values", because this varies depending on the weather station, and in some cases, a Cumulus User can ask Cumulus to recalulate a value instead of using what is sent by their weather station.

However, Cumulus does include code that expects a weather station to provide a defined minimum set of source values:

  1. Current air temperature
  2. Current Relative Humidity
  3. At least one wind speed
  4. Current air pressure (absolute or sea-level)

Cumulus will stop processing any information from a weather station unless the above 4 source values are being supplied and reveal they are being updated (failure is set is after a total of 6 unsuccessful consecutive attempts to read each of these).

This requirement is a default, but it can be changed:

Cumulus also expects that your weather station can provide:

  • A rainfall counter (this could be annual rainfall, or count of rocker bucket gauge tips)

Although the lack of that rainfall counter source value will affect functionality, Cumulus will continue to process other source values that are available.

Some weather stations may also provide one, or more, of these optional source values (not a complete definitive list):

  • Dew-point Temperature
  • Wind Chill Temperature
  • Evaporation
  • Sunshine hours
  • Solar radiation
  • UV index
  • Air pollution measurement


Derived value

A dictionary will define derived as "obtained from a source", and that is the meaning adopted on this page. Steve Loft (in the Cumulus Support Forum) used the terminology "derived" for two purposes.

  1. One type of derived value takes a source value, applies any multiplier (may be both first order and second order multipliers) and/or constant that has been defined in calibration settings, and converts the output to the units selected by the Cumulus user.
  2. The other type of derived value takes more than one source value, applies a standard calculation, and ouputs a new derivative
    • Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file

"Create Missing"

This also has two meanings in a Cumulus context:

  1. If a particular standard log file line has fewer fields than the latest line;
    • Create Missing is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
  2. If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
    • Please see Amending dayfile page for full details.
    • Create Missing is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.

If you are using Cumulus MX, there is a download linked from here that does both of these. There are also editors within the admin interface for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.

If you are using the legacy Cumulus 1 software:

  1. For the standard log file meaning above, provided you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then this post in support forum includes a script that produces a HTML form where you specify the log file name you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated. Please note that script was written to run on an old version of PHP that was current at the time the script was written, it will need some editing to work on latest PHP.
  2. For the daily summary log meaning above, go to the Edit menu, and select Dayfile.txt. This brings up an editor with a button labelled "Create Missing", that will not affect any existing line, but can insert missing lines, see Amending_dayfile#Create_Missing.

Accurate or Not?

If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1. That functionality is known as "Create Missing" and is found within the edit dayfile.txt screen.

  1. Only use the editor when Cumulus is not doing catch-up or end-of-day, or any other action that could create an editing conflict.
  2. With the contents of the file loaded into the editing table, click the Create Missing button
  3. Cumulus 1 will now work through every line from start to end
  4. If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
  5. It will read from the Standard log files all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
  6. It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
    • Note that normally dayfile.txt lines are created from Today.ini which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
    • Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
    • If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
    • Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)

Derived spot values

Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.

Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):

  • Dew point, a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity
  • Wet Bulb, is only calculated by Cumulus 1, not MX
  • Wind Chill, again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
  • Canadian Humidity Index (Humidex), USA Heat Index, and Apparent Temperature are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
    • (The implementation of these by Cumulus software is briefly mentioned here).
    • The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
  • Feels Like Temperature is calculated by the Cumulus MX flavour only, the actual calculation formula has varied in different releases, but use a variation on
  • Heating Degree Days and Cooling Degree Days; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
    • A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.

The links above will take you to where the derived values are explained in the Category:Terminology pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.


There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.

Field Count Variations

When the standard data logging file was introduced it only had 16 (or fewer?) fields. As time has gone by, extra fields have been added to the file. At time of writing, 29 fields have been in the file since release 3.6.12 (build 3088), and currently the "To Do" database does not include any suggestions that would add more fields.

When the daily summary log file was introduced it had 15 fields. As time has gone by, extra fields have been added to the file. At release 3.6.12 there were 54 fields, but at earlier and later releases there are fewer fields. At the last update of this page (release 3.7.0) there were 52 fields. The number of fields in a line of the file might be changed in a future release.

When you use Cumulus to edit any of these files, it expects the file to have the number of fields defined in the release you are using. If an existing line in the file has fewer fields, Cumulus can still read it, but Cumulus will add trailing field separators if the file line is edited.

Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.

Why do "Create Missing"?

Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the Correcting_Extremes page. You are unlikely to worry about missing past values.

However, if you want to be sure that your all-time extremes, or monthly-all-time extremes, are correct, then this table shows how the start date for these extremes varies. You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".

If you are using the Historic Charts feature introduced from release 3.9.2 - b3097 (7 December 2020), you may notice that these new charts have gaps in available data, and the dates with/without data vary depending on what is being plotted. You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".


How to do "Create Missing"

As mentioned earlier, there are a number of options, here are the detailed instructions for each option.

CreateMissing.exe

This is only available if you already have MX installed and have been using it.

Download link at Software#Create_Missing page, unzip, and install in same folder as CumulusMX.exe. The read me for this utility program is at https://github.com/cumulusmx/CreateMissing/blob/master/README.md.

Check you do not have a file called dayfile.txt.sav in your data sub-folder. If such a file exists, the utility program will not run.

Run the utility by changing directory to the folder where you installed it:

  • On Linux operating systems, you need execute rights in that folder (prefix with sudo if you don't have rights) type mono CreateMissing.exe.
  • On Microsoft Windows operating systems, type CreateMissing in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.

The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days), but as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.

This utility program looks in Cumulus.ini for:

  1. The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (although it can be edited to another date, such as when you imported earlier data or moved to a new home after you first used Cumulus). That will be the earliest date the utility program processes. However, if a dayfile.txt file exists and has an earlier date, then a prompt will ask you if you want to use the earlier date or the "StartDate=" date.
  2. The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters. This identifies which standard log lines belong to each day by checking against date and time of that line.

The utility program looks in the data sub-folder to see if dayfile.txt exists (if it does, its file name has a .sav suffix added after the .txt). If a file with .sav suffix already exists, the utility program will stop, as it cannot create a file if it already exists.

The utility program will output to any terminal session open and to a file saved in MXdiags sub-folder.

Here is a short section of typical output (there will also be messages when each log file is opened or finished with):

2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.719 Date: 04/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 05/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 06/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 07/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 08/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 09/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 10/07/2020 : Entry is OK
2021-06-08 19:35:44.721 Date: 11/07/2020 : Adding missing data
2021-06-08 19:35:44.777 Date: 12/07/2020 : Adding missing data
2021-06-08 19:35:44.791 Date: 13/07/2020 : Adding missing data
2021-06-08 19:35:44.805 Date: 14/07/2020 : Adding missing data
2021-06-08 19:35:44.819 Date: 15/07/2020 : Adding missing data
2021-06-08 19:35:44.834 Date: 16/07/2020 : Adding missing data
2021-06-08 19:35:44.848 Date: 17/07/2020 : Adding missing data
2021-06-08 19:35:44.863 Date: 18/07/2020 : Adding missing data
2021-06-08 19:35:44.877 Date: 19/07/2020 : Adding missing data
2021-06-08 19:35:44.892 Date: 20/07/2020 : Adding missing data
2021-06-08 19:35:44.905 Date: 21/07/2020 : Adding missing data
2021-06-08 19:35:44.919 Date: 22/07/2020 : Adding missing data
2021-06-08 19:35:44.933 Date: 23/07/2020 : Adding missing data
2021-06-08 19:35:44.948 Date: 24/07/2020 : Adding missing data
2021-06-08 19:35:44.962 Date: 25/07/2020 : Adding missing data
2021-06-08 19:35:44.977 Date: 26/07/2020 : Adding missing data
2021-06-08 19:35:44.992 Date: 27/07/2020 : Adding missing data
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.007 Date: 30/07/2020 : Entry is OK


Using a PHP script on your web server

If you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then this post in support forum includes a script that produces a HTML form where you specify the log file name you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated.

That might sound a bit technical, so here are some step by step instructions:

  1. Download the processStandardLog.php script from here.
  2. File transfer (or copy for local web server) that script to your web server
  3. File transfer (or copy for local web server) all the standard log files from the data sub-folder in your Cumulus installation to a suitable holding folder (you may need to create it) on your web server
  4. Open in a browser PATH/processStandardLog.php by preceding the file name with the path as defined from the root on your web server
  5. This loads a web page where you have a field asking you to enter a path and file name for the data log you want to process
  6. Continue to follow the instructions on the web page
  7. When it has created a replacement file, you can enter details for another data log, and continue until all your data logs have been processed
  8. Take a backup of your existing Cumulus installation (you should be doing that on a regular basis anyway, so I will not give instructions here)
  9. Carefully delete any non-current data log in your data sub-folder that you have a replacement for, and file transfer (or copy back) the replacement data logs from your web server into the local data sub-folder, noting that the file extension will need to be renamed from .csv to .txt.

Using the data log editor provided in MX

Crystal Clear info.png This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!

In the MX admin interface go to the Data Logs menu tab, and select the Data Logs page.

There is a box for selecting the data log you want to edit. Once you have loaded that, the first (up to) 10 lines are shown. Navigation links let you select 'First', 'Previous', 'Next', and 'Last' pages, also a small number of pages can be selected directly.

Once you select a line, an Edit button is enabled, click that and you can manually input the missing values for that line. Save that edit, and you can select another line. Once you have edited all the lines on that page, you can select another page, and repeat the process. Then you can select another log, and repeat the process.

It is a long-winded way to edit, and the MX editor does not even validate what you have entered. An alternative is to edit each log file externally, and you can read how to do that in the "Work around for standard log files" section below.

Some readers of the Cumulus support forum will know that a third-party replacement for the MX editor was worked on, but never got incorporated into MX. The idea was to replace the alt_editor software used by Mark Crossley, with a standard HTML form script. This allowed in-line editing, it allowed the derived values to be calculated and displayed (so you simply decided whether to accept the replacements as suggested for the various fields), and finally it applied some validation to each field to ensure any manual edit inserted a value that was within the allowed range. The main reason for its rejection from the public MX was the complex way in which different files included in the admin interface interact, and the consequent issue that changes made for this replacement had a knock on effect on other pages in the admin interface. The author could not afford the time to redesign the whole admin interface so the proposed replacement could be integrated.


Lack of editor in Cumulus 1

Cumulus 1 provides a viewer for the data logs. On the View menu, select 'Data Logs, then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which. You cannot do any editing.

If you find that this viewer cannot load a data log, it is probably because you ignored the read me that is part of the Cumulus 1 installation procedure, see FAQ: I can't find my data files. If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.

Cumulus 1 does not provide any functionality to edit the standard data logs, whether to correct a rogue value, or to add a missing derivative.

Work around for standard log files

An option is to edit the file outside Cumulus using a Comma Separated File editor, a plain text editor, or a spreadsheet program (like the free open source Libre Office Calc or the commercially charged for Microsoft Excel).

Note: Cumulus 1 applies an exclusive lock to current standard log file, and conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while the original Cumulus software is running. A full discussion of the problems with conflicts of access to the standard log file can be found in this support forum topic.

If you decide to edit the current log outside Cumulus, then remember that, if you leave Cumulus running it will continue to append new lines. Therefore, you either need to close Cumulus while you are doing the edit; or (if you are able to merge two files) close Cumulus while you replace its file with a merge of what you have edited and the extra lines added since you took away a copy to edit.

It is best practice, to take a back-up copy of all your Cumulus installation before starting any editing. It is also best practice to take a further copy of any file you want to amend, and to do your edit on that copy, so you do not edit any Cumulus file directly. The original full backup will preserve the existing file, so you can regress to it, should Cumulus find an error in your edit.

Also note these log files do not include a header line, and should not be edited to include it. All flavours of Cumulus provide, in the data sub-folder, a file called monthlyfileheader.txt which contains the headers appropriate to the release you are running, and you can add that temporarily to your spreadsheet if it helps you with editing, but don't forget to delete it before saving the file ready to make it available to Cumulus.

Here are some other rules to follow when editing the standard log files:

  • You can't edit any log file with a word processor, as they add control characters, and other information, that Cumulus cannot understand.
  • Editing is straight forward if you use a specialised comma separated value file editor, such editors will split the content by field so it is easy to ensure you only amend field content and do not accidentally change a field divider, plus these editors will not add additional content to any line as they can cope with the number of fields varying in different lines and don't change encoding.
  • If you want to use a text editor, it is best if you choose one designed for computer programmers or developers. Such an editor will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM).
  • If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected. Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ...").
  • If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer).
  • Ensure no blank lines are introduced by your editing.
  • Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.


Other logs

Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your .ini files is missing some fields, they cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.

However, the techniques for correcting rogue values described on the Correcting_Extremes page, can be used for inserting missing values in the daily and longer period extreme records.


General External Editing Rules

  • Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
  • Take another copy and use that for your editing, don't edit the actual file being used by the software.
    • This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
    • It also means that you can go back to the last working copy, you can't upset your "revert" copy.
  • The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
  • Generally, it is easiest if you use either a specialised "Comma Separated Value" file editor or a text editor.
    • These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
  • You can use a spreadsheet application, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
    • You need to ensure that your spreadsheet treats every column as plain text, don't let it recognise dates or times and convert them into another format, don't let it convert any numeric field into another format
    • If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
  • Don't remove any figures from fields where figures currently exist, simply overtype one entry with another entry in same format.
  • If your file has previously been edited by the relevant editor in MX, a field that looks empty may actually contain one or two space characters.
  • If you are editing a field which was empty previously, remember that Cumulus does not accept the concept of nulls (entering -999 or "Null"), there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted in one field if any subsequent field in same line is not empty.
    • Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
    • Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and for up to 6 times it will repeat the last valid value if the station fails to send a value it should provide (normally six successive readings will happen between entries in the standard data log, so repeated values are less likely to affect log files).
  • The character (or in a few locales, two characters) used for separating both the day of the month from the month, and for separating the month from the year, must be consistent throughout the whole file (and must not be a single space). Normally, the separator will be either "-" or "/". Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus. Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
  • USA date format with month before day of month, and finally year, is not permitted for log files.
  • All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
  • Make sure that any editing does not create any blank lines in the file. Cumulus assumes an empty line means end of processing.
  • Don't add a header line to the file, Cumulus expects all lines to be data lines.
  • Be aware that different devices use different line terminators, so ensure that after editing a file, the line terminator is correct for the device that is running Cumulus:
    • The single character representing line feed (in most encodings, LF is binary equivalent of a decimal 10) is used for both UNIX and Linux devices (including Raspberry Pi Operating System)
    • The single character representing carriage return (in most encodings, CR is binary equivalent of a decimal 13) is used for Apple operating systems (like Mac)
    • The two character sequence first CR then LF is used to terminate lines in all Windows operating systems (part of Microsoft's determination to be different)
    • Problems with terminating characters are normally intercepted by operating system, before the contents of a line reaches any software like Cumulus, but if partial editing or merging has produced a file with mixed line terminators, there is a high possibility this will stop any software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.
    • Finally, if you are going to use a script (such as JavaScript or PHP) to attempt to read a Cumulus file, that script might only recognise a different line terminator to that your device operating system recognises (most likely with processing on a windows device, the script will treat one of the terminating characters (CR) as part of the adjacent field's text, and only treat the LF as a line terminator).