Calculate Missing Values: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search

Calculate Missing Values (view source)

Revision as of 10:33, 10 April 2022

23,252 bytes added ,  10:33, 10 April 2022
m
→‎CreateMissing.exe: That sentence still read wrongly!
No edit summary
m (→‎CreateMissing.exe: That sentence still read wrongly!)
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both flavours.[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:MX txt Files]][[Category:Cumulus 1]]
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both the legacy Cumulus 1 and to MX. It is intended that this page will cover all the different ways in which you add back any missing data. This means it should cover:
* Data that you want to add from another system for a period when you were not running Cumulus
* How Cumulus captures archive data if it is available for a period when Cumulus was stopped
* Derivatives that have been added since particular log files lines were stored
* Solving problems when a line was not successfully written to the daily summary log file
 
[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:Files_with_Comma_Separated_Values]][[Category:Cumulus 1]]


=How Cumulus Works=
=How Cumulus Works=
{{TOCright}}


The way that Cumulus works is that:
The way that Cumulus works is that:
Line 8: Line 16:
# It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various [[:Category:Ini_Files|period tracking files]]
# It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various [[:Category:Ini_Files|period tracking files]]
#* You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.
#* You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.
# It periodically stores the spot (current) derived values in a collection of [[Monthly_log_files]],
# It periodically stores the spot (current) derived values in a collection of [[Monthly_log_files]], (and so when you move to a different device, or upgrade to a new release, then providing you copy these files to the new location you will not lose any data)
#* for MX these include  [[Standard log files]],  [[Extra Sensor Files]], and [[Air_Link_Log.txt]], at time of writing
#* for MX these files include  [[Standard log files]],  [[Extra Sensor Files]], and [[Air_Link_Log.txt]], at time of writing
#* for Cumulus 1 these are [[Speciallog.txt]], [[Standard log files]], and [[Extra Sensor Files]], for all version 1.9.3 and 1.9.4 releases
#* for Cumulus 1 these files are [[Speciallog.txt]], [[Standard log files]], and [[Extra Sensor Files]], for all version 1.9.3 and 1.9.4 releases
# At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]]
# At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]]


{{TOCright}}
==Reading archive data==
 
If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.
 
This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can '''rewind''', by stopping Cumulus, optionally replacing files in [[Data folder| '''data''' folder]] with earlier copies of those files from [[Backup folder| '''backup''' folder or ''backup/daily'' folder]], and restarting Cumulus.  Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.
 
* For Cumulus 1, to stop Cumulus, you select [[Cumulus Screenshots|Exit]] from the main menu.
* For MX, how you stop MX depends on your device, and whether running as a service, please see [[MX on Linux]] or [[MX on Windows OS]] pages as appropriate for advice.
Put simply, Cumulus stores the latest time it successfully read data from the weather station in [[Today.ini|today.ini]]. When Cumulus is restarted, if it is possible to read the historic data from the weather station, then any entries between the time stored and the current time will be read.
 
With Cumulus 1, there is some dependence on weather station type, but usually two passes are made through the external logging memory, the first pass investigates what records are available in the weather station, by reading backwards in time, and the second pass works forward in past time reading and processing those records.
 
With MX, again there will be some dependence on weather station type, and the process has not been documented by the developer; it appears just a single pass is made.
 
=Importing data from other systems for periods when Cumulus was not running=
 
<big>This section needs more work on it</big>
 
Essentially, what we need to do, is to compare the format of the data we have available against the format of the relevant Cumulus file.
 
==CSV outputs from software like EasyWeather==
 
You might want to read [[EasyWeather_Format#Transferring_past_observations_from_EasyWeather.dat_to_Cumulus|Transferring_past_observations_from_EasyWeather.dat_to_Cumulus]], although that is now an obsolete article, Easy Weather has changed, and Cumulus MX is different from the original Cumulus described there.
 
The CSV output from such files has fields in the wrong order, may not match our Cumulus units, and does not have a match for each of the fields we need in our Cumulus - see [[Standard_log_files#Importing_pre-Cumulus_data|importing into standard log file fields]].  The solution is to use a CSV file editor, or a spreadsheet, to move the fields into a better order, to apply a formula where necessary to change units, and create any missing fields. 
 
Cumulus does not allow for "null", field not available, except in that it can allow shorter lines with just the fields applying to an earlier release.  For MX, [[#CreateMissing.exe|CreateMissing.exe]] can calculate some of the derivatives missing from earlier releases for the [[Standard_log_files]], and that utility can generate the necessary [[Dayfile.txt|dayfile.txt]] lines.
 
On the Support Forum - Dane - offered a translation service - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=13&t=250 here], but the last post is May 2018, so I am unsure whether it is still available as I type this in July 2021.
 
==Weather Display==
 
Please see [[Software#Weather_Display_Converter|Software page - Weather_Display_Converter]].  When you run this routine, it brings up a screen where you select the units that you want the Cumulus [[Standard_log_files]] it produces to use, as the units that weather display uses for the export are fixed.
 
The routine was written for Cumulus 1, so will not produce all the fields that MX uses, but [[#CreateMissing.exe|CreateMissing.exe]] can calculate some of the derivatives missing from earlier releases, as well as producing [[Dayfile.txt|dayfile.txt]].
 
==Weather Link log file==
 
There is a routine, see [[Software#WeatherLink_Converter|Software page - WeatherLink_Converter]] for this.  When you run this routine, it brings up a screen where you select the units.  Please note, this utility assumes the units you select on that screen apply both to the file (OR MULTIPLE FILES) being converted and to the (multiple) Cumulus [[Standard_log_files]] it produces.
 
It was written for Cumulus 1, so will not produce all the fields that MX uses, but [[#CreateMissing.exe|CreateMissing.exe]] can calculate some of the derivatives missing from earlier releases, as well as producing [[Dayfile.txt|dayfile.txt]].
 
==Weather Link Live==
 
PLEASE CAN SOMEBODY FULLY DOCUMENT THIS.
 
My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data.  I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.
 
==OTHERS????==
 
PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.


Meanwhile, people should ask for help in the support forum.


=Some definitions=
=Some definitions=
Line 24: Line 83:
A weather station sends values based on its sensors to Cumulus. If Cumulus reports that value without changing it (an offset and/or multiplier might be applied to convert it to the unit wanted by the Cumulus User), the value is described as a '''source value''' because Cumulus is reporting something that has its source elsewhere.
A weather station sends values based on its sensors to Cumulus. If Cumulus reports that value without changing it (an offset and/or multiplier might be applied to convert it to the unit wanted by the Cumulus User), the value is described as a '''source value''' because Cumulus is reporting something that has its source elsewhere.


There is not a single list of what weather values are called "source values", because this varies depending on the weather station, and in some cases, a Cumulus User can ask Cumulus to recalulate a value instead of using what is sent by their weather station.
There is not a single list of what weather values are called "source values", because this varies depending on the weather station, and in some cases, a Cumulus User can ask Cumulus to recalculate a value instead of using what is sent by their weather station.


However, Cumulus does include code that expects a weather station to provide ''a defined minimum set of source values'':
However, Cumulus does include code that expects a weather station to provide ''a defined minimum set of source values'':
Line 34: Line 93:


This requirement is a default, but it can be changed:
This requirement is a default, but it can be changed:
*For recent releases of MX, this is classified as an advance setting (see [[Cumulus.ini]])
*For recent releases of MX, this is classified as an advanced setting "No sensor check" (see [[Cumulus.ini]])
*For earlier releases of MX, this is "No sensor check" (see [[Cumulus.ini(MX_3.0.0_to_3.9.7)#Read-only_parameters_in_the_Station_section_.28releases_3.0.0_to_3.6.12.29|here for how to change default]]
*For earlier releases of MX, this is "No sensor check" (see [[Cumulus.ini (MX_3.0.0_to_3.7.0)#Read-only_parameters_in_the_Station_section_.28releases_3.0.0_to_3.6.12.29|here for how to change default]])
*For legacy Cumulus 1, [[Cumulus.ini_(Cumulus_1)#Read-only_parameters_in_the_Station_section|see here for how to change default]]
*For legacy Cumulus 1, [[Cumulus.ini_(Cumulus_1)#Read-only_parameters_in_the_Station_section|see here for how to change default]]


Line 50: Line 109:
* UV index
* UV index
* Air pollution measurement
* Air pollution measurement


==Derived value==
==Derived value==
Line 59: Line 117:
#* Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file
#* Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file


=="Create Missing"==  
=="Calculate Missing"==  


This also has two meanings in a Cumulus context:
This also has two meanings in a Cumulus context:
# If a particular standard log file line has fewer fields than the latest line;  
# If a particular standard log file line has fewer fields than the latest line;  
#* '''Create Missing''' is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
#* '''Calculate Missing''' is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
# If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
# If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
#* Please see [[Amending dayfile]] page for full details.
#* Please see [[Amending dayfile]] page for full details.
#* '''Create Missing''' is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.  
#* '''Calculate Missing''' is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.  


If you are using Cumulus MX, there is a download [[Software#Create_Missing| linked from here]] that does both of these. There are also editors within the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|admin interface]] for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.
If you are using Cumulus MX, there is a download [[Software#Create_Missing| linked from here]] that does both of these. There are also editors within the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|admin interface]] for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.
Line 76: Line 134:
===Accurate or Not?===
===Accurate or Not?===


If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1.   That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
This Wiki page describes some techniques for calculating and inserting values that are missing from standard log files and from daily summary log file.
#Only use the editor when Cumulus is '''not''' doing catch-up or end-of-day, or any other action that could create an editing conflict.
 
#With the contents of the file loaded into the editing table, click the '''Create Missing''' button
Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your [[:Category:Ini Files|.ini]] files is missing some fields, these missing fields cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.
#Cumulus 1 will now work through every line from start to end
 
#If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.
#It will read from the [[Standard log files]] all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
 
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
For the standard log files, all the fields in any one line relate to the same time, therefore for derived values calculated from other fields in the same line, you should have the same value whether it was calculated when that line was originally stored, or calculated afterwards.  I say '''should''' because the calculation formula is not always the same for all releases, in particular there are differences between how Cumulus 1 and how MX calculate some derivatives.
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
 
#*Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
For entries in today.ini, month.ini, year.ini, alltime.ini, and monthlyalltime.ini files, you don't have access to the source values used for the original calculation afterwards.
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,  
* The original values are calculated as Cumulus is running
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)
**Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini (and the other files listed) each minute if an extreme happens
** Obviously, dayfile.txt is updated from today.ini, so it is just as accurate
* Any "Calculate Missing" operation, done subsequently, does not have access to old data, it can only look in the spot values that have been logged.
**If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,  
** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)


=Derived spot values=
=Derived spot values=
Line 121: Line 183:
Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.
Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.


=Why do "Create Missing"?=
=Why do "Calculate Missing"?=


Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the [[Correcting_Extremes]] page.  You are unlikely to worry about missing past values.
Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the [[Correcting_Extremes]] page.  You are unlikely to worry about missing past values.
Line 130: Line 192:




=How to do "Create Missing"=
=How to do "Calculate Missing"=


As mentioned earlier, there are a number of options, here are the detailed instructions for each option.
As mentioned earlier, there are a number of options, here are the detailed instructions for each option.
Line 136: Line 198:
==CreateMissing.exe==
==CreateMissing.exe==


This is only available if you already have MX installed and have been using it.  
This utility is only for those who have already installed Cumulus MX:
* It uses some files that are in the MX installation package
* It uses some files that are created when MX is run
 
 
''Please note the developer does not fully describe his utility at [https://github.com/cumulusmx/CreateMissing/blob/master/README.md his github page] so the author of this Wiki update cannot guarantee the documentation here is correct''


Download link at [[Software#Create_Missing]] page, unzip, and install in same folder as CumulusMX.exe. The read me for this utility program is at https://github.com/cumulusmx/CreateMissing/blob/master/README.md.
This utility was written by Mark Crossley specifically to insert any missing data:
* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder|data sub-folder]]
** The utility will not run if a file called  '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once
* If any of the following fields are not populated in [[Standard_log_files#List of fields in the file|MMMYYlog.txt]] lines, the utility can populate them as it works through that file to work out daily extremes (see calculations in table below).
** [[Wind chill|Wind Chill]]
** [[Apparent_temperature|Australian Apparent Temperature]]
** [[Feels Like|Feels Like temperature]]
** [[Heat index|North American Heat Index]]
** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below


Check you do not have a file called '''dayfile.txt.sav''' in your [[Data Folder|data sub-folder]]. If such a file exists, the utility program will not run.


Run the utility by changing directory to the folder where you installed it:
===Obtaining the Create Missing Utility===
* On Linux operating systems, you need execute rights in that folder (prefix with '''sudo''' if you don't have rights) type <code>mono CreateMissing.exe</code>.
* On Microsoft Windows operating systems, type <code>CreateMissing</code> in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.


The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days), but as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.
There is a download link on [[Software#Create_Missing]] page, unzip to reveal the components in the package, and install all of those in same folder as CumulusMX.exe.
 
If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the '''CreateMissing.exe''' file may need to be given execute access (see [[MX_on_Linux#chmod]])
 
Please note that  "Create Missing Version 1.1.0" can only be used with "Cumulus MX release 3.12.0" and above. It makes use of components not included in earlier MX releases.
 
Earlier "Create Missing" versions can be found at https://github.com/cumulusmx/CreateMissing, and these will work with earlier MX releases, but be aware that Create Missing version 1.0.2 had been tested and was found to have a bug.
 
===Running the Create Missing Utility===
 
The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days).
 
To run CumulusMX.exe needs .NET or MONO to be installed, and because MX is already running, CreateMissing.exe can be run, as it too requires .NET or MONO.
 
# Open up the MX interface in a browser, and navigate to  Settings menu -> Station Settings -> General Settings -> Advanced Options -> Records Began Date
#* The date that is shown there is the date where "Create Missing" will start by default, so if you have MMMYYlog.txt log files with an earlier date, edit the date here, keeping to same format
#* Click '''Save Settings''' button if you have made a change to this date
# Close your browser, and (if using an interactive screen for your computer) open up your file manager, or (if using a terminal session for access to your computer) navigate to your CumulusMX folder
# Navigate to your [[Data_folder|data sub-folder]]
# If there is a file there called '''dayfile.txt.sav''', rename that file to '''dayfile.txt.sav.bak''' (or any other name that does not already exist)
# Now change directory back up to parent folder <code>cd ..</code>, i.e the folder containing "CreateMissing.exe"
#* On Linux operating systems, you need execute rights on that file (prefix with '''sudo''' if you don't have rights) type <code>mono CreateMissing.exe</code>.
#* On Microsoft Windows operating systems, type <code>CreateMissing</code> in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.
 
Remember, as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.
 
===How the Create Missing Utility works===
 
The utility program will output to any terminal session open and to a file saved in [[MXdiags_folder|MXdiags sub-folder]].


This utility program looks in [[Cumulus.ini]] for:  
This utility program looks in [[Cumulus.ini]] for:  
# The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (although it can be edited to another date, such as when you imported earlier data or moved to a new home after you first used Cumulus). That will be the earliest date the utility program processes. However, if a dayfile.txt file exists and has an earlier date, then a prompt will ask you if you want to use the earlier date or the "StartDate=" date.
# The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (as mentioned above it can be edited to another date, to include imported earlier data or to exclude data that relates to a former location).  
# The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters. This identifies which standard log lines belong to each day by checking against date and time of that line.
#* That will be the earliest date the utility program processes.  
#* However, if a dayfile.txt file exists and that has an earlier date, then "Create Missing" will only continue if you accept that earlier date.
# The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters.  
#* This identifies which standard log lines belong to each day by checking against date and time of that line.
# The thresholds for Heating Degree Days, Cooling Degree Days, and Chill Hours
# The starting month for Chill Hours Season


The utility program looks in the [[Data Folder|data sub-folder]] to see if dayfile.txt exists (if it does, its file name has a '''.sav''' suffix added after the ''.txt''). If a file with .sav suffix already exists, the utility program will stop, as it cannot create a file if it already exists.
This utility program looks in the [[Data_folder|data sub-folder]]:
# If there is a file there called '''dayfile.txt.sav''', the utility aborts
# If there is a file there called '''dayfile.txt''', the utility renames that to ''dayfile.txt.sav''
# The utility creates an empty file, naming it '''dayfile.txt'''
# The utility continues by opening the MMMYYlog.txt file for the earliest date (see above) it is going to process
#* See table below for how contents of this file are read/updated and used to create lines in the new '''dayfile.txt''' file
# Each subsequent  MMMYYlog.txt file is processed in turn


The utility program will output to any terminal session open and to a file saved in [[MXdiags_folder|MXdiags sub-folder]].
===How the utility reports progress===


Here is a short section of typical output (there will also be messages when each log file is opened or finished with):
Here is a short section of typical output (from version 1.0.2 that had a bug and never processed 1st day of month):
<pre>
<pre>
2021-06-08 19:35:44.108 Loading log file - data/Jul20log.txt
2021-06-08 19:35:44.191 01/07/2020 : No monthly data was found, not updating this record
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
Line 186: Line 301:
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.007 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.141 Date: 31/07/2020 : Adding missing data
2021-06-08 19:35:45.156 Finished processing log file - data/Jul20log.txt
2021-06-08 19:35:45.156 Loading log file - data/Aug20log.txt
</pre>
</pre>


=== How the utility creates a dayfile.txt line ===
{| class="wikitable" border="1"
|-
!style="background:pink; width:250px" | dayfile.txt field
|colspan="4" style="background:lightgray; width:150px" | Standard log file fields
!style="background:pink; width:400px" | Description
|-
|style="background:pink;"| Daily derivative
|style="background:lightgray;"| Preferred field
|style="background:lightgray;"| First source
|style="background:lightgray;"| Second source
|style="background:lightgray;"| Third source
|style="background:pink;"| (how calculated)
|-
| [[Meteorological_day|date]]
|
| Day-Month-Year
| Hour-Minute
|
| From processing lines linked with that Meteorological day.
|-
|Highest wind [[Wind_measurement#Weather_Stations_and_Cumulus|gust]] speed
| colspan="4" | Cumulus '''Gust''' wind speed
| Stores highest value of that log file field in that Meteorological day.
|-
|[[Wind_measurement#Wind_Direction | Bearing]] of highest wind gust
| colspan="4" | Average wind bearing (in degrees)
| Stores the bearing recorded at same time as maximum value in previous field
|-
|Time of highest wind gust
| colspan="4" | Hour-Minute
| Stores the time in log file line used in two previous fields
|-
|Minimum [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | temperature]]
| colspan="4" | Current temperature
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of minimum temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum temperature
| colspan="4" | Current temperature
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Minimum [[Pressure_Measurement | sea level pressure]]
| colspan="4" | Current sea level pressure
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of minimum pressure
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum sea level pressure
| colspan="4" | Current sea level pressure
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum pressure
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum [[Rain_measurement#Rain_Rate | rainfall rate]]
| colspan="4" | [[FAQ#How_is_my_rain_rate_calculated.3F | Current rainfall rate]]
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum rainfall rate
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Total rainfall for the day
| colspan="4" | Total rainfall today so far
| Stores the entry in the last log file field in that Meteorological day.
|-
|[[Average temperature]] for the day
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in minutes obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out product of above interval time times "Current temperature" field
# Sum the interval times in step 1 for whole day
# Sum the products in step 2 for whole day
# When completed loop, store the sum in step 3 divided by the sum in step 4
|-
|Daily [[Windrun | wind run]]
|
| Hour-Minute
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out product of above interval time times "Current average wind speed" field
# Sum the products in step 2 for whole day
# When completed loop, store the sum in step 3
|-
|Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]]
| colspan="4" | Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of Highest Avg. Wind speed
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Lowest [[Temperature_(and_humidity)_measurement | humidity]]
| colspan="4" | Current [http://en.wikipedia.org/wiki/Relative_humidity relative humidity]
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of lowest humidity
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Highest humidity
| colspan="4" | Current relative humidity
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of highest humidity
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Total evapotranspiration
| colspan="4" | Evapotranspiration
| Stores highest value of that log file field in that Meteorological day.
|-
|Total hours of sunshine
| colspan="4" | Hours of sunshine so far today
| Stores highest value of that log file field in that '''calendar''' day (i.e. midnight to midnight)
|-
|High USA [[Heat index]]
| Heat Index
| Current relative humidity
| Current temperature
|
| The heat index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
| Time of high heat index
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
| High [[Apparent temperature]]
| Apparent temperature
| Current relative humidity
| Current temperature
|
| Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high apparent temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low apparent temperature
| Apparent temperature
| Current relative humidity
| Current temperature
|
| Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
|-
|Time of low apparent temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High hourly rain
| colspan="4" | Total rainfall today so far
| High hourly rain is a derived value.  Loop through every log file field in that Meteorological day, build up a series of hourly values (total rainfall in this entry minus total rainfall an hour earlier), find maximum of all those hourly values, and store that.
|-
|Time of high hourly rain
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Greatest [[wind chill]] (high wind speed, low temperature)
| Wind chill
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
|
| Wind Chill can be reported by weather station or it can be derived. If the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of greatest wind chill
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]]
| colspan="4" | Current dew point
| Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores highest value of that log file field in that Meteorological day.
|-
|Time of high dew point
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low dew point
| colspan="4" | Current dew point
| Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores lowest value of that log file field in that Meteorological day.
|-
|Time of low dew point
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Today's dominant/average wind direction
|
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Average wind bearing (in degrees)
|
| The dominant/average wind direction is a derived value. 
# Loop through every log file pair of fields in that Meteorological day:
#* Calculate increment in X as product of wind speed times sine of bearing, and sum those increments
#* Calculate increment in Y as product of wind speed times cosine of bearing, and sum those increments
# Convert final X and Y coordinates back to a bearing in degrees
|-
|[[Heat/cold degree days and Chill hours | Heating degree days]] (HDD)
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out increment in HDD by subtracting current temperature from  HDD threshold
# Work out product multiplying result in step 1 by result in step 2, and sum those products
# At end of loop store the final sum
|-
|[[Heat/cold degree days and Chill hours | Cooling degree days]] (CDD)
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out increment in HDD by subtracting CDD threshold from current temperature
# Work out product multiplying result in step 1 by result in step 2, and sum those products
# At end of loop store the final sum
|-
|High solar radiation
| colspan="4" | current solar radiation
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of high solar radiation
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High UV Index
| colspan="4" | UV Index
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of high UV Index
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High [[Feels Like]] temperature
| Feels Like temperature
| Current relative humidity
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
| Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high feels like temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low Feels Like temperature
| Feels Like temperature
| Current relative humidity
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
| Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
|-
|Time of low feels like temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High Canadian Humidity Index or [[Humidex]]
| Humidex
| Current relative humidity
| Current temperature
|
| The Canadian Humidity index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high Humidex
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
| Cumulative Seasonal Chill Hours
|
| Current temperature
| Hour-Minute
|
| "Chill Hours" is a derived value, loop through every log file field in that Meteorological day:
# Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out if there is increment in Chill hours by seeing if "Current temperature" field is below Chill Hours threshold temperature
# If there is an increment, sum value from step 1
# At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day
|}


==Using a PHP script on your web server==
==Using a PHP script on your web server==
Line 209: Line 622:
<div style="background: LemonChiffon;padding:5px; margin:2px;">
<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software.  It has been updated to cover MX, but that was for a MX release that is no longer latest!
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software.  It has been updated to cover MX, but that was for a MX release that is no longer latest!
When this section was written, the number of lines shown was fixed at a maximum of 10; later releases have given the option to display different numbers of lines, and there may be other changes still to be documented here.
</div>
</div>
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
Line 223: Line 638:
==Lack of editor in Cumulus 1==
==Lack of editor in Cumulus 1==


Cumulus 1 provides a [[Cumulus_Screenshots#View_Menu|viewer]] for the data logs. On the '''View''' menu, select ''''Data Logs''', then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which.  You cannot do any editing.
Cumulus 1 provides a [[Cumulus_Screenshots#View_Menu|viewer]] for the data logs, that does not permit editing of the file.  
 
On the '''View''' menu, select ''''Data Logs''', then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which.  You cannot do any editing.


If you find that this viewer cannot load a data log, it is probably because you ignored the '''read me''' that is part of the Cumulus 1 installation procedure, see  [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]].  If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.
If you find that this viewer cannot load a data log, it is probably because you ignored the '''read me''' that is part of the Cumulus 1 installation procedure, see  [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]].  If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.
Line 249: Line 666:
* Ensure no blank lines are introduced by your editing.
* Ensure no blank lines are introduced by your editing.
*Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.
*Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.
=Other logs=
Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your [[:Category:Ini Files|.ini]] files is missing some fields, they cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.




Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/9113...10319"

Navigation menu