Calculate Missing Values: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search

Calculate Missing Values (view source)

Revision as of 11:36, 1 October 2022

21,398 bytes added ,  11:36, 1 October 2022
m
→‎Derived spot values
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both flavours.[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:MX txt Files]][[Category:Cumulus 1]]
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both the legacy Cumulus 1 and to MX. It is intended that this page will cover all the different ways in which you add back any missing data. This means it should cover:
* Data that you want to add from another system for a period when you were not running Cumulus
* How Cumulus captures archive data if it is available for a period when Cumulus was stopped
* Derivatives that have been added since particular log files lines were stored
* Solving problems when a line was not successfully written to the daily summary log file
 
[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:Files_with_Comma_Separated_Values]][[Category:Cumulus 1]]


=How Cumulus Works=
=How Cumulus Works=
Line 10: Line 16:
# It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various [[:Category:Ini_Files|period tracking files]]
# It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various [[:Category:Ini_Files|period tracking files]]
#* You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.
#* You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.
# It periodically stores the spot (current) derived values in a collection of [[Monthly_log_files]],
# It periodically stores the spot (current) derived values in a collection of [[Monthly_log_files]], (and so when you move to a different device, or upgrade to a new release, then providing you copy these files to the new location you will not lose any data)
#* for MX these include  [[Standard log files]],  [[Extra Sensor Files]], and [[Air_Link_Log.txt]], at time of writing
#* for MX these files include  [[Standard log files]],  [[Extra Sensor Files]], and [[Air_Link_Log.txt]], at time of writing
#* for Cumulus 1 these are [[Speciallog.txt]], [[Standard log files]], and [[Extra Sensor Files]], for all version 1.9.3 and 1.9.4 releases
#* for Cumulus 1 these files are [[Speciallog.txt]], [[Standard log files]], and [[Extra Sensor Files]], for all version 1.9.3 and 1.9.4 releases
# At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]]
# At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]]


Line 19: Line 25:
If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.
If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.


This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can '''rewind''', by stopping Cumulus and restarting.  Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.
This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can '''rewind''', by stopping Cumulus, optionally replacing files in [[Data folder| '''data''' folder]] with earlier copies of those files from [[Backup folder| '''backup''' folder or ''backup/daily'' folder]], and restarting Cumulus.  Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.


* For Cumulus 1, to stop Cumulus, you select [[Cumulus Screenshots|Exit]] from the main menu.
* For Cumulus 1, to stop Cumulus, you select [[Cumulus Screenshots|Exit]] from the main menu.
Line 63: Line 69:
My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data.  I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.
My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data.  I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.


==OTHERS????===
==OTHERS????==


PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.
PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.


 
Meanwhile, people should ask for help in the support forum.


=Some definitions=
=Some definitions=
Line 144: Line 150:
** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)
** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)


=Derived spot values=
= Derived spot values =


Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.
Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.


Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):
Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):
* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity
* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity. The original legacy Cumulus 1, and CumulusMX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by CumulusMX.  
** The original legacy Cumulus, and MX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by MX.  
* '''Wet Bulb''', is not calculated by CumulusMX
* '''Wet Bulb''', is only calculated by Cumulus 1, not MX
* [[Wind_chill|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
* [[Wind_chill|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
* [[Humidex|'''Canadian Humidity Index (Humidex)''']], [[Heat_index|'''USA Heat Index''']], and [[Apparent_temperature|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
* [[Humidex|'''Canadian Humidity Index (Humidex)''']], [[Heat_index|'''USA Heat Index''']], and [[Apparent_temperature|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
** (The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like|here]]).  
** The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like|here]].  
** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
* [[Feels_Like|'''Feels Like Temperature''']] is calculated by the Cumulus MX flavour only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like|has varied]] in different releases, but use a variation on
* [[Feels_Like|'''Feels Like Temperature''']] is calculated by CumulusMX only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like|has varied]] in different releases.
* [[Heat/cold_degree_days_and_Chill_hours|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
* [[Heat/cold_degree_days_and_Chill_hours|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
**A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.


The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.
The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.


 
There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature, please see the [[MX_Administrative_Interface#Changing_Settings|interface]] and [[Cumulus.ini]] pages for how to find the settings.
There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.
* For MX, please see [[MX_Administrative_Interface#Changing_Settings]] and [[Cumulus.ini]] pages for how to find the settings
* For legacy Cumulus 1, please see [[Cumulus_Screenshots#Station|Configuration -->> Station]] menu and [[Cumulus.ini_(Cumulus_1)]] pages


=Field Count Variations=
=Field Count Variations=
Line 192: Line 193:
==CreateMissing.exe==
==CreateMissing.exe==


This is only available if you already have MX installed and have been using it.  
Mark Crossley has written a utility that can add any of the following fields to your [[Standard_log_files#List of fields in the file|MMMYYlog.txt]] lines:
* [[Wind chill|Wind Chill]]
* [[Apparent_temperature|Australian Apparent Temperature]]
* [[Feels Like|Feels Like temperature]]
* [[Heat index|North American Heat Index]]
* [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
* [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
 
The primary purpose of the utility is however to create a new ''dayfile.txt'' populating each field as summarised in table below.
 
''Please note the developer does not fully describe his utility at [https://github.com/cumulusmx/CreateMissing/blob/master/README.md his github page] so the author of this Wiki update cannot guarantee the detailed documentation here is correct''
 
 
===Requirements for using Create Missing utility===
 
This utility is only for those who have already installed Cumulus MX:
* It uses some [[Software#By_Mark_Crossley|.dll files]] that are in the MX installation package referenced beside the [https://github.com/cumulusmx/CreateMissing/releases release download]
* Each "CreateMissing.exe" release relates to the fields in [[dayfile.txt]] at a particular "CumulusMX.exe" release (see [[Software#Create_Missing]] page)
* It uses some information in [[Cumulus.ini]] a file created when MX is run
 
Although the developer works on "CreateMissing.exe" independently of "CumulusMX.exe" code, so they can get out of step, normally the latest release of the former works with the latest release of the latter.
 
In a Microsoft Windows environment, Create Missing uses the .NET software that is normally already available.
 
In a UNIX-derived operating system (e.g. a computer running Linux or Raspberry Pi operating system), there is a need to [[Preparing_your_Linux_computer_for_MX#Checking_if_mono-complete_is_in_Source_List|install MONO]] to run either "CreateMissing.exe" or "CumulusMX.exe".
 
 
 
===Obtaining the Create Missing Utility===
 
There is a download link on [[Software#Create_Missing]] page for the latest release of the utility. The MX 3.20.0 release zip also includes a copy of the utility; it was not included in release packages for earlier MX releases, and I can't predict whether it will be included in any subsequent MX release.
 
All releases are available at [https://github.com/cumulusmx/CreateMissing/releases Github release download], but be aware that Create Missing version 1.0.2 definitely had a bug as mentioned by developer in the forum, and the forum reports issues found by some users over handling the first day of any month with some other releases. Each release works only with specific "CumulusMX.exe" releases as described (with links to details) in previous subsection.
 
# Run whatever package your computer uses to extract/unzip packages.
# The package should list 2 or 3 components found in the zip file
# Extract those components to a new folder in your download area
# Copy/install the "CreateMissing.exe" and "CreateMissing.exe.config" into same folder as CumulusMX.exe.
# If there is a "Updates.txt" included, you will need to rename that file before copying as it conflicts with a file of that same name issued with "CumulusMX.exe" and "ExportToMySQL.exe" release downloads.
 
If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the '''CreateMissing.exe''' file may need to be given execute access (see [[Preparing_your_Linux_computer_for_MX#chmod]]).
 
===Preparing to run the Create Missing Utility===
 
# Open up the MX interface in a browser, and navigate to  ''Settings menu -> Station Settings -> General Settings -> Advanced Options -> Records Began Date''
#* The date that is shown there is the date where "Create Missing" will start by default, so if you have MMMYYlog.txt log files with an earlier date, edit the date here
#*# Ensure any new date you enter there uses exactly the same format as the date that was there
#*# Click '''Save Settings''' button if you have made a change to this date
# Close your browser, and (if using an interactive screen for your computer) open up your file manager, or (if using a terminal session for access to your computer) navigate to your CumulusMX folder
# Navigate to your [[Data_folder|data sub-folder]]
# If there is a file there called '''dayfile.txt.sav''', rename that file to '''dayfile.txt.sav.bak''' (or any other name that does not already exist)
 
Optionally, you may wish to take a backup of the existing contents of the "data" directory onto a separate storage device, because the utility edits files in there, and if something goes wrong mid-edit (e.g. power cut, storage device failure) you could lose your valuable data.
 
 
===Running the Create Missing Utility===
 
The utility can be run while "CumulusMX.exe" is running (see below) either interactively or as a service, or while MX is stopped.
 
'''On Microsoft Windows operating systems''':
# First change the path to your Cumulus MX root folder
# Start a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon)
# Now type <code>CreateMissing</code>  (or type "CreateMissing.exe", both will work)
 
 
'''On Unix-derived operating systems (such as Linux)''':
# First change your command line path to your Cumulus MX root folder (cd CHOSEN PATH/CumulusMX)
# Now type <code>sudo mono CreateMissing.exe</code>
#* If the user you are using already has execute access it is possible to leave out 'sudo' (the present writer has not tested this)
#* If mono is already running, it has been suggested it might be possible to leave out 'mono' (the present writer has not tested this)
 
 
''It is important to understand'' that when you start MX interactively, or as a service, it reads the contents of '''dayfile.txt''' into an internal array (held in random access memory - RAM).
* The only other time that  'CumulusMX.exe'' accesses the  '''dayfile.txt''' is at rollover when it copies what it generates internally for the day just ended into the file
* You should never run '''CreateMissing.exe''' near to your rollover time
* At any other time, if you run '''CreateMissing.exe''' while ''CumulusMX.exe'' is running, any updates to the file are ''not'' seen by MX until that software is restarted. 
* Release 3.20.0 (beta build 3199 onwards) adds a new "Utils" menu with a new option to refresh the internally held values without restarting MX.
* The internally held values, not the contents of '''dayfile.txt''' itself, are displayed when you use any of the [[Correcting_Extremes|extreme record editors]]
 
 
===How the Create Missing Utility works===
 
The utility program will output to any terminal session open and also (with further detail) to a file saved in [[#How the utility reports progress|MXdiags directory]].
 
This utility program looks in [[Cumulus.ini]] for:
# The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (as mentioned above it can be edited to another date, to include imported earlier data or to exclude data that relates to a former location).
#* That will be the earliest date the utility program processes.
#* However, if a dayfile.txt file exists and that has an earlier date, then "Create Missing" will only continue if you accept that earlier date.
# The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters.
#* This identifies which standard log lines belong to each day by checking against date and time of that line.
# The thresholds for Heating Degree Days, Cooling Degree Days, and Chill Hours
# The starting month for Chill Hours Season
 
This utility program looks in the [[Data_folder|data sub-folder]]:
# If there is a file there called '''dayfile.txt.sav''', the utility aborts
# If there is a file there called '''dayfile.txt''', the utility renames that to ''dayfile.txt.sav''
 


These notes relate to "Create Missing Version 1.1.0" which is for "Cumulus MX release 3.12.0" and above. At the time of amending this section Create Missing version 1.0.2 had been tested and was found to have a bug.
The sequence followed by the utility is:
# The utility will not run if a file called '''dayfile.txt.sav''' already exists in [[data folder]]
# The utility reads [[Cumulus.ini]] and displays the 'record start date' it finds
# It asks if this date is correct
# If you answer with anything other than 'Y' or 'y', the utility aborts
# If you answer with 'Y' or 'y', the utility continues
# It creates an internal array to hold the equivalent of '''dayfile.txt''' in random access memory - RAM
# If there is an existing [[dayfile.txt]], its contents are used to populate that two-dimensional array
# It renames any existing 'dayfile.txt' to '''dayfile.txt.sav''' in your [[Data_folder|data sub-folder]]
# It then works through the MMMYYlog.txt files that have lines for dates after that 'record start date'
# For each of these [[Standard log files]] there is another sequence:
## The inner sequence involves opening the file (the log in MXDiags records when each file is opened, files may be opened more than once)
## The dates being processed are shown in the terminal window; as each output is followed by just a "Line Feed" character the lines will overwrite each other in a Microsoft Windows environment (which expects "Carriage Return" and "Line Feed" in sequence to terminate a line), will be on successive lines in a Linux environment (where the normal line terminator is "Line Feed"), and will be on one long line in a Mac operating system (where the normal line terminator is "Carriage Return")
## If a line read in the source file does not include a particular [[#Derived_value|derived value]] (see list [[#CreateMissing.exe|above]]), then the utility will calculate that required value from the [[#Source_value|spot source field values]] in the same line, and update that line
## The exact calculation done for each item in dayfile.txt is listed in the table [[#How the utility creates a dayfile.txt line]] below, but here are the basic principles
## For solar data, it examines lines in the file with times between two successive midnights
##* If the array already holds a "sun hours" figure for the date quoted in those lines, the utility skips to next midnight to midnight period
##* If the array is missing a "sun hours" figure for the date quoted in those lines, the utility stores the sun hours recorded in last entry of that period in the part of the internal array with matching date
## From release 1.3.0, CreateMissing.exe will also calculate the maximum rainfall in a 24-hour period; to do this it looks at rain counter for the line in the source file it is currently processing, and attempts to find rain counter in another line nearest to 24 hours later (if the time interval between the lines is not exactly 24 hours then the difference figure reported will be for whatever period is available) looking up the rain counter there; it will track the maximum difference ending on a date that date determines where in the array the figure is stored; if the difference in rain counter yields a negative figure it is ignored
## For other daily data, the lines examined will start at rollover time and continue until just before the next rollover time (with allowance for any DST change)
##* If the daily data is an extreme, then the highest or lowest (source or derived) value seen in that range of lines is added to the array if an entry does not already exist
##* If the daily data is cumulative (over a day or over a season), then if the internal array does not already have a value, that from the last value in the range of lines is stored
##* If the daily data is not directly related to fields held in the source file (e.g. cumulative chill hours), then it will be calculated from what is available (for chill hours, if the recorded temperature in line being examined is below threshold, then the time interval passed since the previous entry is added to an internally held count; note this MX calculation is slightly different to Steve Loft approach that worked on average of temperature in a particular line and the temperature in previous line being below threshold)
## When all lines in a particular file have been examined, that file is closed (the log in MXDiags records when each file is closed, files may be closed more than once)
## The next file (chronologically) is opened (the log in MXDiags records when each file is opened, files may be opened more than once) and the inner sequence continues
# When all the source files have been processed, the utility continues
# The utility creates a new file, naming it '''dayfile.txt'''
# The utility copies what is stored in its internal array to the new file
# To prevent the terminal screen closing, the utility ends with a "Press any key to continue" prompt


Download link at [[Software#Create_Missing]] page, unzip, and install in same folder as CumulusMX.exe.  The read me for this utility program is at https://github.com/cumulusmx/CreateMissing/blob/master/README.md.


Check you do not have a file called '''dayfile.txt.sav''' in your [[Data folder|data sub-folder]]. If such a file exists, the utility program will not run.


Run the utility by changing directory to the folder where you installed it:
=== How the utility creates a dayfile.txt line ===
* On Linux operating systems, you need execute rights in that folder (prefix with '''sudo''' if you don't have rights) type <code>mono CreateMissing.exe</code>.
{| class="wikitable" border="1"
* On Microsoft Windows operating systems, type <code>CreateMissing</code> in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.
|-
!style="background:pink; width:250px" | dayfile.txt field
|colspan="4" style="background:lightgray; width:150px" | Standard log file fields
!style="background:pink; width:400px" | Description
|-
|style="background:pink;"| Daily derivative
|style="background:lightgray;"| Preferred field
|style="background:lightgray;"| First source
|style="background:lightgray;"| Second source
|style="background:lightgray;"| Third source
|style="background:pink;"| (how calculated)
|-
| [[Meteorological_day|date]]
|
| Day-Month-Year
| Hour-Minute
|
| From processing lines linked with that Meteorological day.
|-
|Highest wind [[Wind_measurement#Weather_Stations_and_Cumulus|gust]] speed
| colspan="4" | Cumulus '''Gust''' wind speed
| Stores highest value of that log file field in that Meteorological day.
|-
|[[Wind_measurement#Wind_Direction | Bearing]] of highest wind gust
| colspan="4" | Average wind bearing (in degrees)
| Stores the bearing recorded at same time as maximum value in previous field
|-
|Time of highest wind gust
| colspan="4" | Hour-Minute
| Stores the time in log file line used in two previous fields
|-
|Minimum [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | temperature]]
| colspan="4" | Current temperature
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of minimum temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum temperature
| colspan="4" | Current temperature
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Minimum [[Pressure_Measurement | sea level pressure]]
| colspan="4" | Current sea level pressure
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of minimum pressure
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum sea level pressure
| colspan="4" | Current sea level pressure
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum pressure
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum [[Rain_measurement#Rain_Rate | rainfall rate]]
| colspan="4" | [[FAQ#How_is_my_rain_rate_calculated.3F | Current rainfall rate]]
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum rainfall rate
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Total rainfall for the day
| colspan="4" | Total rainfall today so far
| Stores the entry in the last log file field in that Meteorological day.
|-
|[[Average temperature]] for the day
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in minutes obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out product of above interval time times "Current temperature" field
# Sum the interval times in step 1 for whole day
# Sum the products in step 2 for whole day
# When completed loop, store the sum in step 3 divided by the sum in step 4
|-
|Daily [[Windrun | wind run]]
|
| Hour-Minute
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out product of above interval time times "Current average wind speed" field
# Sum the products in step 2 for whole day
# When completed loop, store the sum in step 3
|-
|Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]]
| colspan="4" | Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of Highest Avg. Wind speed
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Lowest [[Temperature_(and_humidity)_measurement | humidity]]
| colspan="4" | Current [http://en.wikipedia.org/wiki/Relative_humidity relative humidity]
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of lowest humidity
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Highest humidity
| colspan="4" | Current relative humidity
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of highest humidity
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Total evapotranspiration
| colspan="4" | Evapotranspiration
| Stores highest value of that log file field in that Meteorological day.
|-
|Total hours of sunshine
| colspan="4" | Hours of sunshine so far today
| Stores highest value of that log file field in that '''calendar''' day (i.e. midnight to midnight)
|-
|High USA [[Heat index]]
| Heat Index
| Current relative humidity
| Current temperature
|
| The heat index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
| Time of high heat index
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
| High [[Apparent temperature]]
| Apparent temperature
| Current relative humidity
| Current temperature
|
| Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high apparent temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low apparent temperature
| Apparent temperature
| Current relative humidity
| Current temperature
|
| Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
|-
|Time of low apparent temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High hourly rain
| colspan="4" | Total rainfall today so far
| High hourly rain is a derived value.  Loop through every log file field in that Meteorological day, build up a series of hourly values (total rainfall in this entry minus total rainfall an hour earlier), find maximum of all those hourly values, and store that.
|-
|Time of high hourly rain
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Greatest [[wind chill]] (high wind speed, low temperature)
| Wind chill
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
|
| Wind Chill can be reported by weather station or it can be derived. If the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of greatest wind chill
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]]
| colspan="4" | Current dew point
| Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores highest value of that log file field in that Meteorological day.
|-
|Time of high dew point
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low dew point
| colspan="4" | Current dew point
| Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores lowest value of that log file field in that Meteorological day.
|-
|Time of low dew point
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Today's dominant/average wind direction
|
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Average wind bearing (in degrees)
|
| The dominant/average wind direction is a derived value.
# Loop through every log file pair of fields in that Meteorological day:
#* Calculate increment in X as product of wind speed times sine of bearing, and sum those increments
#* Calculate increment in Y as product of wind speed times cosine of bearing, and sum those increments
# Convert final X and Y coordinates back to a bearing in degrees
|-
|[[Heat/cold degree days and Chill hours | Heating degree days]] (HDD)
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out increment in HDD by subtracting current temperature from  HDD threshold
# Work out product multiplying result in step 1 by result in step 2, and sum those products
# At end of loop store the final sum
|-
|[[Heat/cold degree days and Chill hours | Cooling degree days]] (CDD)
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out increment in HDD by subtracting CDD threshold from current temperature
# Work out product multiplying result in step 1 by result in step 2, and sum those products
# At end of loop store the final sum
|-
|High solar radiation
| colspan="4" | current solar radiation
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of high solar radiation
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High UV Index
| colspan="4" | UV Index
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of high UV Index
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High [[Feels Like]] temperature
| Feels Like temperature
| Current relative humidity
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
| Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high feels like temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low Feels Like temperature
| Feels Like temperature
| Current relative humidity
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
| Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
|-
|Time of low feels like temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High Canadian Humidity Index or [[Humidex]]
| Humidex
| Current relative humidity
| Current temperature
|
| The Canadian Humidity index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high Humidex
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
| Cumulative Seasonal Chill Hours
|
| Current temperature
| Hour-Minute
|
| "Chill Hours" is a derived value, loop through every log file field in that Meteorological day:
# Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out if there is increment in Chill hours by seeing if "Current temperature" field is below Chill Hours threshold temperature
# If there is an increment, sum value from step 1
# At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day
|-
| Maximum rainfall in a 24 hour period ending on particular day
| colspan="4" | Rainfall Counter
| Two log file lines are examined, approximately 24 hours apart, the rainfall counter value in earlier of the two lines is subtracted from that in later of the two lines. If the difference is negative it is ignored. If the difference is zero, or positive, then it is inserted into the  Rain 24 hours for the relevant Meteorological day date for the time in the later line, unless it is lower than any number already there
|-
| Time of maximum rainfall in a 24 hour period
| colspan="4" | Hour-Minute
| Stores the time in later of the two log file lines used in the previous field
|}


The utility program, CreateMissing.exe, can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days), but as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.
===After running Create Missing===


This utility program, CreateMissing.exe, looks in [[Cumulus.ini]] for:
If MX was running while Create Missing was run, then MX will remain unaware that "dayfile.txt" has been updated, and that can cause a problem at rollover when MX appends a new line to the file.
# The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (although it can be edited to another date, such as when you imported earlier data or moved to a new home after you first used Cumulus).  That will be the earliest date the utility program processes.
#* However, if a dayfile.txt file exists and has an earlier date, then a prompt will ask you if you want to use the earlier date or the "StartDate=" date. If you answer "Y" to use earlier date, the utility program will continue, starting at that earlier date. If you answer with anything else, the utility program will exit.
#* The utility program will also check the start date against the current date, and will exit unless you have more than one day completed since you started using MX.
# The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters. This identifies which standard log lines belong to each day by checking against date and time of that line.
# Units and Number of decimal places associated with temperature, wind, wind average, rainfall, pressure, Ultra Violet, sunshine, evapotranspiration, wind run, and temperature trend.
# Thresholds for Heating Degree Days, Cooling Degree Days, and chill hours.
# station type (needed to determine which source values are available)
# month when Chill hours season starts
# options (use zero bearing, use 10 minute wind average, use speed or gust for wind average calculation, fix maximum humidity, read or calculate dew point, read or calculate wind chill, synchronisation time, , set clock option, read or calculate pressure trends, log extra sensors or not, ignore station clock or not, round wind speeds or not, air quality sensor or not, check that minimum number of sensors are updating or not, time for averaging wind bearings, time for averaging wind speed, and time for peak gust)


The utility program, CreateMissing.exe, uses the same module to calculate derived values (like dew point, wind chill, apparent temperature, feels like, and Humidity Index) from source values as CumulusMX.exe. This is how it is able to insert missing fields in the standard log files. As it processes each line in any standard log file, if a particular field is missing, it will calculate it from other fields in the line if it can.  The utility program, CreateMissing.exe, can update the various [[:Category:Ini Files]] with any new extremes found while it is calculating new fields for any standard log file.
If you are running a MX release up to 3.19.3, then you should first stop and then restart MX, so it loads the contents of the new file into its internal array (see [[#Running the Create Missing Utility]]. From release 3.20.0 (beta build 3199 onwards) released as build 3202, there is a '''Utils''' menu in the interface, and you should select ''Reload Dayfile'' menu item and on the web page subsequently displayed click the ''Reload Dayfile'' button.


The utility program,CreateMissing.exe, looks in the [[Data folder|data sub-folder]] to see if dayfile.txt exists (if it does, its file name has a '''.sav''' suffix added after the ''.txt''). If a file with .sav suffix already exists, the utility program will stop, as it cannot create a file if it already exists.
Create Missing changes the content of your [[Standard log files|MMMYYlog.txt files]] and your [[Amending dayfile|dayfile.txt]], so you may then wish to work through all the [[Correcting_Extremes|extreme record editors]] and decide if you want to update any entries.


As the utility program, CreateMissing.exe, reads each line in any [[Standard log files|standard log file]] it is processing, it works out from the date and time in that line which meteorological date to assign that line to, using the rollover time and use 10am in summer settings.  Any day with less than 5 standard log file lines is ignored for dayfile.txt, as is current day.
#At the start of each day, any maximum field in the output file (that is created as [[dayfile.txt]]) is set to an extreme negative, any minimum field in the same file is set to an extreme maximum, any daily total field is set to zero, and the cumulative chill hours field is set (either to the figure for the day before, or reset to zero for first day of new season).
# As each standard file line in that meteorological day is processed, any maximum field in the output file is updated if the read figure is higher than previous figure, any minimum field in the same file is updated if the read figure is lower than the previous figure, the total rainfall calculation changed between versions 1.0.0 and 1.0.1 (basically rain total is taken from the final line per day in the standard log file field 9)
# For fields like Heating Degree Days and Cooling Degree Days, CreateMissing.exe is tracking the temperature field 2 against the threshold it has read from Cumulus.ini, so if the degree day figure is missing, the utility is able to insert a figure based on the difference between temperature and threshold and the time since previous log (for heating degree days it is heating threshold minus temperature times minutes since last previous divided by 1440; for cooling it is temperature minus cooling threshold times minutes divided by 1440).
# For total sunshine, day starts just after midnight, and ends next midnight; even if that is not rollover time; the figure stored in dayfile.txt is stored against the meteorological day that applies at 1 minute past midnight. Basically if rollover time is not midnight, the sunshine on a particular calendar day is assigned to previous date in dayfile.txt.
# The average temperature added to dayfile if it is not already in existence, is based on the same approach as used in [[today.ini]], i.e. the utility maintains a cumulative total of time passed for the meteorological day, and a cumulative total incremented as each standard log file line is processed which is an increment of "minutes_since_previous_line  multiplied by (latest_temperature + last_temperature) divided by 2".
# Chill hours output is incremented from existing figure by number of minutes since last line divided by 60, if the temperature field is below the threshold.
# Tracking the maximum rainfall in last hour figure for a day is obviously a bit complicated, as the utility has to work on a running hourly rainfall total (total rainfall on line being processed minus total rainfall on line for one hour ago).


The utility program will output diagnostic messages both to any terminal session open, and to a file saved in [[MXdiags_folder|MXdiags sub-folder]].
===How the utility reports progress===


Here is a short section of typical output (there will also be messages when each log file is opened or finished with):
Here is a short section of typical output (from version 1.0.2 that had a bug and never processed 1st day of month) in a log that was stored in [[MXdiags folder|MXdiags]]:
<pre>
<pre>
2021-06-08 19:35:44.108 Loading log file - data/Jul20log.txt
2021-06-08 19:35:44.191 01/07/2020 : No monthly data was found, not updating this record
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
Line 262: Line 668:
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.007 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.141 Date: 31/07/2020 : Adding missing data
2021-06-08 19:35:45.156 Finished processing log file - data/Jul20log.txt
2021-06-08 19:35:45.156 Loading log file - data/Aug20log.txt
</pre>
</pre>


Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/9405...10805"

Navigation menu