Calculate Missing Values: Difference between revisions

= Derived spot values =

* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity. The original legacy Cumulus 1, and CumulusMX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by CumulusMX.  

* '''Wet Bulb''', is not calculated by CumulusMX

** The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like|here]].  

* [[Feels_Like|'''Feels Like Temperature''']] is calculated by CumulusMX only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like|has varied]] in different releases.

There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature, please see the [[MX_Administrative_Interface#Changing_Settings|interface]] and [[Cumulus.ini]] pages for how to find the settings.

* It uses some [[Software#By_Mark_Crossley|.dll files]] that are in the MX installation package referenced beside the [https://github.com/cumulusmx/CreateMissing/releases release download]

* It uses some information in [[Cumulus.ini]] a file created when MX is run

In a Microsoft Windows environment, Create Missing uses the .NET software that is normally already available.

 

In a UNIX-derived operating system (e.g. a computer running Linux or Raspberry Pi operating system), there is a need to [[Preparing_your_Linux_computer_for_MX#Checking_if_mono-complete_is_in_Source_List|install MONO]] to run either "CreateMissing.exe" or "CumulusMX.exe".

There is a download link on [[Software#Create_Missing]] page for the latest release of the utility. The MX 3.20.0 release zip also includes a copy of the utility; it was not included in release packages for earlier MX releases, and I can't predict whether it will be included in any subsequent MX release.

 

 

If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the '''CreateMissing.exe''' file may need to be given execute access (see [[Preparing_your_Linux_computer_for_MX#chmod]]).

 

 

# If there is a file there called '''dayfile.txt.sav''', rename that file to '''dayfile.txt.sav.bak''' (or any other name that does not already exist)

Optionally, you may wish to take a backup of the existing contents of the "data" directory onto a separate storage device, because the utility edits files in there, and if something goes wrong mid-edit (e.g. power cut, storage device failure) you could lose your valuable data.

The utility can be run while "CumulusMX.exe" is running (see below) either interactively or as a service, or while MX is stopped.

 

# First change the path to your Cumulus MX root folder

# Start a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon)

# Now type <code>CreateMissing</code>  (or type "CreateMissing.exe", both will work)

 

# First change your command line path to your Cumulus MX root folder (cd CHOSEN PATH/CumulusMX)

# Now type <code>sudo mono CreateMissing.exe</code>

#* If the user you are using already has execute access it is possible to leave out 'sudo' (the present writer has not tested this)

#* If mono is already running, it has been suggested it might be possible to leave out 'mono' (the present writer has not tested this)

''It is important to understand'' that when you start MX interactively, or as a service, it reads the contents of '''dayfile.txt''' into an internal array (held in random access memory - RAM).

* The only other time that  'CumulusMX.exe'' accesses the '''dayfile.txt''' is at rollover when it copies what it generates internally for the day just ended into the file

* At any other time, if you run '''CreateMissing.exe''' while ''CumulusMX.exe'' is running, any updates to the file are ''not'' seen by MX until that software is restarted.  

 

The utility program will output to any terminal session open and also (with further detail) to a file saved in [[#How the utility reports progress|MXdiags directory]].

The sequence followed by the utility is:

# The utility will not run if a file called '''dayfile.txt.sav''' already exists in [[data folder]]

# The utility reads [[Cumulus.ini]] and displays the 'record start date' it finds

# It creates an internal array to hold the equivalent of '''dayfile.txt''' in random access memory - RAM

# If there is an existing [[dayfile.txt]], its contents are used to populate that two-dimensional array

# It renames any existing 'dayfile.txt' to '''dayfile.txt.sav''' in your [[Data_folder|data sub-folder]]

# It then works through the MMMYYlog.txt files that have lines for dates after that 'record start date'

# For each of these [[Standard log files]] there is another sequence:

## The inner sequence involves opening the file (the log in MXDiags records when each file is opened, files may be opened more than once)

## The dates being processed are shown in the terminal window; as each output is followed by just a "Line Feed" character the lines will overwrite each other in a Microsoft Windows environment (which expects "Carriage Return" and "Line Feed" in sequence to terminate a line), will be on successive lines in a Linux environment (where the normal line terminator is "Line Feed"), and will be on one long line in a Mac operating system (where the normal line terminator is "Carriage Return")

## If a line read in the source file does not include a particular [[#Derived_value|derived value]] (see list [[#CreateMissing.exe|above]]), then the utility will calculate that required value from the [[#Source_value|spot source field values]] in the same line, and update that line

## The exact calculation done for each item in dayfile.txt is listed in the table [[#How the utility creates a dayfile.txt line]] below, but here are the basic principles

## For solar data, it examines lines in the file with times between two successive midnights

##* If the array is missing a "sun hours" figure for the date quoted in those lines, the utility stores the sun hours recorded in last entry of that period in the part of the internal array with matching date

## From release 1.3.0, CreateMissing.exe will also calculate the maximum rainfall in a 24-hour period; to do this it looks at rain counter for the line in the source file it is currently processing, and attempts to find rain counter in another line nearest to 24 hours later (if the time interval between the lines is not exactly 24 hours then the difference figure reported will be for whatever period is available) looking up the rain counter there; it will track the maximum difference ending on a date that date determines where in the array the figure is stored; if the difference in rain counter yields a negative figure it is ignored

## For other daily data, the lines examined will start at rollover time and continue until just before the next rollover time (with allowance for any DST change)

##* If the daily data is an extreme, then the highest or lowest (source or derived) value seen in that range of lines is added to the array if an entry does not already exist

##* If the daily data is cumulative (over a day or over a season), then if the internal array does not already have a value, that from the last value in the range of lines is stored

##* If the daily data is not directly related to fields held in the source file (e.g. cumulative chill hours), then it will be calculated from what is available (for chill hours, if the recorded temperature in line being examined is below threshold, then the time interval passed since the previous entry is added to an internally held count; note this MX calculation is slightly different to Steve Loft approach that worked on average of temperature in a particular line and the temperature in previous line being below threshold)

## When all lines in a particular file have been examined, that file is closed (the log in MXDiags records when each file is closed, files may be closed more than once)

## The next file (chronologically) is opened (the log in MXDiags records when each file is opened, files may be opened more than once) and the inner sequence continues

# The utility creates a new file, naming it '''dayfile.txt'''

# The utility copies what is stored in its internal array to the new file