Calculate Missing Values: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search

Calculate Missing Values (view source)

Revision as of 15:47, 9 June 2021

226 bytes added ,  15:47, 9 June 2021
m
no edit summary
No edit summary
mNo edit summary
Line 59: Line 59:
#* Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file
#* Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file


=="Create Missing"==  
=="Calculate Missing"==  


This also has two meanings in a Cumulus context:
This also has two meanings in a Cumulus context:
# If a particular standard log file line has fewer fields than the latest line;  
# If a particular standard log file line has fewer fields than the latest line;  
#* '''Create Missing''' is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
#* '''Calculate Missing''' is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
# If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
# If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
#* Please see [[Amending dayfile]] page for full details.
#* Please see [[Amending dayfile]] page for full details.
#* '''Create Missing''' is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.  
#* '''Calculate Missing''' is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.  


If you are using Cumulus MX, there is a download [[Software#Create_Missing| linked from here]] that does both of these. There are also editors within the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|admin interface]] for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.
If you are using Cumulus MX, there is a download [[Software#Create_Missing| linked from here]] that does both of these. There are also editors within the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|admin interface]] for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.
Line 76: Line 76:
===Accurate or Not?===
===Accurate or Not?===


If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1.   That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
This Wiki page describes some techniques for calculating and inserting values that are missing from standard log files and from daily summary log file.
#Only use the editor when Cumulus is '''not''' doing catch-up or end-of-day, or any other action that could create an editing conflict.
 
#With the contents of the file loaded into the editing table, click the '''Create Missing''' button
Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your [[:Category:Ini Files|.ini]] files is missing some fields, these missing fields cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.
#Cumulus 1 will now work through every line from start to end
 
#If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.
#It will read from the [[Standard log files]] all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
 
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
For the standard log files, all the fields in any one line relate to the same time, therefore for derived values calculated from other fields in the same line, you should have the same value whether it was calculated when that line was originally stored, or calculated afterwards.  I say '''should''' because the calculation formula is not always the same for all releases, in particular there are differences between how Cumulus 1 and how MX calculate some derivatives.
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
 
#*Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
For entries in today.ini, month.ini, year.ini, alltime.ini, and monthlyalltime.ini files, you don't have access to the source values used for the original calculation afterwards.
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,  
* The original values are calculated as Cumulus is running
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)
**Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini (and the other files listed) each minute if an extreme happens
** Obviously, dayfile.txt is updated from today.ini, so it is just as accurate
* Any "Calculate Missing" operation, done subsequently, does not have access to old data, it can only look in the spot values that have been logged.
**If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,  
** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)


=Derived spot values=
=Derived spot values=
Line 121: Line 125:
Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.
Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.


=Why do "Create Missing"?=
=Why do "Calculate Missing"?=


Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the [[Correcting_Extremes]] page.  You are unlikely to worry about missing past values.
Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the [[Correcting_Extremes]] page.  You are unlikely to worry about missing past values.
Line 130: Line 134:




=How to do "Create Missing"=
=How to do "Calculate Missing"=


As mentioned earlier, there are a number of options, here are the detailed instructions for each option.
As mentioned earlier, there are a number of options, here are the detailed instructions for each option.
Line 209: Line 213:
<div style="background: LemonChiffon;padding:5px; margin:2px;">
<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software.  It has been updated to cover MX, but that was for a MX release that is no longer latest!
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software.  It has been updated to cover MX, but that was for a MX release that is no longer latest!
When this section was written, the number of lines shown was fixed at a maximum of 10; later releases have given the option to display different numbers of lines, and there may be other changes still to be documented here.
</div>
</div>
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
Line 223: Line 229:
==Lack of editor in Cumulus 1==
==Lack of editor in Cumulus 1==


Cumulus 1 provides a [[Cumulus_Screenshots#View_Menu|viewer]] for the data logs. On the '''View''' menu, select ''''Data Logs''', then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which.  You cannot do any editing.
Cumulus 1 provides a [[Cumulus_Screenshots#View_Menu|viewer]] for the data logs, that does not permit editing of the file.  
 
On the '''View''' menu, select ''''Data Logs''', then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which.  You cannot do any editing.


If you find that this viewer cannot load a data log, it is probably because you ignored the '''read me''' that is part of the Cumulus 1 installation procedure, see  [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]].  If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.
If you find that this viewer cannot load a data log, it is probably because you ignored the '''read me''' that is part of the Cumulus 1 installation procedure, see  [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]].  If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.
Line 249: Line 257:
* Ensure no blank lines are introduced by your editing.
* Ensure no blank lines are introduced by your editing.
*Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.
*Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.
=Other logs=
Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your [[:Category:Ini Files|.ini]] files is missing some fields, they cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.




Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/9116"

Navigation menu