5,838
edits
Correcting Extremes: Difference between revisions
m
Some spelling corrections, and layout sequence improvements
m (minor resequencing) |
m (Some spelling corrections, and layout sequence improvements) |
||
| Line 1: | Line 1: | ||
[[Category:Cumulus Files]][[Category:Ini Files]] | [[Category:Cumulus Files]][[Category:Ini Files]] | ||
=Terminology= | |||
For simplicity, the terminology "extremes" is used on this page, the meaning includes: | |||
# The ''totals'' maintained (such as rainfall, chill hours, and the various "degree days"), and | |||
# The high/low "extreme records" for various periods (such as all-time, and this month). | |||
This Wiki page has been created to cover both {{Version badge 1}} and {{Template:Version badge Mx}}, but do be aware that some terminology varies between the two flavours. All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered between Wiki pages for each specific file, this redesign brings together all such text here). | |||
= | ==Rogue value== | ||
Throughout this Wiki the term '''rogue value''' is used to mean you see a value somewhere in Cumulus that you believe should not be there. | |||
Generally, "rogue" usage refers to a single data point. However, where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files. Specifically, on this Wiki page, the meaning covers any incorrect entry in one or more of the [[:Category:Ini Files|extreme record files]]. | |||
The extreme records that are maintained in this way are: | Here are some typical examples: | ||
* it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong | |||
* perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points | |||
* an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong | |||
=How Cumulus software tracks "extremes"= | |||
As Cumulus weather software processes each reading from your weather station, it checks that value (and any [[Calculate_Missing_Values#Derived spot values|derived]] from it) against the extremes currently stored in various RAM variables, and if necessary updates the extreme records that are affected. Please note these extreme records are held by Cumulus software in internal variables. Periodically, these totals and high/low "extreme records" are written out to [[:Category:Log_Files|.ini files]] (ensuring such "extremes" are kept when MX is stopped and restarted). | |||
The totals/extreme records that are maintained in this way are: | |||
{| class="wikitable" border="1" | {| class="wikitable" border="1" | ||
| Line 70: | Line 81: | ||
=Built in extreme record editors= | =Built in extreme record editors= | ||
| Line 205: | Line 162: | ||
Cumulus 1 only provides one way to access the [[Webtags#Recent_History|Recent_History]], and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags. | Cumulus 1 only provides one way to access the [[Webtags#Recent_History|Recent_History]], and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags. | ||
Cumulus MX stores its [[Recent history]] in a SQLite3 database that you can read. | Cumulus MX stores its [[Recent history]] in a SQLite3 database that you can read/edit as explained at [[Cumulusmx.db#Reading.2Fediting_database_table_outside_MX]]. | ||
=General Editing Advice= | |||
The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values. | |||
You may have a feel as to which files in above table will need correction, but if in doubt it is highly recommended that you always start your extreme record correction by seeing if the error is present in the [[Alltime.ini]] file that holds all-time extreme records. That approach is best, partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier. | |||
| Line 279: | Line 244: | ||
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page. | Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page. | ||
= | |||
==Correcting multiple extremes== | |||
There are two cases to consider: | |||
===Correcting extremes recorded for every logging entry (plus every day/month/year of long period)=== | |||
Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units. | |||
In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any [[Extra Sensor Files]], any [[Standard log files]], in [[dayfile.txt]], plus the multiple [[Category:Ini Files|extreme record files]]. | |||
The built-in editors only correct one extreme record at a time, so they cannot be used for such a task. | |||
It is important to remember that there are [[Calculate_Missing_Values#Some_definitions|source and derived values]] in Cumulus. If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see [[Calculate Missing Values]] page for further advice. | |||
The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see [https://cumulus.hosiene.co.uk/viewtopic.php?p=155539#p155539 changed rainfall units example]), or to use a spreadsheet (be careful not to affects dates or times) like '''Libre Office Calc''' where you can paste special a multiplier to all cells in a particular column. | |||
===Correcting extremes just for a few logging entries (plus selected days/months/years in a short period)=== | |||
There are a further two sub-cases that fall in this category. Both are near impossible to resolve! | |||
Both Cumulus 1 and MX have had bugs in some releases of their software. This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process! | |||
[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt, for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period). | |||
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records | |||
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort. | |||
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey. | |||
In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent '''not-working'''. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null. Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations. However, there is no official solution to this problem! | |||
==All-time corrections== | |||
For the all-time extreme records, each individual update is logged in [[Alltimelog.txt]] from version 1.8.9 onwards. Depending on sequence of extreme values, you may get an accurate previous value from this tracking log. | |||
* The tracking log will not tell you a correct high/low extreme record if ''the rogue extreme occurred before the actual high/low extreme'' was experienced. This is because the rogue extreme stopped any subsequent true extreme being recorded. | |||
* If the ''actual high/low extreme that you want to retain was recorded before the rogue extreme'', then you can see that true value, and its time-stamp, in the tracking log. Based on that time-stamp, the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit. | |||
==Introduction of Monthly All Time Extreme records== | |||
From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'. | From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'. | ||
| Line 286: | Line 290: | ||
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]). | Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]). | ||
==Monthly-all-time corrections== | |||
If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the [[Monthlyalltime.ini|monthly-all-time]] file. | |||
* From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'. | |||
* If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected. | |||
* If you use MX, then [[Monthlyalltimelog.txt]]) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded. | |||
==Correction of extremes for past year== | |||
An earlier correction may have identified that the rogue value was in a past year, so this sub-section explores whether you can continue correction pathway: | |||
* [[File:Badge v1.png]]Cumulus 1 never allows you to see a '''year.ini''' file when the year is completed, because at the end of the year it is initialised ready for the new year. | |||
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, the MX beta (3.0.0), and later MX releases, at the start of a new year, saves the old year.ini (whenever it was last updated) as a file with a name like '''year2015.ini''', and only then overwrites the ''year.ini'' file. | |||
** Although MX does not provide any functionality to read this old file, let alone edit it, you may want to use a standard text editor to amend the relevant part of this old file. Your edit to either ''alltime.ini'' or ''monthlyalltime.ini'' should have told you what old value in old file is wrong, and told you correct value to replace that. | |||
==Correction of extremes for past month== | |||
An earlier correction may have identified that the rogue value was in a past month, so this sub-section explores whether you can continue correction pathway: | |||
* [[File:Badge v1.png]]Cumulus 1 never allows you to see a '''month.ini''' file when the month is completed, because at the end of the month it is re-initialised ready for the new month. | |||
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, the MX beta (3.0.0), and later MX releases, at the end of a month, saves the old '''month.ini''' (whenever it was last updated) as a file with a name like '''month201501.ini''' (i.e. "month", followed by year, followed by month number, and with file type ".ini"), before writing standard "reset high/low values" to '''month.ini'''. | |||
** Although MX does not provide any functionality to read this old file, let alone edit it, you may want to use a standard text editor to amend the relevant part of this old file. Your edit to either ''alltime.ini'' or ''monthlyalltime.ini'' should have told you what old value in old file is wrong, and told you correct value to replace that. | |||
==Current month/year corrections== | |||
If your earlier correction (''finding how the rogue value has affected monthly-all-time has given you a date'' in the current month/year), that is a steer to whether the [[Month.ini|this month]] extremes file will need correction, and whether [[Year.ini|this year]] extreme records file will need correction. | |||
They are relatively small files, so it should be easy to use the [[#Built in extreme record editors]] to edit them. | |||
=Correction of extremes for past days= | ==Correction of extremes for past days== | ||
If the rogue value relates to yesterday, or an earlier day, then you must edit [[Amending dayfile|dayfile.txt]] to make any necessary corrections for past days. | |||
It is entirely optional whether you choose to update [[Yesterday.ini|yesterday.ini]] if that contains a rogue value. That file is overwritten at both midnight and at next rollover, so in general there is no benefit gained from any editing. | |||
=== Correcting an error in rainfall for a past day=== | |||
For the legacy Cumulus 1, rainfall corrections are covered at [[FAQ#My_station_invented_some_rain_that_didn.27t_really_occur.2C_and_I_want_to_set_it_to_zero_.28or_some_other_figure.29]]. | |||
The total rainfall for a month, or 12-month season (rainfall labelled as ''Rain this year'' can start counting from zero on first [[Meteorological day]] of any month chosen by user (Interface: '''Station settings''' → ''Annual Rainfall'' %rarr; '''Start of rainfall season'''), is held in an internal (RAM) variable in Cumulus software. When you start Cumulus software, the program looks in [[dayfile.txt]] and reads the amount for each past day (of current month/season) in that file, then adds the rainfall for today-so-far. For rain this season, Cumulus flavour alters calculation: | |||
* Cumulus 1 - if the year entered in [[Cumulus_Screenshots#Station]] "Annual Rainfall" frame, box labelled ''Year'', matches the current '''calendar''' year, then the amount in box labelled ''Amount'' is added. Note if your season does not start in "January" then this only affects the part of the season that is in the specified year. | |||
* Cumulus MX - if your rainfall season starts in a month within year specified in box labelled "Year to which year-to-date amount applies", then the amount in box labelled "Year-to-date amount" is added to your rain for that season as shown in ''Rain this year'' (and the <ryear> tag if used for [[MySqlConnect|Custom SQL]], MQTT, [[Cumulus_MX_Local_API]], or [[Cumulus_template_file]]) | |||
So if the rainfall reported for current month, or current "season" (year) is wrong, you need to correct [[dayfile.txt]], both Cumulus 1 and MX have [[Amending_dayfile#Editors_built_into_Cumulus|built-in dayfile.txt editors]]. | |||
==Today corrections== | |||
As described below, the Cumulus '''Edit' menu includes a '''Today's rain''' option where you can adjust the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus|total rainfall for today]] (e.g. if you or the wind have knocked your rain gauge). | |||
''There is no facility provided to edit any other content'' of [[today.ini]]. Manual correction may be possible, but do read advice on [[today.ini]] page, in particular noting that MX only reads "today.ini" when first started, MX uses an internal array that represents content of file while MX is running. | |||
In working through the various files in above table, remember that if the rogue value was recorded today, then [[today.ini]] will be wrong: | |||
* | * In Cumulus 1, you possibly could edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file | ||
* In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see [[MX on Linux]] or [[MX on Windows OS]]) and edit the file using a text editor, or programmer's editor, that will not add unwanted content to the file. | |||
=== Correcting an error in today's total rainfall === | |||
Easy - correct today's total using the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus | 'today's rain']] editor on the edit menu. | |||
* [[File:Badge v1.png]]select 'Today's rain in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1, | |||
* [[File:Badge vMx.png]]select '''Today's rain''' in the [[MX_Administrative_Interface#Today.27s_rain.27|edit menu]] of MX interface. | |||
* | This edit will actually alter the start of day rainfall counter figure: | ||
* | *If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter | ||
*If you want today's rain to seem greater (perhaps the rain gauge got blocked by a leaf), Cumulus will decrease the start of day counter | |||
Please note that this edit does not affect "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]], your database will retain incorrect values, as these are not updated by this correction. | |||
Please see [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit those related fields. | |||
=Some definitions= | |||
==Flavour, Release, Version, and Build== | ==Flavour, Release, Version, and Build== | ||