Correcting Extremes: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
6,742 bytes added ,  09:10, 15 July 2021
m
mNo edit summary
(20 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page brings together text that was originally on other pages. It is designed to cover both
{{Version badge 1}} and {{Template:Version badge Mx}}, but do be aware that some terminology varies between the two flavours.
[[Category:Cumulus Files]][[Category:Ini Files]]
=Introduction=
=Introduction=


As Cumulus processes each reading from your weather station, it checks that value (and any [[Correcting_Missing_Values#Derived spot values|derived]] from it) against the extremes currently stored in various [[:Category:Log_Files|.ini files]], and if necessary updates the extreme records that are affected. The extreme records that are maintained in this way are:
As Cumulus processes each reading from your weather station, it checks that value (and any [[Calculate_Missing_Values#Derived spot values|derived]] from it) against the extremes currently stored in various [[:Category:Log_Files|.ini files]], and if necessary updates the extreme records that are affected.
 
All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered amongst pages covering the various files).
 
The extreme records that are maintained in this way are:


{| class="wikitable" border="1"
{| class="wikitable" border="1"
|-
|-
!style="width:200px"|Period
!style="width:200px"|Period
!style="width:50px"|File
!style="width:50px"|File storing extremes
!style="width:50px"|Example web page
!style="width:300px"|How to correct
!style="width:200px"|Notes
!style="width:50px"|Link to web tag section
!style="width:300px"|Notes
|-
|-
|For current day so far
|For current day so far
|[[today.ini]]
|[[today.ini]]
| Editor for "Today's rain" (no editor for other derivatives)
|[[Webtags#Today|today.htm]]
|[[Webtags#Today|today.htm]]
|Many entries in this file get transferred to [[dayfile.txt]] at end of day
|Many entries in this file (for non-midnight rollover, use is made of [[yesterday.ini]] too) get transferred to [[dayfile.txt]] at end of day.
|-
| For past days
| [[dayfile.txt]]
| See [[Amending dayfile]]
| Web tags only exist for [[Webtags#Yesterday|yesterday]]
| Often used as source for corrections - see  [[#Accuracy Note]]
|-
|-
|For current month-to-date
|For current month-to-date
|[[month.ini]]
|[[month.ini]]
| Editor for "This month's records"
|[[Webtags#Monthly|thismonth.htm]]
|[[Webtags#Monthly|thismonth.htm]]
|
| Please see [[#Accuracy Note]]
|-
|-
|For current year-to-date
|For current year-to-date
|[[year.ini]]
|[[year.ini]]
| Editor for "This year's records"
|[[Webtags#Yearly|thisyear.htm]]
|[[Webtags#Yearly|thisyear.htm]]
|
| Please see [[#Accuracy Note]]
|-
|-
|For all readings since a '''start date''
|For all readings since a '''start date''
|[[alltime.ini]]
|[[alltime.ini]]
| Editor for "All Time Records"
|[[Webtags#All_Time|records.htm]]
|[[Webtags#All_Time|records.htm]]
|See table below for start date
|See table below for start date
Line 32: Line 55:
|For a particular month in all years
|For a particular month in all years
|[[monthlyalltime.ini]]
|[[monthlyalltime.ini]]
| Editor for "Monthly Records"
|[[Webtags#Monthly_All_Time_Records|monthlyrecord.htm]]
|
|
|}
|}
Following the links in the second column leads you to more information about the relevant file. Following links in third column leads you to more information about the web tags that you can incorporate in your own [[Customised_templates|templates]].
Explaining columns in above table:
# The first column is self-explanatory
# The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
# The third column gives the name for the '''Edit''' menu item to choose to edit these extreme records
# The links in fourth column leads you to more information about the web tags associated with that period,  you can incorporate those in your own [[Customised_templates|templates]].
 
==How editing accuracy depends on source selected==
 
The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:
# The figure taken from a search for that extreme by examining all entries in the [[dayfile.txt]] for that period
# The figure taken from a search for that extreme by examining all entries in the [[Standard log files]] for that period
 
Normally the first returns the more accurate result (unless the '''dayfile.txt''' line, either was created with a rogue value, or has been corrupted). Let me explain why:
* Using [[Standard log files]] as source for recalculating past extremes:
** Let us assume you are using the default logging interval of 10 minutes
** Unlike some other weather station software available (which logs highest and lowest since previous log), Cumulus logs spot values
** That means the [[Monthly log files]] do not capture any extremes recorded in the time (by default 599 seconds) between logs
** Therefore the detailed log files are not normally the most accurate source
** Please note, this less accurate way of deducing daily extremes/totals (to update dayfile.txt) is used by Cumulus software:
*** For Legacy Cumulus 1: [[Amending_dayfile#Create_Missing_on_legacy_dayfile_editor|Create Missing in legacy dayfile editor]].
*** For MX: [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] utility.
* Using [[dayfile.txt]] as source for recalculating past extremes
** MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
** Therefore extremes recorded in '''today.ini''' (and from there into '''dayfile.ini'') are based on the full sampling done by Cumulus
** This means none, or very few, extremes are missed
** In March 2021, a new utility '''Create Records''' was planned (for use with MX only), as at July 2021 no progress has been made in coding it.  It appears that this utility will read '''dayfile.txt''' and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other [[:Category:Ini Files|files like monthly-all-time and all-time]]. ''<big>Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available</big>''.
 
However, if you either have had an electrical supply problem or there has been corruption of data within your weather station generating a rogue value, it is possible that the files currently being used by Cumulus get corrupted.  Provided you act, as soon as possible afterwards, [[Calculate_Missing_Values#Reading_archive_data|you can make use]] of the [[Backup folder|backups]] that Cumulus takes. All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see [[today.ini]]) - either with the end of day or start of day contents.  For this month and this year extreme records, it is often better to find the previous extreme by examining a backup copy, providing a true extreme has not happened since, than by using the values displayed in the editor as those might be corrupted in the same problem.
 
For all-time (and for MX only, monthly-all-time) extreme records, there is a log [[Alltimelog.txt]] (and for MX only,[[Monthlyalltimelog.txt]]) that logs each time the extreme is updated, so you can get an accurate previous value from there.


The purpose of this article is to help you to edit incorrect entries in any of the .ini files in the second column, to understand where else to investigate, and where necessary amend other log files.
==Arrangement of information on this page==


The article starts with corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).
The page continues by first describing corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).


There is more information in [[:Category:Log_Files]] and the pages relating to individual files, for all of the extreme holding files.
There is more information in [[:Category:Log_Files]] and the pages relating to individual files, for all of the extreme holding files.
{{TOCright}}


==Why might your extremes need to be corrected?==  
==Why might your extremes need to be corrected?==  


If there is a rogue value read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.
If there is a [[Correcting_Extremes#Rogue_value|rogue value]] read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.


It is also possible that you have discovered that you made a mistake in setting up or calibrating a sensor, and this leads you to identifying a constant/multiplier adjustment
It is also possible that you have discovered that you made a mistake in setting up or calibrating a sensor, and this leads you to identifying a constant/multiplier adjustment
Line 56: Line 109:
For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time.  Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.
For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time.  Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.


{{Version badge 1}}There were bugs introduced sometimes in builds of the original Cumulus (1). Information about a few of the bugs and fixes can be found in [[at File:Changes.zip]], although that does not cover any 1.7.x versions, and does not detail bugs created and fixed within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort as there are a lot of posts to search. (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log for the relevant period, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).  
[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt,  for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).  


[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries.  The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries.  The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.


[[File:Icon_info.png]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has added more extreme records to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
[[Image:Icon info.png|left|30px]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:


{| class="wikitable" border="1"
{| class="wikitable" border="1"
|-
|-
!style="width:200px"|Parameter
!style="width:300px"|Parameter
!style="width:50px"|First released
!style="width:200px"|First released
!style="width:50px"|First in Version
!style="width:100px"|First in Version
!style="width:50px"|First in Build
!style="width:100px"|First in Build
|-
|-
|highest/lowest apparent temperature
|highest/lowest apparent temperature
Line 75: Line 128:
|957
|957
|-
|-
|highest/lowest feels like temperture
|highest/lowest feels like temperature
|24 June 2020
|24 June 2020
|3.6.10
|3.6.10
Line 100: Line 153:
|(lost)
|(lost)
|-
|-
|highest daily windrun
|highest daily wind run
|3 Jul 2011
|3 Jul 2011
|1.9.2 beta
|1.9.2 beta
Line 106: Line 159:
|}
|}


Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki.  Consequently,  all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elesewhere in this page.
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki.  Consequently,  all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page.
 


=Correction of All Time Extreme records=
=Correction of All Time Extreme records=
Line 119: Line 171:


Of course, it is possible that the old value in '''Alltimelog.txt''' is not appropriate. It might be that after the rogue value was stored in '''alltime.ini''', a new extreme was seen, and this new extreme was different to the previous value stored in '''Alltimelog.txt''', but it did not cause an update in '''alltime.ini''' because of the rogue value that was stored there being more extreme.
Of course, it is possible that the old value in '''Alltimelog.txt''' is not appropriate. It might be that after the rogue value was stored in '''alltime.ini''', a new extreme was seen, and this new extreme was different to the previous value stored in '''Alltimelog.txt''', but it did not cause an update in '''alltime.ini''' because of the rogue value that was stored there being more extreme.
Please see [[#Accuracy Note]] for other accuracy points.


===Looking at graphical representations===
===Looking at graphical representations===
Line 146: Line 200:
When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:
When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:
* [[File:Badge v1.png]] select '''All-time records''' in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
* [[File:Badge v1.png]] select '''All-time records''' in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
* [[File:Badge vMx.png]]select '''All Time records''' in the [[MX_Administrative_Interface#Today.27s_rain.27|edit menu]] of MX admin interface.
* [[File:Badge vMx.png]]select '''All Time records''' in the [[MX_Administrative_Interface#All_time_records|edit menu]] of MX admin interface.


The way the editor allows you to change a value depends on which flavour you are using:
The way the editor allows you to change a value depends on which flavour you are using:
Line 153: Line 207:


Please remember any edit you make here will not affect the related extreme in your monthly-all-time extremes recorded in [[monthlyalltime.ini]]. Nor can you update [[Standard_log_files]], or [[dayfile.txt|daily summary log]], by any edit made here. If you are using MX, it has not affected related tables on your database server.
Please remember any edit you make here will not affect the related extreme in your monthly-all-time extremes recorded in [[monthlyalltime.ini]]. Nor can you update [[Standard_log_files]], or [[dayfile.txt|daily summary log]], by any edit made here. If you are using MX, it has not affected related tables on your database server.


==Actions after you have edited all-time entries==
==Actions after you have edited all-time entries==
Line 175: Line 228:
==Initialisation of monthly-all-time extreme records==
==Initialisation of monthly-all-time extreme records==


Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Creating_Missing_Values]]).
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]).


== How do I correct my monthly all-time records? ==
== How do I correct my monthly all-time records? ==
Line 182: Line 235:
*[[File:Badge v1.png]] If you are a Cumulus 1 user, you do not have a [[monthlyalltimelog.txt]] file to look in to see the log entry with the previous and new values. This information was logged into a file in [[Diags_folder|the diagnostic files folder]]. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.  
*[[File:Badge v1.png]] If you are a Cumulus 1 user, you do not have a [[monthlyalltimelog.txt]] file to look in to see the log entry with the previous and new values. This information was logged into a file in [[Diags_folder|the diagnostic files folder]]. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.  
* [[File:Badge vMx.png]] If you are a MX user, see if you have a [[monthlyalltimelog.txt]] file in the same [[data_folder| data folder]], and then you will know what value to revert to.
* [[File:Badge vMx.png]] If you are a MX user, see if you have a [[monthlyalltimelog.txt]] file in the same [[data_folder| data folder]], and then you will know what value to revert to.
  Here the old and new values can be looked up in the [[monthlyalltimelog.txt]] file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in [[Diags_folder]].
Here the old and new values can be looked up in the [[monthlyalltimelog.txt]] file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in [[Diags_folder]].




Line 188: Line 241:


The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.
The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.


==Actions after you have edited monthly-all-time entries==
==Actions after you have edited monthly-all-time entries==
Line 261: Line 313:
==Rogue value==
==Rogue value==


In this article, the term '''rogue value''' is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumuluative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be progated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.
In this article, the term '''rogue value''' is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.


Here are a typical examples:
Here are a typical examples:
* it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
* it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
* perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumuluative a series of wrong date points
* perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
* an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong
* an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong


==Flavour, Release, Version, and Build==
'''Flavour''' is used to represent the original Cumulus, Cumulus 2, and Cumulus MX, collectively. Where the text says applicability is dependant on flavour, it means that the action you do depends on whether you are installing/running Cumulus MX or the original Cumulus software.
'''Release''' is used to signify what the Cumulus developer makes available for download after there has been a modification to the software.  For most Cumulus 1 builds, the release consisted of a executable that would create all the folders and files needed to run that software. For Cumulus 2, releases were a zip file and were numbered using a identifier with many digits.  MX releases are as a zip file that is labelled with the build number.
'''Version''' here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.


[[Category:Log Files]]
'''Build''' number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number. For recent MX releases, the developer has changed version number every time there is a new build released.
5,838

edits

Navigation menu