Recent History (preserving history) - Revision history

HansR at 11:55, 29 September 2022

2022-09-29T11:55:11Z

← Older revision		Revision as of 11:55, 29 September 2022
Line 1:		Line 1:
	This page is the new ''Recent History'' page. This page is part of a complete rewrite of the ''Webtags'' page.
	As long as this message exists please refer to Recent History (preserving history).

	This page is Work in Progress until this message is deleted.

	If any questions or remarks please send a PM to [https://cumulus.hosiene.co.uk/memberlist.php?mode=viewprofile&u=9016 HansR] on the forum. Do not use the ''discussion'' tab unless accompanied by a PM: there is no automated signal to HansR if you use that page.

	This page and its subpages will document the Webtags from the point of view of the CumulusMX software and not the Cumulus 1 legacy software. Cumulus 1 is no longer maintained and will eventually disappear. If you are still using it, please refer to the original [[Webtags_(preserving_history)\|Webtags]] page where @sfws has tried to give both packages equal value. That effort is no longer maintained in this page. We hope you will understand this approach.

	Please do not edit this page while it is being worked on i.e. while this message is still here.

	<hr/>

	[[Category:Terminology]] [[Category:Cumulus Files]]		[[Category:Terminology]] [[Category:Cumulus Files]]

HansR: HansR moved page (preserving history) to Recent History (preserving history): Title correction

2022-09-29T11:52:46Z

HansR moved page (preserving history) to Recent History (preserving history): Title correction

← Older revision

Revision as of 11:52, 29 September 2022

HansR: HansR moved page Recent history to (preserving history) without leaving a redirect: Rewrite of contents

2022-09-29T11:51:41Z

HansR moved page Recent history to (preserving history) without leaving a redirect: Rewrite of contents

← Older revision

Revision as of 11:51, 29 September 2022

HansR at 11:50, 29 September 2022

2022-09-29T11:50:45Z

← Older revision		Revision as of 11:50, 29 September 2022
Line 1:		Line 1:
	This page is the new ''Recent History'' page. This page is part of a complete rewrite of the ''Webtags'' page.		This page is the new ''Recent History'' page. This page is part of a complete rewrite of the ''Webtags'' page.
			As long as this message exists please refer to Recent History (preserving history).

	This page is Work in Progress until this message is deleted.		This page is Work in Progress until this message is deleted.

HansR at 14:16, 28 September 2022

2022-09-28T14:16:44Z

← Older revision		Revision as of 14:16, 28 September 2022
Line 1:		Line 1:
			This page is the new ''Recent History'' page. This page is part of a complete rewrite of the ''Webtags'' page.
	{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to all Cumulus flavours (including the abandoned Cumulus 2).

			This page is Work in Progress until this message is deleted.

			If any questions or remarks please send a PM to [https://cumulus.hosiene.co.uk/memberlist.php?mode=viewprofile&u=9016 HansR] on the forum. Do not use the ''discussion'' tab unless accompanied by a PM: there is no automated signal to HansR if you use that page.

			This page and its subpages will document the Webtags from the point of view of the CumulusMX software and not the Cumulus 1 legacy software. Cumulus 1 is no longer maintained and will eventually disappear. If you are still using it, please refer to the original [[Webtags_(preserving_history)\|Webtags]] page where @sfws has tried to give both packages equal value. That effort is no longer maintained in this page. We hope you will understand this approach.

			Please do not edit this page while it is being worked on i.e. while this message is still here.

			<hr/>

	[[Category:Terminology]] [[Category:Cumulus Files]]		[[Category:Terminology]] [[Category:Cumulus Files]]

Sfws at 15:51, 16 June 2022

2022-06-16T15:51:52Z

← Older revision		Revision as of 15:51, 16 June 2022
Line 1:		Line 1:
	{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to all Cumulus flavours (including the abandoned Cumulus 2).		{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to all Cumulus flavours (including the abandoned Cumulus 2).
	[[Category:Terminology]]		[[Category:Terminology]] [[Category:Cumulus Files]]

	=What is Recent History?=		=What is Recent History?=

	Recent history functionality in Cumulus is the ability to store weather values~~, at up to 1 minute intervals~~, for the last 7 days.		Recent history functionality in Cumulus is the ability to store weather values, for the last 7 days.

	It is ~~important~~ to ~~realise,~~ ~~these~~ ~~are~~ ~~spot~~ ~~readings~~, ~~and~~ will miss any extremes that occur between the times when the recent history values are stored. The weather value that tends to vary the most is wind speed, so that is where peaks are most often missed.		Only some spot readings are stored in Recent History, the stored values will miss any extremes that occur between the times when the recent history values are stored. The weather value that tends to vary the most is wind speed, so that is where peaks are most often missed.

	It is ~~also~~ ~~important~~ to ~~realise~~ ~~that~~ ~~weather~~ ~~stations~~ ~~are~~ ~~not~~ ~~supplying~~ ~~values~~ to Cumulus on a ~~continuous~~ ~~basis~~, ~~depending~~ on ~~the~~ ~~weather~~ ~~station~~ ~~type~~ ~~values~~ ~~may~~ be ~~available~~ at ~~various~~ ~~intervals,~~ ~~either~~ ~~every~~ so ~~many~~ ~~seconds~~ or ~~perhaps~~ at a ~~rate~~ ~~less~~ ~~than~~ ~~once~~ a minute. This can mean that the readings stored on a 1-minute basis may actually be duplicates if the station has not supplied a new value since the previous minute.		When Cumulus is running normally, entries are stored once a minute, regardless of how often Cumulus interrogates the weather station (might be every 10 seconds, or more or less frequently) and regardless of how often the weather station gets updated readings from sensors (might be every few seconds, every 40 seconds, every minute or less frequently). This can mean that the readings stored on a 1-minute basis may actually be duplicates if the station has not supplied a new value since the previous minute, or may not capture every reading from sensors if they are read more than once in a minute.

			When Cumulus is restarted, if the weather station provides historical catch-up data, then the interval between entries for that period when Cumulus was not running, is stored at whatever interval the historical catch-up data uses. Since release 3.12.0, entries less than a week old, from when MX was previously running will be retained, and those will be at one minute interval; earlier releases will not collect any entries from when Cumulus was previously running, just those for up to a week while it was not running.

	=How is the functionality used?=		=How is the functionality used?=

Sfws: /* How are recent history values stored? */

2022-06-16T15:37:25Z

How are recent history values stored?

← Older revision		Revision as of 15:37, 16 June 2022
Line 50:		Line 50:

	Depends on the release you are running:		Depends on the release you are running:
	* Release 3.12.0 onwards: Held in [[cumulusmx.db\|RecentData table in cumulusmx.db]] (uses SQLite), so available ~~outside~~ MX		* Release 3.12.0 onwards: Held in [[cumulusmx.db\|RecentData table in cumulusmx.db]] (uses SQLite). There are many software tools available that can read/edit SQLite3, this is not the place to get too technical, but take a look at '''sqlitebrowser''' in a Linux environment, and '''phpLiteAdmin''' in Microsoft Windows. Also, you could use any application that can read ODBC files (e.g. Libre Office's Base functionality) can read the MX databases.
		⚫	* Cumulus 2: Was held in an external SQLite database, together with other data that Cumulus 1 had held in text files, as above
	* Releases 3.0.0 to 3.11.4: Held in a SQLite database table that is stored in-memory within MX code		* Releases 3.0.0 to 3.11.4: Held in a SQLite database table that is stored in-memory within MX code
⚫	* Cumulus 2: Was held in an external SQLite database, together with other data that Cumulus 1 had held in text files
	* In the legacy Cumulus 1, the values were stored in an array held within the Cumulus code.		* In the legacy Cumulus 1, the values were stored in an array held within the Cumulus code.

Sfws: Minor resequencing of page

2021-08-08T11:35:31Z

Minor resequencing of page

Show changes

Sfws: /* What is Recent History? */ typo

2021-08-01T23:21:23Z

What is Recent History?: typo

← Older revision		Revision as of 23:21, 1 August 2021
Line 8:		Line 8:
	This effectively allows a request to Cumulus to specify 1 to 10 079 minutes ago, and it will retrieve the value for the nearest (if values are not stored for every minute) available time. The values available have increased as Cumulus has developed, if your release does not calculate all the derivatives you want, please see [[#If the derivative you want is not available in your Cumulus release]].		This effectively allows a request to Cumulus to specify 1 to 10 079 minutes ago, and it will retrieve the value for the nearest (if values are not stored for every minute) available time. The values available have increased as Cumulus has developed, if your release does not calculate all the derivatives you want, please see [[#If the derivative you want is not available in your Cumulus release]].

	The normal way to retrieve a specific value is by using [[Webtags#Recent_History\|recent history tag names]] and [[Webtags/Parameters#Input_modification_Parameters\|specifying how many minutes ago you want (to avoid using large numbers, parameters allow you to specify days ago, hours ago and minutes ago in any combination).		The normal way to retrieve a specific value is by using [[Webtags#Recent_History\|recent history tag names]] and [[Webtags/Parameters#Input_modification_Parameters\|specifying how many minutes ago]] you want (to avoid using large numbers, parameters allow you to specify days ago, hours ago and minutes ago in any combination).

	However, data from recent history is also used when the legacy Cumulus is generating images containing graphs for its [[Customised_templates#The_Standard_Templates_for_Cumulus_1\|"Trends.htm"]] web page.		However, data from recent history is also used when the legacy Cumulus is generating images containing graphs for its [[Customised_templates#The_Standard_Templates_for_Cumulus_1\|"Trends.htm"]] web page.

	MX uses data from recent history in its [[API_Graph_Data\|Application Programming Interface]] for the [[MX_Administrative_Interface#Charts]], and uses data from recent history (using several different .json files), see [[Category:JSON_Files#Uploading_data_to_a_web_server_outside_MX]] to make the data available for plotting charts on your web server.		MX uses data from recent history in its [[API_Graph_Data\|Application Programming Interface]] for the [[MX_Administrative_Interface#Charts]], and uses data from recent history (using several different .json files), see [[:Category:JSON_Files#Uploading_data_to_a_web_server_outside_MX]] to make the data available for plotting charts on your web server.

	Finally, there are a number of internal calculations, such as calculating [[Average temperature]], [[Heat/cold degree days and Chill hours]], which (in MX, Steve Loft never specified how these work in the legacy software) use the recent history data.		Finally, there are a number of internal calculations, such as calculating [[Average temperature]], [[Heat/cold degree days and Chill hours]], which (in MX, Steve Loft never specified how these work in the legacy software) use the recent history data.

Sfws: Text for this page taken from web tag page

2021-08-01T23:19:44Z

Text for this page taken from web tag page

New page

{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to all Cumulus flavours (including the abandoned Cumulus 2).
[[Category:Terminology]]
=What is Recent History?=
The functionality was introduced from version 1.9.3 (beta build 1033 release 10 April 2012).

Recent history functionality in Cumulus is the ability to store weather values, at up to 1 minute intervals, for the last 7 days. It is important to realise, these are spot readings, and will miss any extremes that occur between the times when the recent history values are stored. The weather value that tends to vary the most is wind speed, so that is where peaks are most often missed. However, it is also important to realise that weather stations are not supplying values to Cumulus on a continuous basis, depending on the weather station type values may be available at various intervals, either every so many seconds or perhaps at a rate less than once a minute. This can mean that the readings stored on a 1-minute basis may actually be duplicates if the station has not supplied a new value since the previous minute.

This effectively allows a request to Cumulus to specify 1 to 10 079 minutes ago, and it will retrieve the value for the nearest (if values are not stored for every minute) available time. The values available have increased as Cumulus has developed, if your release does not calculate all the derivatives you want, please see [[#If the derivative you want is not available in your Cumulus release]].

The normal way to retrieve a specific value is by using [[Webtags#Recent_History|recent history tag names]] and [[Webtags/Parameters#Input_modification_Parameters|specifying how many minutes ago you want (to avoid using large numbers, parameters allow you to specify days ago, hours ago and minutes ago in any combination).

However, data from recent history is also used when the legacy Cumulus is generating images containing graphs for its [[Customised_templates#The_Standard_Templates_for_Cumulus_1|"Trends.htm"]] web page.

MX uses data from recent history in its [[API_Graph_Data|Application Programming Interface]] for the [[MX_Administrative_Interface#Charts]], and uses data from recent history (using several different .json files), see [[Category:JSON_Files#Uploading_data_to_a_web_server_outside_MX]] to make the data available for plotting charts on your web server.

Finally, there are a number of internal calculations, such as calculating [[Average temperature]], [[Heat/cold degree days and Chill hours]], which (in MX, Steve Loft never specified how these work in the legacy software) use the recent history data.

=How are recent history values stored?=

Depends on the release you are running:
* Release 3.12.0 onwards: Held in [[cumulusmx.db|RecentData table in cumulusmx.db]] (uses SQLite), so available outside MX
* Releases 3.0.0 to 3.11.4: Held in a SQLite database table that is stored in-memory within MX code
* Cumulus 2: Was held in an external SQLite database, together with other data that Cumulus 1 had held in text files
* In the legacy Cumulus 1, the values were stored in an array held within the Cumulus code.

==How does the data change?==

While Cumulus is left running,, every minute:
# The oldest set of records, in database table row number 0, (in legacy 1.9.3 or 1.9.4, in array element 0), is discarded
# Cumulus then works through subsequent table rows (or array elements), as it reads each one, it moves it to the previous (now empty) row (or array element)
# The number of table rows, or array elements, is actually variable, this is because of two reasons:
## Table rows, or array elements, will be less frequent (or even missing) for the period before Cumulus is started
## Cumulus uses current time, read from the computer, in its decision as to where to store each new recent history value set, so it can overwrite existing table rows, or array elements, if date-time matches one that already exists, such as in the hour after when clocks go back.

===Warning when Daylight Saving Time starts or ends===

Remember, ''when clocks go back'', that for a whole hour following the clock change, clock times (now in winter time) repeat the previous clock times (that were in summer time). The way that Cumulus has coded this functionality, instead of the new sets of spot values being added after the old rows/elements, they overwrite the old table rows, or array elements, for the same times. Consequently, you cannot retrieve any values for the times that have been overwritten, and you must be careful when you specify to Cumulus which values you wish to retrieve.

As for ''when the clocks go forward'', the new values do get added after the old rows/elements, but of course there will be one hour's worth of times that simply do not exist. Consequently, any attempt to retrieve values around the time of the clock change has to be careful not to specify any time in that non-existent period, as Cumulus will just return the value before, or the value after, depending on which is nearest.

Since SQLite database tables use the row number as the primary key (see [[cumulusmx.db]]), the datetime column does not have to contain a unique value, so technically it would be possible for MX to keep track of which row was last updated, so that when clocks go back, it writes to the next row, instead of overwriting the existing row with that datetime value.

===What happens when I need to stop and restart Cumulus?===

Cumulus software has been designed on the assumption that it is left running constantly. However, in practice users do stop and restart the software.

This might be for short periods while they upgrade to a new Cumulus release, or reboot their computer because of some issue (or to prevent a possible issue). Also, computers running Microsoft's operating system are periodically rebooted as part of the mandatory software updates.

Other people do not want to leave their computer running all the time, and deliberately switch it off when they are not using it for some other purpose.

Earlier, it was described how (in MX anyway) the data for plotting charts, and doing some key internal calculations, is derived from recent history data. When you restart Cumulus, you may be able to see the charts can include data plotted from before Cumulus restarted, but that the resolution has changed.

Some weather stations have a memory, or separate logging feature, that allows them to store weather data. Cumulus software, for these weather station types, has the option to read these history records from the weather station in its archive data reading process as the software starts. However, it is rare for the weather station to store data every minute, hence the resolution of archive data might be any period between every ten minutes and every 60 minutes, depending on settings. (There are advantages in the weather station uses the same period as is set for the various [[Monthly log files]] that Cumulus can update, and MX does include some optional single use functionality to try to force your weather station to use the same interval). This explains the change in resolution seen on charts, but obviously also affecting the internal calculations you can't see.

The recent history data collected while Cumulus is running is lost when you stop Cumulus, unless you are running release 3.12.0 or later, as explained below. If it is retained, obviously it depends how much time has passed with MX stopped, as to how much lies within the 7 day period and can be used after restart.

====MX release 3.12.0 (beta build 3134) onwards====

The recent history is stored in a SQLite3 database table [[cumulusmx.db#Release 3.12.0 onwards|RecentData]] and therefore if you stop MX, the recent history data up to the time MX stopped has become persistent, and is available when MX starts again. Thus the charts, and internal calculations, mentioned above can make use of recent history data from the previous Cumulus session for the period 7 days ago to the time when MX was stopped.

When you do restart MX, some data, for the period since the time when MX was stopped, may be available if your weather station can store historic data. On restarting MX can optionally (depending on setting and depending on weather station type) read in archive data from those historic records for that period. As mentioned in last sub-section, this archive data is at the station logging resolution, normally different to the 1-minute interval available while MX is running.

If Cumulus MX is offline for a prolonged period, (and when you first run 3.12.0, as the '''RecentData''' table does not yet exist) then all the data for the previous just over 10 thousand minutes will be at this lower station logging interval resolution, as none will be available from the previous run of MX.

====MX 3.0.0 to MX 3.11.4====

In this release range, MX holds the recent history values in a [https://cumulus.hosiene.co.uk/viewtopic.php?p=100098#p100098 SQLite database], but the database is held '''in-memory''', not as an external database, so it only exists while Cumulus is running. Therefore, if Cumulus stops, all this high resolution data is lost.

When you do restart MX, if your weather station can store historic data, then its logger is read during the restart as archive data for the period from 7 days ago until the time of restart, and obviously only available at the station logging interval resolution of that historic data (be it every 10 minutes, or every 30 minutes or whatever).

====Legacy builds at versions 1.9.3 and 1.9.4====

The 'recent historical data' is stored in an array stored by the Cumulus code, as explained earlier. This means it is lost when the legacy Cumulus is stopped.

The only way that the legacy Cumulus can build up the array for the period before Cumulus is restarted is if your weather station includes a memory that stores historical readings. I believe some builds had a bug relating to reading this data into the array, but such data read from your weather can be incorporated into the array elements for 7 days ago until when Cumulus was restarted.

=If the derivative you want is not available in your Cumulus release=

As Cumulus has developed, it has been able to calculate more [[Feels Like|weather derivatives]] and more recent history tag names have become available (sometimes recent history tag names have been added in a later release than the release that started calculated the derivative).

== Other weather derivatives ==

Although Humidex, 'Apparent Temperature', 'Feels Like temperature', and other derivatives Cumulus can calculate, are not available in all releases, they can be calculated in a script from recent 'outside temperature', 'wind speed', and 'relative humidity' values (using the same time selection for all). There are other derivatives that can be calculated similarly from a set of simultaneous values. Note that Cumulus 1 and MX do not always use identical formula, and although MX added Feels Like it has changed the formula a few times.

The relevant formulae using JavaScript, adjust for other languages, for some of these are shown below:

=== Canadian Humidity Index ===

If you are in USA and use Fahrenheit instead of Celsius, you will need to omit the 5/9 term, but as the index is dimensionless no other conversion is needed. This example is for 3 hours ago, change the input modification parameter to suit your need.

Cumulus 1:

H = <#RecentOutsideTemp h=3> + 5/9 * (6.1094 * Math.exp(5417.753 *(1/273.16 - 1/ (273.16 + <#RecentDewPoint h=3> )))-10);

Cumulus MX:

svp = 6.112 * Math.exp((17.62 * <#RecentOutsideTemp h=3) / (243.12 + parseFloat(<#RecentOutsideTemp h=3)));
H = (5/9 * (<#RecentHumidity h=3> /100 * svp - 10)) + <#RecentOutsideTemp h=3;

=== Apparent Temperature and Feels Like ===

Note this apparent temperature formula uses Celsius for temperature and '''metres per second''' for wind speed. You will need to do the appropriate conversions from the quoted recent history tags if you use different units. The Australian Apparent temperature formula is same for Cumulus 1 and MX:

var actualVaporPress = <#RecentHumidity h=3>/100) * 6.105 * Math.exp(17.27 * <#RecentOutsideTemp h=3>) / (237.7 + parseFloat(<#RecentOutsideTemp h=3>))));
var appTempDegC = parseFloat(<#RecentOutsideTemp h=3) + (0.33 * actualVaporPress) - (0.7 * <#RecentWindSpeed h=3>) - 4;

Feels Like was implemented as a recent history web tag at version 3.6.11 (see [[#Feels_Like|Feels Like section below Current condition web tags]]) for the gradual introduction of feels like elsewhere. For earlier MX versions, and if you are using Cumulus 1, you can calculate it:

The formulas below use Celsius for temperature and '''km per hour''' for wind speed. Again, you will need to do the appropriate conversions from the quoted recent history tags if you use different units.

Calculation from recent history tags is much more complicated because there are 3 different calculations: Feels Like reports exactly same as wind chill for temperatures '''below''' 10°C or 50°F so the WC here should equal <#RecentWindChill h=3>:
<pre>if(<#RecentWindSpeed h=3> < 4.828) WC = <#RecentOutsideTemp h=3>;
else{
wind_pow = Math.pow(<#RecentWindSpeed h=3>, 0.16);
WC = (13.12 + 0.1625 * <#RecentOutsideTemp h=3>) - (11.37 * wind_pow) + (0.3965 * <#RecentOutsideTemp h=3> * wind_pow);// Brackets used to ensure "+" is interpreted as addition not concatenation
} </pre>

For temperatures '''above''' 20°C or 68°F Feels Like uses a different way to calculate apparent temperature that it uses at these higher temperatures (this formula only used for 3.6.10 onwards):
<pre>var actualVaporPress = <#RecentHumidity h=3>/100) * 6.112* Math.exp((17.62 * <#RecentOutsideTemp h=3>)/(243.12 + <#RecentOutsideTemp h=3>)) / 10.0; // Not same as at build 3084
/* uses kilometres per hour for wind speed */
/* What Cumulus MX will use to calculate apparent temperature for feels like is changed very slightly */
if(<#RecentWindSpeed h=3> > 72) <#RecentWindSpeed h=3> =72;
AT= (1.04 * <#RecentOutsideTemp h=3>) + (2 * actualVaporPress) - (0.1805553 * <#RecentWindSpeed h=3>) - 2.7;</pre>

For in-between temperatures it uses a more complicated merge of the two formulas for AT and WC as defined above:
<pre>app_temp_mult = (<#RecentOutsideTemp h=3> - 10) / 10;
wind_chill_mult = 1 - app_temp_mult;

FL= AT * app_temp_mult + WC * wind_chill_mult;</pre>