5,838
edits
Reports folder: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
→MX: updated in line with text by Mark in forum
m (→MX: updated in line with text by Mark in forum) |
|||
| (5 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
{{Version badge Mx}}{{Version badge 1}}This page applies to both flavours. | {{Version badge Mx}}{{Version badge 1}}This page applies to both flavours. It has been updated to cover all releases up to 3.12.0. | ||
[[Category:Cumulus Files]][[Category:MX txt Files]] | [[Category:Cumulus Files]][[Category:MX txt Files]] | ||
=The Reports folder= | |||
All flavours of Cumulus software include in their release distribution, a sub-folder called '''Reports'''. Please note that it starts with a capital letter, whilst (for Cumulus 1 and 2, all; for MX, most) other folders have totally lower case names. | All flavours of Cumulus software include in their release distribution, a sub-folder called '''Reports'''. Please note that it starts with a capital letter, whilst (for Cumulus 1 and 2, all; for MX, most) other folders have totally lower case names. | ||
When you first install Cumulus this folder is empty (for technical readers, MX releases do have one hidden file in the folder, '''.gitignore'''), it is just created in case you decide to use report functionality. | |||
=Optional functionality= | =Optional functionality= | ||
| Line 18: | Line 22: | ||
|- | |- | ||
| Switch this functionality on by selecting '''NOAA Settings''' in the ''Settings'' menu. | | Switch this functionality on by selecting '''NOAA Settings''' in the ''Settings'' menu. | ||
(In image "month specification" is abbreviated to "MS") | (In image "month specification" is abbreviated to "MS") | ||
| Line 29: | Line 31: | ||
| [[File:Cumulus_Configuration_Menu.png]] | | [[File:Cumulus_Configuration_Menu.png]] | ||
|} | |} | ||
==NOAA Report Settings== | |||
The various settings are listed [[Cumulus.ini#Optional_Report_Settings|here for MX]] and [[Cumulus.ini_(Cumulus_1)#Section:_NOAA|here for the legacy software]]. You will get the maximum understanding from reading both those links. | |||
Basically, some settings determine text to appear in the reports, and others specify thresholds used for various calculations. There is a mixture of settings that have a default, and settings that are initially blank. | |||
There are defaults for the file names, and the next section will explain that although you can change these, such changes will stop you using any scripts provided for viewing these reports on your web site, so think about your technical skills and whether you can write your own scripts, or modify supplied scripts. | |||
City and State are American terminology. The actual content of the first might be village, town, district, or city. The actual content of the second might be a region, area name, or county. | |||
Looking at the other initially blank settings, most of these are for monthly figures. You decide where to get the figures from, your local meteorological service might publish climate figures based on 10-year or 30-year averages as required by World Meteorological Office, your tourist authority may have some figures for your area, or there might be another weather station not far away that you can ask. | |||
= NOAA style Report Naming = | = NOAA style Report Naming = | ||
| Line 60: | Line 75: | ||
The configuration file contains information like passwords, so it must not be accessible to your web site. The latest file name for your monthly and annual reports are available by using [[Webtags#Miscellaneous|web tags]] so can be included in a [[Cumulus template file]] that is [[Customised_templates#What_is_meant_by_.27Cumulus_processes_templates.27|processed by Cumulus]] and therefore that information can be passed to your web site. | The configuration file contains information like passwords, so it must not be accessible to your web site. The latest file name for your monthly and annual reports are available by using [[Webtags#Miscellaneous|web tags]] so can be included in a [[Cumulus template file]] that is [[Customised_templates#What_is_meant_by_.27Cumulus_processes_templates.27|processed by Cumulus]] and therefore that information can be passed to your web site. | ||
As stated earlier, the default web site provided with MX, and third party report viewers for your web site, assume that the files containing the report text are named by the default names. Third party report viewers will also make assumptions about the encoding used, based on the Cumulus default encoding at the time the third party viewer was written, '''that may not''' be the default MX now uses, or the default in Cumulus 1. | As stated earlier, the default web site provided with MX, and third party report viewers for your web site, assume that the files containing the report text are named by the default names. Third party report viewers will also make assumptions about the [[#Encoding|encoding used]], based on the Cumulus default encoding at the time the third party viewer was written, '''that may not''' be the default MX now uses, or the default in Cumulus 1. | ||
If you customise any part of the file name, you must write your own script to be able to interpret the file name to find reports for displaying on your web server. If you want to see these reports on your web server and are not able to write your own scripts, don't modify anything, and skip all instructions about possible alternatives. | If you customise any part of the file name, you must write your own script to be able to interpret the file name to find reports for displaying on your web server. If you want to see these reports on your web server and are not able to write your own scripts, don't modify anything, and skip all instructions about possible alternatives. | ||
==The prefix literal== | ==The prefix literal== | ||
| Line 152: | Line 166: | ||
|NOAAMO_February_2010.txt (for some English locales) | |NOAAMO_February_2010.txt (for some English locales) | ||
|} | |} | ||
=Daily automatic report generation= | |||
If you have switched on the NOAA report functionality, then in the end-of-day process, Cumulus will output the monthly and yearly reports for the day that has just ended. This means when a new month, or new year, starts, the report is only available from the second day. | |||
All reports are generated by processing data that Cumulus has already stored somewhere, see subsequent subsections as it varies by flavour. | |||
==MX== | |||
===NOAA reports in MX=== | |||
Steve Loft's beta 3.0.0 did not include any code for generating NOAA reports, therefore there is no connection between the way that Cumulus MX from release 3.1.0 generates the reports and the way the legacy Cumulus 1 generated reports. | |||
Mark Crossley does share the source code for MX, this shows that MX creates the monthly and yearly reports afresh each day, reading information from [[dayfile.txt]] for past days. This means there is no difference between a report produced automatically by MX at end of day, and a report you manually ask it to produce at any time. | |||
If you change threshold settings for degree days in MX, that only affects subsequent daily summary log entries, so again although the report shows the latest thresholds, these might not apply to all information on the report. You would need to use Mark's [[Calculate_Missing_Values#CreateMissing.exe|Create Missing utility]] to approximately recalculate all the daily summary log (dayfile.txt) entries before you could manually rerun report generation based entirely on the new thresholds. | |||
===Derivation of figures in Monthly NOAA report in MX=== | |||
Mark's '''monthly report''' uses: | |||
#Daily [[average temperature]] (see below as depends on release) | |||
#Highest daily [[Temperature_(and_humidity)_measurement|temperature]] and time (taken from fields 7 and 8 of the daily summary log) | |||
#Lowest daily temperature and time (taken from fields 5 and 6 of the daily summary log) | |||
#[[Heat/cold_degree_days_and_Chill_hours|Heating degree days]] (taken from field 41 of the daily summary log) | |||
#[[Heat/cold_degree_days_and_Chill_hours|Cooling degree days]] (taken from field 42 of the daily summary log) | |||
#Daily [[Rain measurement|Rainfall]] (taken from field 15 of the daily summary log) | |||
#[[Wind_measurement|Average Wind Speed]] | |||
#* (apparently from 3.12.0 calculated from the wind speeds recorded in [[Standard log files|log file for current month]]; MX has changed its processing so that several derivatives that were calculated from every reading received from weather station are now calculated from the spot readings that MX logs) | |||
#* (apparently for 3.1.0 to 3.11.9, calculated from '''Wind run''' in field 17 of daily summary log, by dividing by 24 - the number of hours in a day,) | |||
#Highest daily average wind speed and time (taken from fields 18 and 19 of the daily summary log) | |||
#Dominant Direction (taken from field 40 of the daily summary log) | |||
DAILY AVERAGE shown for each day of month on monthly report: | |||
* For releases 3.1.0 to 3.11.4, taken from field 16 of daily summary log file | |||
* From release 3.12.0, there is a choice between that integrated average (calculated from every temperature reading processed), and the WMO average based on limited manual observations (here calculated from adding fields 5 and 7, then dividing by 2) | |||
MONTHLY AVERAGE shown on monthly and annual reports | |||
* For the summary line at the bottom of the monthly table: | |||
** The average shown is the average of all the figures in the column above | |||
* For the line shown for each month on the yearly report: | |||
** For releases 3.1.0 to 3.4.3, the figure shown is based on the same calculation as the summary line of the monthly table | |||
** For release 3.4.4 to 3.11.4, the figure shown is based on averaging field 16 of daily summary log file for all days in that month (i.e. the integrated average calculated from all temperature readings available) | |||
** From release 3.12.0, there is the choice between averaging field 16, or averaging both fields 5 and 7 | |||
* For the summary line at the bottom of the yearly table: | |||
** For releases 3.1.0 to 3.4.3, the figure shown is based on averaging the figures in the column above, as if all months had the same number of days | |||
** For release 3.4.4 to 3.11.4, the figure shown is based on averaging field 16 of daily summary log file for all days in that year (i.e. the integrated average calculated from all temperature readings available) | |||
** From release 3.12.0, there is the choice between averaging field 16, or averaging both fields 5 and 7 | |||
==Cumulus 1 and Cumulus 2== | |||
Although Steve Loft never shared his code for Cumulus 1, he did hint that his software retained content for the current reports. | |||
In other words, the old report is used as the basis for any new one, the generation procedure just calculates the new line to add to a monthly report, or the line for this month in a yearly report. Then his generation procedure updates the summary. | |||
Consequently, it seems his reports were constructed from very accurate data as each line was being made as data was read from the weather station during the day. Steve warned that for reports produced normally (in end of day process) you would make the lines incompatible by any changes you make to settings, such as thresholds during the month or year covered by the report. This confirms that the legacy Cumulus does not recalculate earlier information according to latest settings, although it shows the latest thresholds in text on the report, as if they apply to all information on the report. | |||
You can of course change thresholds, Steve provided for that, by allowing the regeneration of past reports. Because Cumulus does not track whether you have changed thresholds, when you do choose to manually generate any reports in Cumulus 1, it ignores daily summary entries in dayfile.txt that might have been invalidated by subsequent changes to thresholds. As Cumulus does not have access to every reading it processed originally from the weather station, the manual report generation process uses the [[Standard log files|standard log file]] entries (these hold periodic spot values that typically miss any extremes), as the source and recalculates all the derived values shown on the report. Steve warned "Be aware that a regenerated report for a past period might not be quite as accurate as the report that Cumulus can generate as part of end of day processing", confirming that the processes for originally producing the report and for subsequently regenerating past reports were different. | |||
===Bug in some NOAA reports=== | |||
The average wind speed used for NOAA reports was, by a bug, based on midnight to midnight days regardless of rollover time in use, for any reports produced from when the report functionality was first added to Cumulus 1, up to 1.9.4 build 1085 only: | |||
*From 1.9.4 build 1086, the calculation is correctly based on the rollover time being used. | |||
=A brief history of these reports = | =A brief history of these reports = | ||
| Line 167: | Line 243: | ||
Mark Crossley added these reports to MX at release 3.1.0, using the same layout and same report naming as the legacy, but he had to reinvent the calculations. | Mark Crossley added these reports to MX at release 3.1.0, using the same layout and same report naming as the legacy, but he had to reinvent the calculations. | ||
= | =Monthly report= | ||
For all flavours of Cumulus, the summary at the base of the monthly report is calculated as follows: | For all flavours of Cumulus, the summary at the base of the monthly report is calculated as follows: | ||
| Line 197: | Line 252: | ||
* There are 4 thresholds for temperature, and the number of days below or above that threshold is counted | * There are 4 thresholds for temperature, and the number of days below or above that threshold is counted | ||
* For each rain threshold, a count is made of days with rainfall above that figure | * For each rain threshold, a count is made of days with rainfall above that figure | ||
=Annual Report= | =Annual Report= | ||
| Line 213: | Line 267: | ||
* for the the Mean Max, Mean Min, and Dep. From Norm columns, the figure shown is the arithmetic average of figures above | * for the the Mean Max, Mean Min, and Dep. From Norm columns, the figure shown is the arithmetic average of figures above | ||
* for the mean column, | * for the mean column, | ||
** from release 3.4.4, the figure shown is the mean of all entries for this year in | ** from release 3.12.0, there is (as [[#MX|described earlier]]) a choice about which fields in [[Dayfile.txt]], the daily summary file, are used to calculate this figure. | ||
** in earlier MX releases, and the legacy Cumulus, the figure shown is the arithmetic average of figures above | ** for releases 3.4.4 to 3.11.4, the figure shown is the mean of all entries for this year in field 16 of daily summary log file | ||
* totals of figures above are shown for the Heat Deg Days, Cool Deg Days, and Max/Min comparisons with thresholds | ** in earlier MX releases, and in the legacy Cumulus, the figure shown is the arithmetic average of figures above | ||
* totals of figures above are shown for the Heat Deg Days, Cool Deg Days, and Max/Min comparisons with thresholds (see [[#MX|earlier]] for the implications if you change those thresholds) | |||
==Precipitation table== | ==Precipitation table== | ||
| Line 229: | Line 284: | ||
=Format of reports= | =Format of reports= | ||
All reports are pure text files. However, they contain more than just A to Z, 0 to 9, and punctuation. The additional symbol characters included in the report mean that it is important that the report and whatever is being used to view it are set to use the same set of character codes, the way that binary representations of characters are made depends on encoding. | All reports are pure text files. | ||
However, they contain more than just A to Z, 0 to 9, and punctuation. The additional symbol characters included in the report mean that it is important that the report and whatever is being used to view it are set to use the same set of character codes, and the way that binary representations of characters are made depends on encoding. | |||
== Encoding == | == Encoding == | ||
| Line 270: | Line 327: | ||
The last line shown there is critical, it indicates that the web page uses "utf-8" encoding. | The last line shown there is critical, it indicates that the web page uses "utf-8" encoding. | ||
You will find that all standard web templates included with MX start as shown above. For Cumulus 1, from build 1094 up the various builds defined for final release, the above code is used. However for earlier builds of Cumulus 1, the standard web pages start as follows: | You will find that all standard web templates included with MX start as shown above. | ||
For Cumulus 1, from build 1094 up the various builds defined for final release, the above code is used. | |||
However for earlier builds of Cumulus 1, the standard web pages start as follows: | |||
<pre><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | <pre><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | ||
<html xmlns="http://www.w3.org/1999/xhtml"> | <html xmlns="http://www.w3.org/1999/xhtml"> | ||
| Line 276: | Line 337: | ||
<head> | <head> | ||
<meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type" /></pre> | <meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type" /></pre> | ||
The last line there shows how the original web templates (designed by Beth Loft) used the ISO 8859-1 character set. | The last line there shows how the original web templates (designed by Beth Loft) used the ISO 8859-1 character set. The original NOAA reports also used ISO-8859-1 encoding. | ||
As Cumulus 1 does not contain any web page to display NOAA reports, legacy Cumulus users were forced to install third-party scripts to display the reports. Unfortunately there was potential for inconsistency between the encoding selecting for the web page that included a place to display the report, the encoding the package expected Cumulus to use, and the encoding selected in the script that found the report and read it into the web page. Thus you can find in the support forum lots of posts from people who saw unexpected characters, or other errors in the reports when they used these third-party packages, due to encoding issues. | |||
===TECHNICAL BIT=== | ===TECHNICAL BIT=== | ||
'''With that introduction, you can now choose whether to read the rest of this section which uses more technical terminology.''' | '''With that introduction, you can now choose whether to read the rest of this section which uses more technical terminology.''' | ||
''Let me explain that technical term, essentially encoding refers to the character set used by any file''. | ''Let me explain that technical term, essentially encoding refers to the character set used by any file''. | ||
A computer uses binary, binary can only be in state 0 or state 1, so a combination of 0 and 1 states needs to be defined for every character you want to represent. What you can include in that character set depends to some extent on how many binary bits are used to be mapped to individual characters | A computer uses binary, binary can only be in state 0 or state 1, so a combination of 0 and 1 states needs to be defined for every character you want to represent. | ||
What you can include in that character set depends to some extent on how many binary bits are used to be mapped to individual characters. A single computer byte has 8 bits, but some installations might use 7 bits for characters and the last bit for parity or some other controlling use. | |||
If you use 7 bits, you have 127 combinations, enough for standard 26 letters in both capitals, and lower case, plus 10 digits (0 to 9), some punctuation, and some control characters (like new line, end of file, and so on). | |||
If you use 8 bits, a whole byte, you have 254 combinations, and you can start coping with accented letters, with alphabets that don't have 26 letters, and even add some symbols. | |||
You can have more than 8 bits, if you allow multiple bytes to specify a single character. However, this brings in extra complications for the encoding, as you need to define how many bytes are being used, and the order in which the bits within the multiple bytes are used, for example some encodings will work forward through bits within bytes, but backwards through the individual bytes. | |||
Obviously, once you start using more than one byte, you can have 16, 32, 64, or even more bits to use and can include lots more characters and the bigger character sets start including lots of symbols and the biggest add smilies or emotion icons. | |||
With any fixed number of bits available, there will be a limit to how many characters can be defined, and different organisations might select different characters to include. | With any fixed number of bits available, there will be a limit to how many characters can be defined, and different organisations might select different characters to include. Modern encodings often focus on including emotion icons and are written in the context of communicating messages. In older mathematical contexts, a division symbol or a degree symbol might be critical. In more technical contexts, the focus might be on different types of arrows, some other drawing aid, or other technical symbols. | ||
This | This is what leads to multiple encoding standards. Is a particular arrangement of bits to be used to represent the degree symbol, to represent an envelope symbol, to represent a flow chart segment, to represent a smiling face, or do you need several currency symbols? The general problem is that unless you match the encoding used initially, any retrieval cannot know what character to display for certain combinations of bits. | ||
The encoding variability even extends to lower and upper case for letters. Some encodings put capital letters at lower binary values than lower case letters, and some put capitals at the higher binary values. | |||
