Dayfile.txt: Difference between revisions
m (→List of Fields: Sunshine hours note) |
m (Correcting Typos (mostly missing, but critical, colon characters in links), and mention of some more release 3.20.0 changes) |
||
(12 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to both Cumulus flavours. [[Category:Files_with_Comma_Separated_Values]] |
|||
<big>'''This article is about the Daily Summary logging file'''</big> |
|||
As part of a redevelopment of Wiki, this page has been simplified, by moving some content to new pages. Old links in the support forum, that were to content no longer on this page, will bring you here. |
|||
= Introduction = |
|||
'''Content previously on this page has been moved as follows''': |
|||
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. In all flavours of Cumulus, this file is only updated (with exclusive lock) during the end of meteorological day process. In that process it is also read, if the generation of NOAA reports has been requested. |
|||
* Explanations about Cumulus terminology can be accessed from [[:Category:Terminology]] page |
|||
* Each line in this file represents one day, but the start time need not be same for all fields, |
|||
** so please see [[Meteorological_day|Meteorological day]], and [[today.ini]] for more details about that |
|||
* Advice regarding editing this file has been moved to [[Amending dayfile]] page, this also covers date-separator issues and dealing with missing or corrupted lines |
|||
* General advice about Cumulus files with a '''.txt''' extension has been consolidated on [[:Category:Files_with_Comma_Separated_Values]] page |
|||
* Advice about correcting any rogue extreme figures inadvertently stored in this file has been consolidated in new [[Correcting Extremes]] page |
|||
* If some lines in your file have fewer fields than other lines the advice has been consolidated on [[Calculate Missing Values]] page |
|||
** For MX there is a [[Calculate_Missing_Values#CreateMissing.exe|Create Missing Utility]], which checks spot readings in the [[Standard_log_files|MMMyylog.txt]] files, adds any missing derivative spots (e.g. heat index) and uses these figures to recalculate daily derivatives and uses those to replace missing fields/lines in dayfile.txt |
|||
** For the legacy Cumulus, there was a [[Amending_dayfile#Create_Missing_on_legacy_dayfile_editor|dayfile editor with create missing option]], that performed equivalent function |
|||
** If you import historic data from before you started using Cumulus into [[Standard log files]], |
|||
** then see [[Calculate Missing Values]] page for how to generate new lines in dayfile.txt |
|||
* Cumulus MX is more fussy, than Cumulus 1, about various formatting issues, see below, also see [[Migrating_from_Cumulus_1_to_MX#dayfile.txt]] section for more advice |
|||
The content that remains on this page is summarised by the table of contents that follows. |
|||
Both Cumulus 1 and MX have ways to [[Amending dayfile|edit '''dayfile.txt''']] while Cumulus is running. Cumulus 2 does not even allow you to view this log file. |
|||
==Creating dayfile.txt== |
|||
=About this file= |
|||
This file is not included in any release distribution of Cumulus. |
|||
* This Wiki page describes one of the [[:Category:Cumulus Files|files]] not included in any release download. |
|||
Cumulus software creates the file, if the file does not exist, when the software does its first end of day action. |
|||
* This daily summary file (Steve Loft shortened that to "day" plus "file") uses a ".txt" extension, and is essentially a plain text file in CSV format. |
|||
** CSV stands for "Comma Separated Values", but in the Cumulus context the field separator can be other characters including ";" or "|" |
|||
* The file is created in [[data folder|data sub-folder]] of your Cumulus installation when Cumulus needs to store its first line in this file. |
|||
* Cumulus MX reads the whole of this file into an internal array (stored in RAM) each time you restart the software, |
|||
** Please see [[Amending_dayfile#Warning_if_editing_outside_MX|here for implications]] |
|||
** Therefore MX users who move to a different location should archive the old "dayfile.txt", to avoid any discontinuities in any graphs plotted from active file |
|||
Whenever Cumulus does an end of day action (either in normal processing or in catch-up on a restart of Cumulus while processing archive data), Cumulus will create a new line in '''dayfile.txt''' from what it has stored in [[Today.ini#When_Cumulus_is_running|today.ini]] and at that link you can read how the parameters in '''today.ini''' map across to fields in '''dayfile.txt'''. |
|||
([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Ini_Files|.ini files]]). |
|||
If you import historic data from before you started using Cumulus, and have this in [[Standard log files]], then the daily summary for imported data can be converted into dayfile.txt line format (and a dayfile.txt file created/updated) using any of the approaches described on [[Calculate_Missing_Values|calculate missing values]] page. |
|||
==How Cumulus Creates and Updates this file== |
|||
==[[File:Badge vMx.png]] WARNING== |
|||
* Cumulus reads values supplied by your weather station, (either directly while Cumulus running, or for some weather station types can read historic data during catch-up on restarting Cumulus) |
|||
It is important to note that MX is very fussy about consistency in all lines of '''dayfile.txt'''. |
|||
* Cumulus converts them to the units you prefer, |
|||
* Cumulus applies any calibration (multiplier and offset) you have set, |
|||
* For a sub-set of those readings (perhaps those every minute if readings are collected every 10 seconds), the spot values of source items like temperature, humidity, wind speed, can be used in calculations of derived items like "wind chill", "dewpoint" and "feel-like" temperature. |
|||
* Cumulus then sees if the resulting soource, or derived, value implies [[:Category:Ini Files|any extreme records file]] needs to be updated |
|||
** Daily extremes are held in [[today.ini]] and that is main source used when a new line is added to "dayfile.txt" |
|||
MX reads the entire '''dayfile.txt''' file, to drive the '''historic charts functionality'''; both in the [[MX Administrative Interface|admin interface]], and the example included [[New Default Web Site Information|web page]]. Consequently, any error in say the date field (or change of separating character) will stop historic charts working. |
|||
==Why this file should be backed up== |
|||
Date field: Cumulus 1 did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file. |
|||
* This file contains daily extremes, the accuracy of those daily extremes depends on the interval between the spot readings that are used as explained above. |
|||
Real number fields: MX uses the locale to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. |
|||
* If the file is corrupted, and not regenerated from a back-up, the only source of spot readings is the [[Standard_log_files|MMMyylog.txt]] files, and their interval might be only every half an hour (it has to be an exact fraction of an hour). |
|||
* As explained [[Monthly_log_files#Enhancement never implemented|here]] minima and maxima between such logging intervals are not recorded by Cumulus software |
|||
* Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file, |
|||
Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough to find a file that has not been corrupted: |
|||
Number of fields: The number of fields in the file increases in various versions as shown at the end of this article. C1 can cope with 15 to 45 fields in each line. MX (except for the early releases) assumes 48 fields in each line. If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than 48 fields, you can add the missing derived fields, please see [[Calculate Missing Values]] page. |
|||
* The file is backed up when MX is restarted into a date/time stamped sub-folder of the [[Backup folder|'''backup''' folder]] |
|||
*The file is also backed up (to a date/time stamped sub-folder within '''daily''' sub-folder of that backup sub-folder) during the end of day process, but (depending on the release you are running) the back-up copy may, or may not, include a line for the previous day (the line that is appended in the same end of day) |
|||
Retention of these back ups: |
|||
Please see [[Amending dayfile]] page for information about how to edit this file. |
|||
* [[File:Badge vMx.png]] MX only keeps the last 9 of the date/time stamped sub-folders. |
|||
* [[File:Badge v1.png]] Legacy Cumulus 1 only keeps up to 8 of the date/time stamped sub-folders. |
|||
== Specific issues for MX == |
|||
Please see [[Correcting Extremes]] page for information about fixing where the highest/lowest/total recorded for day has been corrupted by rogue values. |
|||
WARNING: It is important to note that some releases of MX are very fussy about consistency in all lines of '''dayfile.txt'''. Earlier releases tended to have better compatibility with the legacy software. Subsequent releases expected every line to be expressed exactly as specified in the locale. MX in its latest release is trying to cope better with inconsistent date separators. |
|||
Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file. |
|||
Real number fields: MX uses the '''locale''' to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the '''Region''' settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if ''list separation character'' was comma) was fairly tolerant when reading old data in files. At the time of writing this, it was proposed the next major release will force this file to use decimal points (full stops not commas) regardless of locale. This proposal may or may not happen, the point is that information here might be incorrect for the release that you are using. |
|||
{{TOCright}} |
|||
Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours. |
|||
Number of fields: The number of fields in the file increases in various versions as shown in [[#List_of_fields]]. |
|||
* C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields. |
|||
* Each release of MX expects a particular number of fields in each line of file; MX has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an internal (RAM held) array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file. |
|||
** If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, there is a utility to run that will create a new dayfile.txt with all fields populated, please see [[Calculate Missing Values]] page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added, and it uses various software libraries contained within .dll files specific to a particular CumulusMX.exe release. |
|||
==How to view or edit this file== |
|||
== Changes in different releases of Cumulus == |
|||
The detailed information has been moved to [[Amending dayfile|viewing/editing '''dayfile.txt''' page]]. |
|||
Be aware that this article was written for Cumulus 1, amended to also cover MX 3.0.0. Whilst the list of fields has been kept updated for more recent MX versions, other parts of this article may not be applicable to latest MX version, check for edits by developer. |
|||
An editor has been included within recent Cumulus releases of each flavour: |
|||
=== Notes for Legacy Cumulus=== |
|||
* [[File:Badge vMx.png]] Available from release 3.4.5 (13 Mar 2020): In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile'' |
|||
{{Version badge 1}} |
|||
* [[File:Badge v1.png]] Available from version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt'' |
|||
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round. |
|||
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order) |
|||
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types. |
|||
Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file. |
|||
*The original Cumulus software reads the contents of '''dayfile.txt''' for many of the display and edit menu options that can be selected from the main Cumulus 1 screen. |
|||
**As the file is read each time you select to display a period, or update for a new period, you will see the results of any edit of dayfile.txt, whether with in-built editor or an external editor. |
|||
The MX editor does not let you change the date, you can only change other fields or delete an entire line, so you have no control over the order of the lines in the file. |
|||
*'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round. |
|||
*'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order) |
|||
*There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types. |
|||
== When Cumulus processes the end of the (meteorological) day == |
|||
*A new row is appended to dayfile.txt, the values are prepared from reading "today.ini" file, not all values available in "today.ini" are stored in dayfile.txt. |
|||
*Some of this information is also stored in [[yesterday.ini]]. |
|||
*Back ups of both today.ini, and dayfile.txt, log files ''in their once a day state (either '''before''' or '''after''' changes in the end of day update'', depending on which version you have running) are copied to [[Backup folder]]. |
|||
==Reading the file== |
|||
===Options for reading dayfile.txt for other uses=== |
|||
See [[Daily Summary]] page for a full discussion. |
See [[Daily Summary]] page for a full discussion of ways, external to Cumulus, to read this file. |
||
*Some people take a copy of the local dayfile.txt, and use it locally for other purposes. |
|||
*A number of third-party routines that make use of this file (and in some cases, others) are listed at [[:Category:WebTools]]. |
|||
** Some people use [[Upload_Dayfile|the method described here]] or [[Toolbox|Cumulus Toolbox]] to get a copy of the local file to use on their web server. (If your web server is local, you can simply copy the file to that server, once a day). |
|||
** Search in the Cumulus support forum for examples of third-part JavaScript projects that read the web copy, for example to insert data for one year ago, or to enable extremes for each day in a week to be included in a web page. |
|||
**Dayfile.txt (and other files!) are used as described at the [[CumulusUtils]] link. |
|||
*Other people load the contents of their dayfile.txt into a database table, allowing SQL enquiries to efficiently display, or summarise, the contents of the file. |
|||
**Cumulus MX provides optional functionality to create a database table, and to automatically insert a new row each time a new line is added to the local log file. |
|||
Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in vaCumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error (line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged).rious other circumstances depending on the release you are running. |
|||
=== Populating a database table === |
|||
* [[File:Badge v1.png]] The legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored: |
|||
*{{Version badge 1}} The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have been edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using. |
|||
** The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], to populate these screens Cumulus has to read all of the [[:Category:Files with Comma Separated Values|Files with Comma Separated Values]] as the derivatives shown generally do not match the extremes tracked in [[:Category:Ini Files|.Ini files]]. |
|||
*[[File:Badge vMx.png]] Please see [[MX_Administrative_Interface#Standard_Daily_Summary_Table]] section for details of how '''CumulusMX.exe''' has an optional feature with a standard option to insert a new row into a database table holding columns relating to the dayfile.txt fields. Either (for all early MX releases) '''ExportMySql.exe''', or (for all later MX releases) '''ExportToMySQL.exe''', can update the database table with past rows. |
|||
** The '''Select a graph''' feature also uses all of the [[:Category:Files with Comma Separated Values|Files with Comma Separated Values]]. |
|||
** The end of day action uses a simple "append" instruction, so it does not need to read the file |
|||
* [[File:Badge vMx.png]] As already mentioned, MX reads the entire '''dayfile.txt''' file when it is first started into an internally held array. That internal copy was introduced in release 3.9.2 - b3097 (7 Dec 2020), it was needed to drive the [[Highcharts_-_Historic|'''historic charts functionality''']]; both in the [[MX Administrative Interface|interface]], and the example included [[New Default Web Site Information|Historic Charts web page]]. |
|||
** From release 3.20.0 - b3200 (21 Aug 2022) there is a '''This Period Highs and Lows''' web page (accessed from ''Records'' menu) which is similar to the ''screens for viewing data for various periods'' except that the extremes it shows match those tracked in [[:Category:Ini Files|.Ini files]]. |
|||
** From release 3.20.0 - b3200 (21 Aug 2022) there is an option to '''Reload dayfile''' in a ''Utils'' menu, this updates the internal held array by reading the file again; essential if an edit has been made to the file. |
|||
==Line numbering and Error messages== |
|||
==== Using a database table ==== |
|||
Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error. |
|||
In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log. |
|||
Cumulus MX stores the contents of the file in an internal two dimensional array. The outer dimension is identified by numbering (from zero) what were lines in the file. The inner dimension has that line number as element zero, and the fields taken from file are therefore numbered from one. |
|||
For examples of some of the third party tools (Cumulus [[Category:User_Contributions|user contributions]]) using the database daily summary table see [[Daily Summary#Some_example_Scripts|here]]. |
|||
Obviously, line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged in the file. This is why you will see line numbers quoted in any error messages related to this file. |
|||
In my case, I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table, i.e. I have one database table column for each of the weather derivatives I show on my web page that show this month's values; it is many more derivatives than are shown on the standard web page, but some are initially hidden. Consequentially, when my daily update script detects from the date that it is processing the last day of a month, it then starts another script that reads all the rows in the daily summary table for that month, and stores the highest/lowest/total (as relevant) in my monthly_summary table (nothing to do with the "monthly" table that MX can generate from its standard log file). This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Just another example of how much you can get from just one (daily summary) log file! |
|||
Some of the editors you may use to view/edit the file may number the lines (in some cases always, in others as an option). If you have seen an error message, then to correct the file you need to use a tool that does number the lines in order to find where the correction is needed. Error messages report just the first error found, so check subsequent lines in case they have same error. |
|||
==== Alternative schemas ==== |
|||
Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files. |
|||
*{{Version badge 1}} |
|||
**With Cumulus 1, you would need to be a programmer and write your own script to update the database table with your own schema. You might use the importCumulusFile article to start you off. |
|||
**You might also, as I did, want your script to validate what it reads from the daily summary log to ensure only valid numbers and times are stored in your database table, while any invalid inputs are stored as nulls by your script. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the contents of the Weather Diary, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). When I used Cumulus 1 I wrote a PHP script to find all these additional values, for example it reads the [[today.ini]] and [[month.ini]] log files as stored in the end of day backup (not the ones being updated for new day in data folder), and the [[log.xml|weather diary in log.xml in data folder]]. |
|||
*[[File:Badge vMx.png]] |
|||
**MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned. |
|||
**MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do. |
|||
== When Cumulus is restarted after a break in running == |
|||
* It reads the daily summary log (dayfile.txt) and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|this Cumulus 1 FAQ]]) |
|||
* Thus you must not have another process attempting access to the daily log when Cumulus is re-starting. |
|||
* For Cumulus 1, back ups of 8 selected log files including dayfile.txt that are copied to start-up folders in the 'cumulus\backup' folder, the last 8 start-up folders only are retained. |
|||
* For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else. |
|||
* For MX, except early releases which did not have this functionality, the whole of dayfile.txt is read when MX starts as it provides the daily data for the historic charts (both in admin interface charts and default web site chart data). |
|||
If your weather station includes it own data logger, then Cumulus does a catch-up during which Cumulus will read "archived records" from that weather station log, and when it detects the archive log has moved to a new meteorological day, Cumulus will run the "end of day" process that moves what is in [[Today.ini#Restart_and_Catch-up|today.ini]] into a new line in dayfile.txt as described in that linked section. |
|||
= List of fields in dayfile.txt = |
= List of fields in dayfile.txt = |
||
For your installed build please see ''[[dayfileheader.txt]]''. This file identifies all the fields in each line of the '''dayfile.txt''' file in the release you are running. |
|||
==The number of fields in a line== |
|||
That file has changed location from release 3.20.0 - b3200 (21 Aug 2022), it is now in [[data folder]], for Cumulus 1 and all earlier MX releases it was in same directory as the Cumulus executable. |
|||
The single file '''dayfile.txt''' can contain lines created over a long period of time ([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Log Files|.ini files]]). |
|||
As explained earlier, the number of fields in a line has (with one exception when fields were added in error) increased as the Cumulus software has been developed. The table below therefore groups the fields under labels that indicate which version/release introduced them. |
|||
For some people, the file was first created by Cumulus 1 (at various versions over time), and then moved to MX. Such files may have different numbers of fields depending on when each line was either created or last updated. Cumulus does not in normal operation modify earlier lines, but both Cumulus 1 and MX provide editors where it is possible to modify any line, and any line modified will from then onwards have the number of fields defined for the Cumulus version being used.. |
|||
==Field numbering== |
|||
There is advice on editing the file on [[Calculate_Missing_Values#Missing_fields_.28or_missing_lines.29_in_dayfile.txt|Calculate Missing Values page]], using the Cumulus in-built editor. Should you want to edit the file outside Cumulus, a "Comma Separated Value" editor has advantage of being able to maintain the differing existing lengths for each line. One example of such a tool in the Microsoft Windows environment is [https://csved.sjfrancke.nl/ freeware CSV editor software by Sam Francke for Windows]. If you do edit the file (within Cumulus or outside), there are [[Calculate_Missing_Values#Rules_when_editing_daily_summary_log|Rules to follow]]. |
|||
The table below was first created when the only flavour of Cumulus software available was what we now call the legacy Cumulus 1. Back then the fields '''were''' numbered starting from zero. This reflected the way fields were identified when processed as elements of an single-dimension array variable representing a line. |
|||
=== Variation by Cumulus version === |
|||
The old numbering from zero had two advantages: |
|||
The number of fields in '''dayfile.txt''' has grown as time has gone by; simply because Cumulus's functionality has been extended both to cover more sensors and to calculate more derivatives. The only way that Cumulus can retain a daily summary for past days is by using this file. |
|||
# It stressed that the date field was different to the rest, all other fields were either values or time-stamps |
|||
# Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a Cumulus 1 line was converted to an array, and you needed to address a single field. |
|||
Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering. As explained [[#Line numbering and Error messages|above]] the fields are now numbered starting from 1 to fit in with an internal array created when the log file is read, the processing code adds a line number in front of the date field on each array 'line' it holds. |
|||
'''For your installed build please see ''dayfileheader.txt'' (stored within the folder that contains your Cumulus executable), as that will list which fields your Cumulus installation uses.''' |
|||
== Information shown in the table == |
|||
If you have been using Cumulus for a while, you may wonder which of your log file lines might be shorter, so the table below shows fields grouped by the Cumulus version when those fields were added. |
|||
* The date '''is supposed to''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field. |
|||
For the original Cumulus software, each line of this file can contain anything from 15 to 45 fields, and having some lines shorter than others does not matter. |
|||
** The dates, as mentioned earlier, should be in ascending order. |
|||
** Please see [[amending dayfile]] page for more information on dates |
|||
** Both Cumulus 1 and MX may get confused if you restart the software close to rollover time, and in that confused state they may create a second line with same date as a date already in the file. |
|||
** If you regularly restart Cumulus close to rollover time, then a whole sequence of dates can be repeated either as a block or as pairs. |
|||
* The remaining fields are all either numerical values, or a time paired with (except for first wind field) preceding numerical value. |
|||
** Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed). Cumulus MX only validates that each field is of correct type, it does not recognise any relationships between fields. |
|||
*The alphabetic column identifiers used by many spreadsheet editors are shown in the table |
|||
** IMPORTANT: If you do use a spreadsheet editor, do ensure '''all columns are set to "text" format''', |
|||
** Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, or let it change the number of decimal places. |
|||
* The type of field is shown in the table: |
|||
** Note you must not include a sign for an unsigned field, |
|||
** Note you can not specify a decimal comma/point in an integer field, |
|||
** Note all fields of type time must use 5 character "HH:mm" format, although Cumulus 1 will accept other separator characters where ":" shown. |
|||
* The field description is shown, together with references to where that terminology is explained |
|||
For Cumulus MX, several of the earliest releases supported only the same 45 fields as used by 1.9.4, a few early releases had 50 fields, just one particular release supported 54 fields (in error), then several releases supported exactly 52 fields, but from release 3.12.0 MX expects exactly 53 fields. |
|||
If you have any lines with 15 to 52 fields then MX can still read that line, but if you edit that line using the editor in MX, it will show all fields when you edit it, and when you save that line. |
|||
Please note that if you use MX, then there is a utility program that can insert missing fields for you. Please see [[Calculate_Missing_Values#CreateMissing.exe|here]]. |
|||
== Information shown in the table == |
|||
== Table listing Fields == |
|||
*The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, the processing code adds a line number in front of the date field on each line it holds. |
|||
** The Cumulus MX user may not be aware of this happening as it is within the internal workings, where data from the file is transferred to an array, or from the array is written back to the file. |
|||
**By using line numbers, MX is able to identify which line has been deleted or edited ignoring the date (although unlike Cumulus 1, MX does not allow you to change a date) |
|||
**Cumulus 1 does not actually number lines, it counts lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number where it first found an error. |
|||
*The original table below was for Cumulus 1 and then field number '''was''' starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages: |
|||
*#Cumulus 1 stressed that the date field was different to the rest, as it was used as identifier. The date '''must''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field. |
|||
*#The remaining fields were all either numerical values, or a time paired with preceding numerical value. Cumulus 1 actually forces this pairing. |
|||
**Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a line was converted to an array, and you needed to address a single field. |
|||
*The alphabetic column identifiers used by many spreadsheets are shown, please see warnings about using spreadsheets for editing earlier on this page |
|||
*The type of field is shown, you must not put a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format |
|||
*The field description is shown, together with references to where that terminology is explained |
|||
=== List of Fields === |
|||
{| class="wikitable" border="1" |
{| class="wikitable" border="1" |
||
|- |
|- |
||
Line 156: | Line 160: | ||
|- |
|- |
||
|0 |
|0 |
||
| |
|Colspan="4" style="background: pink;"|For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line. |
||
If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1. |
If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1. |
||
Line 396: | Line 400: | ||
|[[Heat/cold degree days and Chill hours | Cooling degree days]] |
|[[Heat/cold degree days and Chill hours | Cooling degree days]] |
||
|- |
|- |
||
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist) |
|colspan="4" style="background:lightgray;"|The next two pairs were added in legacy version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist). |
||
Fields listed up to those following here applied to the final legacy Cumulus 1.9.4 and formed the basis for early releases of Cumulus MX. |
|||
|- |
|- |
||
|43 |
|43 |
||
Line 418: | Line 424: | ||
|Time of high UV Index |
|Time of high UV Index |
||
|- |
|- |
||
|colspan="4" style="background:lightgray;"|The next two pairs were added in |
|colspan="4" style="background:lightgray;"|The next two pairs were added in MX release 3.6.0, 2 more derived values and their times |
||
|- |
|- |
||
|47 |
|47 |
||
Line 440: | Line 446: | ||
|Time of low feels like temperature |
|Time of low feels like temperature |
||
|- |
|- |
||
|colspan="4" style="background:lightgray;"|The next two pairs were added in |
|colspan="4" style="background:lightgray;"|The next two pairs were added in release 3.6.12 |
||
*Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times). |
*Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times). |
||
**The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor. |
**The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor. |
||
*From |
*From release 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error. |
||
|- |
|- |
||
|51 |
|51 |
||
Line 454: | Line 460: | ||
|AZ |
|AZ |
||
|5 characters |
|5 characters |
||
|Time of high Humidex |
| style="background:pink;"|Time of high Humidex |
||
'''Bug for releases 3.13.0 to 3.14.2 inclusive''': The major code rewrite for release 3.13.0 replaced the previous code for the processing for all Cumulus files, the new code incorrectly stored ''Time of high feels like temperature'' in this field for all these releases! Corrected in minor code rewrite for 3.14.3 - b3163 25 Jan 2022 (not released to public until 3.15.0 - b3169 Released 31 Jan 2022) |
|||
|- |
|- |
||
|colspan="4" style="background:lightblue;"|Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in |
|colspan="4" style="background:lightblue;"|Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in any other version, so have labelled them as error. |
||
|- |
|- |
||
| 53 ('''error''') |
| 53 ('''error''') |
||
| BA |
| BA |
||
|signed decimal |
| signed decimal |
||
|Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other |
| Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other release) |
||
|- |
|- |
||
| 54 ('''error''') |
| 54 ('''error''') |
||
| BB |
| BB |
||
| 5 characters |
| 5 characters |
||
| Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other |
| Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other release) |
||
|- |
|- |
||
|colspan="4" style="background:lightgray;"|The next value was added in 3.12.0 |
|colspan="4" style="background:lightgray;"|The next value was added in release 3.12.0 |
||
|- |
|- |
||
| 53 (new) |
| 53 (new) |
||
Line 474: | Line 482: | ||
| unsigned decimal |
| unsigned decimal |
||
| [[Heat/cold_degree_days_and_Chill_hours#Chill_Hours_and.2For_Air_Frost|Cumulative Chill Hours]] since start of season |
| [[Heat/cold_degree_days_and_Chill_hours#Chill_Hours_and.2For_Air_Frost|Cumulative Chill Hours]] since start of season |
||
|- |
|||
|colspan="4" style="background:lightgray;"|The next value and time fields were added in release 3.20.0 |
|||
|- |
|||
| 54 |
|||
| BB |
|||
| unsigned decimal |
|||
| Highest rainfall total in (an approximate, as it depends upon timing of earlier entries in [[Recent history]]) 24 hour period ending in [[Meteorological day]] |
|||
|- |
|||
| 55 |
|||
| BC |
|||
| 5 characters |
|||
| Time when (approximate) 24 hour period for rainfall total above ends |
|||
|} |
|} |
||
==Example of the file== |
==Example of the file== |
||
An extract of a few lines of |
An extract of a few lines of a dayfile.txt |
||
<pre> |
<pre> |
||
Line 495: | Line 515: | ||
07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7 |
07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7 |
||
</pre> |
</pre> |
||
[[Category:MX txt Files]][[Category:Cumulus 1]][[Category:Cumulus MX]] |
Latest revision as of 07:14, 27 September 2022
This Wiki page applies to both Cumulus flavours.
As part of a redevelopment of Wiki, this page has been simplified, by moving some content to new pages. Old links in the support forum, that were to content no longer on this page, will bring you here.
Content previously on this page has been moved as follows:
- Explanations about Cumulus terminology can be accessed from Category:Terminology page
- Each line in this file represents one day, but the start time need not be same for all fields,
- so please see Meteorological day, and today.ini for more details about that
- Advice regarding editing this file has been moved to Amending dayfile page, this also covers date-separator issues and dealing with missing or corrupted lines
- General advice about Cumulus files with a .txt extension has been consolidated on Category:Files_with_Comma_Separated_Values page
- Advice about correcting any rogue extreme figures inadvertently stored in this file has been consolidated in new Correcting Extremes page
- If some lines in your file have fewer fields than other lines the advice has been consolidated on Calculate Missing Values page
- For MX there is a Create Missing Utility, which checks spot readings in the MMMyylog.txt files, adds any missing derivative spots (e.g. heat index) and uses these figures to recalculate daily derivatives and uses those to replace missing fields/lines in dayfile.txt
- For the legacy Cumulus, there was a dayfile editor with create missing option, that performed equivalent function
- If you import historic data from before you started using Cumulus into Standard log files,
- then see Calculate Missing Values page for how to generate new lines in dayfile.txt
- Cumulus MX is more fussy, than Cumulus 1, about various formatting issues, see below, also see Migrating_from_Cumulus_1_to_MX#dayfile.txt section for more advice
The content that remains on this page is summarised by the table of contents that follows.
About this file
- This Wiki page describes one of the files not included in any release download.
- This daily summary file (Steve Loft shortened that to "day" plus "file") uses a ".txt" extension, and is essentially a plain text file in CSV format.
- CSV stands for "Comma Separated Values", but in the Cumulus context the field separator can be other characters including ";" or "|"
- The file is created in data sub-folder of your Cumulus installation when Cumulus needs to store its first line in this file.
- Cumulus MX reads the whole of this file into an internal array (stored in RAM) each time you restart the software,
- Please see here for implications
- Therefore MX users who move to a different location should archive the old "dayfile.txt", to avoid any discontinuities in any graphs plotted from active file
(speciallog.txt is another log file that contains all dates in a single file, as do all the .ini files).
How Cumulus Creates and Updates this file
- Cumulus reads values supplied by your weather station, (either directly while Cumulus running, or for some weather station types can read historic data during catch-up on restarting Cumulus)
- Cumulus converts them to the units you prefer,
- Cumulus applies any calibration (multiplier and offset) you have set,
- For a sub-set of those readings (perhaps those every minute if readings are collected every 10 seconds), the spot values of source items like temperature, humidity, wind speed, can be used in calculations of derived items like "wind chill", "dewpoint" and "feel-like" temperature.
- Cumulus then sees if the resulting soource, or derived, value implies any extreme records file needs to be updated
- Daily extremes are held in today.ini and that is main source used when a new line is added to "dayfile.txt"
Why this file should be backed up
- This file contains daily extremes, the accuracy of those daily extremes depends on the interval between the spot readings that are used as explained above.
- If the file is corrupted, and not regenerated from a back-up, the only source of spot readings is the MMMyylog.txt files, and their interval might be only every half an hour (it has to be an exact fraction of an hour).
- As explained here minima and maxima between such logging intervals are not recorded by Cumulus software
- Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,
Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough to find a file that has not been corrupted:
- The file is backed up when MX is restarted into a date/time stamped sub-folder of the backup folder
- The file is also backed up (to a date/time stamped sub-folder within daily sub-folder of that backup sub-folder) during the end of day process, but (depending on the release you are running) the back-up copy may, or may not, include a line for the previous day (the line that is appended in the same end of day)
Retention of these back ups:
- MX only keeps the last 9 of the date/time stamped sub-folders.
- Legacy Cumulus 1 only keeps up to 8 of the date/time stamped sub-folders.
Specific issues for MX
WARNING: It is important to note that some releases of MX are very fussy about consistency in all lines of dayfile.txt. Earlier releases tended to have better compatibility with the legacy software. Subsequent releases expected every line to be expressed exactly as specified in the locale. MX in its latest release is trying to cope better with inconsistent date separators.
Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.
Real number fields: MX uses the locale to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the Region settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if list separation character was comma) was fairly tolerant when reading old data in files. At the time of writing this, it was proposed the next major release will force this file to use decimal points (full stops not commas) regardless of locale. This proposal may or may not happen, the point is that information here might be incorrect for the release that you are using.
Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours.
Number of fields: The number of fields in the file increases in various versions as shown in #List_of_fields.
- C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields.
- Each release of MX expects a particular number of fields in each line of file; MX has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an internal (RAM held) array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file.
- If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, there is a utility to run that will create a new dayfile.txt with all fields populated, please see Calculate Missing Values page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added, and it uses various software libraries contained within .dll files specific to a particular CumulusMX.exe release.
How to view or edit this file
The detailed information has been moved to viewing/editing dayfile.txt page.
An editor has been included within recent Cumulus releases of each flavour:
- Available from release 3.4.5 (13 Mar 2020): In the interface go to Data logs menu and select Dayfile
- Available from version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from Edit menu select Dayfile.txt
- Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
- Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
- There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file.
The MX editor does not let you change the date, you can only change other fields or delete an entire line, so you have no control over the order of the lines in the file.
Reading the file
See Daily Summary page for a full discussion of ways, external to Cumulus, to read this file.
Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in vaCumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error (line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged).rious other circumstances depending on the release you are running.
- The legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored:
- The legacy Cumulus has a number of screens for viewing data for various periods, to populate these screens Cumulus has to read all of the Files with Comma Separated Values as the derivatives shown generally do not match the extremes tracked in .Ini files.
- The Select a graph feature also uses all of the Files with Comma Separated Values.
- The end of day action uses a simple "append" instruction, so it does not need to read the file
- As already mentioned, MX reads the entire dayfile.txt file when it is first started into an internally held array. That internal copy was introduced in release 3.9.2 - b3097 (7 Dec 2020), it was needed to drive the historic charts functionality; both in the interface, and the example included Historic Charts web page.
- From release 3.20.0 - b3200 (21 Aug 2022) there is a This Period Highs and Lows web page (accessed from Records menu) which is similar to the screens for viewing data for various periods except that the extremes it shows match those tracked in .Ini files.
- From release 3.20.0 - b3200 (21 Aug 2022) there is an option to Reload dayfile in a Utils menu, this updates the internal held array by reading the file again; essential if an edit has been made to the file.
Line numbering and Error messages
Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error.
Cumulus MX stores the contents of the file in an internal two dimensional array. The outer dimension is identified by numbering (from zero) what were lines in the file. The inner dimension has that line number as element zero, and the fields taken from file are therefore numbered from one.
Obviously, line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged in the file. This is why you will see line numbers quoted in any error messages related to this file.
Some of the editors you may use to view/edit the file may number the lines (in some cases always, in others as an option). If you have seen an error message, then to correct the file you need to use a tool that does number the lines in order to find where the correction is needed. Error messages report just the first error found, so check subsequent lines in case they have same error.
List of fields in dayfile.txt
For your installed build please see dayfileheader.txt. This file identifies all the fields in each line of the dayfile.txt file in the release you are running.
That file has changed location from release 3.20.0 - b3200 (21 Aug 2022), it is now in data folder, for Cumulus 1 and all earlier MX releases it was in same directory as the Cumulus executable.
As explained earlier, the number of fields in a line has (with one exception when fields were added in error) increased as the Cumulus software has been developed. The table below therefore groups the fields under labels that indicate which version/release introduced them.
Field numbering
The table below was first created when the only flavour of Cumulus software available was what we now call the legacy Cumulus 1. Back then the fields were numbered starting from zero. This reflected the way fields were identified when processed as elements of an single-dimension array variable representing a line.
The old numbering from zero had two advantages:
- It stressed that the date field was different to the rest, all other fields were either values or time-stamps
- Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown was the number to quote in any scripts where a Cumulus 1 line was converted to an array, and you needed to address a single field.
Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering. As explained above the fields are now numbered starting from 1 to fit in with an internal array created when the log file is read, the processing code adds a line number in front of the date field on each array 'line' it holds.
Information shown in the table
- The date is supposed to be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
- The dates, as mentioned earlier, should be in ascending order.
- Please see amending dayfile page for more information on dates
- Both Cumulus 1 and MX may get confused if you restart the software close to rollover time, and in that confused state they may create a second line with same date as a date already in the file.
- If you regularly restart Cumulus close to rollover time, then a whole sequence of dates can be repeated either as a block or as pairs.
- The remaining fields are all either numerical values, or a time paired with (except for first wind field) preceding numerical value.
- Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed). Cumulus MX only validates that each field is of correct type, it does not recognise any relationships between fields.
- The alphabetic column identifiers used by many spreadsheet editors are shown in the table
- IMPORTANT: If you do use a spreadsheet editor, do ensure all columns are set to "text" format,
- Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, or let it change the number of decimal places.
- The type of field is shown in the table:
- Note you must not include a sign for an unsigned field,
- Note you can not specify a decimal comma/point in an integer field,
- Note all fields of type time must use 5 character "HH:mm" format, although Cumulus 1 will accept other separator characters where ":" shown.
- The field description is shown, together with references to where that terminology is explained
Table listing Fields
Field number | Spreadsheet column | Field type | Description | |
---|---|---|---|---|
0 | For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.
If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1. | |||
Those fields included below have been in dayfile.txt from the start of Cumulus 1 (Version 1.0, the First release on 27th January 2004). | ||||
1 | A | 8 characters | Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup) | |
2 | B | Unsigned number | Highest wind gust speed | |
3 | C | unsigned integer | Bearing of highest wind gust | |
4 | D | 5 characters | Time of highest wind gust | |
5 | E | signed decimal | Minimum temperature | |
6 | F | 5 characters | Time of minimum temperature | |
7 | G | signed decimal | Maximum temperature | |
Consistency Note: In some cases Minimum comes before Maximum, in other cases Maximum is before Mimum | ||||
8 | H | 5 characters | Time of maximum temperature | |
9 | I | Unsigned number | Minimum sea level pressure | |
10 | J | 5 characters | Time of minimum pressure | |
11 | K | Unsigned number | Maximum sea level pressure | |
12 | L | 5 characters | Time of maximum pressure | |
13 | M | unsigned number | Maximum rainfall rate | |
14 | N | 5 characters | Time of maximum rainfall rate | |
15 | O | unsigned number | Total rainfall for the day | |
Above here represents the minimum length for every line, a count of 15 items | ||||
[There is no record of which version added this next field. The Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the content of each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 as these do not appear in the release history issued by Steve Loft. The first mention of it in the new forum was not until December 2008, but that was not about when it was released. A web tag for this variable was added in Build 978 of 1.9.1 beta, which was obviously long after it was first calculated.
Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment] | ||||
16 | P | signed decimal | Average temperature for the day | |
(Wind run was added from version 1.8.4) | ||||
17 | Q | unsigned number | Daily wind run | |
(The next pair of entries were added from version 1.8.9 build 907 (June 2010) as part of a total redesign of how dayfile.txt was implemented in Cumulus 1) | ||||
18 | R | unsigned number | Highest Average Wind Speed | |
19 | S | 5 characters | Time of Highest Avg. Wind speed | |
(The two pairs of humidity entries were added in October 2010, a v 1.9.0 beta, the exact build number is now lost) | ||||
20 | T | unsigned integer | Lowest humidity | |
21 | U | 5 characters | Time of lowest humidity | |
22 | V | unsigned integer | Highest humidity | |
23 | W | 5 characters | Time of highest humidity | |
(The next two entries were added from version 1.9.0) | ||||
24 | X | (not documented) | Total evapotranspiration (Only valid for Davis stations, shows zero otherwise) | |
25 | Y | unsigned | Total hours of sunshine (only valid if sunshine sensor connected)
Important if rollover time is 9 am or 10 am: Most fields in this file are updated taking information from today.ini. For a non-midnight rollover, then the Sunshine hours reported here is from 00:01 on the calendar date corresponding to the date in the first field of this file, to subsequent midnight, and that end time is 9 or 10 hours before when this file is updated. Meanwhile, the sunshine hours count has been reset and so the figure in today.ini is not what is wanted here. For that reason the sunshine hours reported here are taken from yesterday.ini. | |
(The next 16 entries were added from version 1.9.1 May 2011) | ||||
26 | Z | signed decimal | High Heat index (added to Cumulus in 1.7.11 only as spot value, not stored) | |
27 | AA | 5 characters | Time of high heat index | |
28 | AB | Signed decimal | High Apparent temperature | |
29 | AC | 5 characters | Time of high apparent temperature | |
30 | AD | signed decimal | Low apparent temperature | |
31 | AE | 5 characters | Time of low apparent temperature | |
32 | AF | unsigned number | High hourly rain | |
33 | AG | 5 characters | Time of high hourly rain | |
34 | AH) | signed decimal | Greatest wind chill (high wind speed, low temperature) (calculated since version 1.8.3 as spot value, not stored) | |
35 | AI | 5 characters | Time of greatest wind chill | |
(The next two pairs for dew point were added in version 1.9.2 beta build) | ||||
36 | AJ | signed decimal | High dew point | |
37 | AK | 5 characters | Time of high dew point | |
38 | AL | signed decimal | Low dew point | |
39 | AM) | 5 characters | Time of low dew point | |
(The next three entries were added in version 1.9.2 Build 1004) | ||||
40 | AN | unsigned integer | Today's dominant/average wind direction | |
41 | AO | unsigned decimal | Heating degree days | |
42 | AP | unsigned decimal | Cooling degree days | |
The next two pairs were added in legacy version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist).
Fields listed up to those following here applied to the final legacy Cumulus 1.9.4 and formed the basis for early releases of Cumulus MX. | ||||
43 | AQ | unsigned decimal | High solar radiation | |
44 | AR | 5 characters | Time of high solar radiation | |
45 | AS | unsigned decimal | High UV Index | |
46 | AT | 5 characters | Time of high UV Index | |
The next two pairs were added in MX release 3.6.0, 2 more derived values and their times | ||||
47 | AU | signed decimal | High Feels Like temperature | |
48 | AV | 5 characters | Time of high feels like temperature | |
49 | AW | signed decimal | Low Feels Like temperature | |
50 | AX | 5 characters | Time of low feels like temperature | |
The next two pairs were added in release 3.6.12
| ||||
51 | AY | signed decimal | High Canadian Humidity Index or Humidex | |
52 | AZ | 5 characters | Time of high Humidex
Bug for releases 3.13.0 to 3.14.2 inclusive: The major code rewrite for release 3.13.0 replaced the previous code for the processing for all Cumulus files, the new code incorrectly stored Time of high feels like temperature in this field for all these releases! Corrected in minor code rewrite for 3.14.3 - b3163 25 Jan 2022 (not released to public until 3.15.0 - b3169 Released 31 Jan 2022) | |
Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in any other version, so have labelled them as error. | ||||
53 (error) | BA | signed decimal | Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other release) | |
54 (error) | BB | 5 characters | Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other release) | |
The next value was added in release 3.12.0 | ||||
53 (new) | BA | unsigned decimal | Cumulative Chill Hours since start of season | |
The next value and time fields were added in release 3.20.0 | ||||
54 | BB | unsigned decimal | Highest rainfall total in (an approximate, as it depends upon timing of earlier entries in Recent history) 24 hour period ending in Meteorological day | |
55 | BC | 5 characters | Time when (approximate) 24 hour period for rainfall total above ends |
Example of the file
An extract of a few lines of a dayfile.txt
01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5 02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7 03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9 04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9 05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3 06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3 07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7