Dayfile.txt: Difference between revisions

1,549 bytes added ,  9 April 2022
m
minor resequencing
m (Minor resequencing of page)
m (minor resequencing)
(2 intermediate revisions by 2 users not shown)
{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to both Cumulus flavours. [[Category:Files_with_Comma_Separated_Values]]
 
TheAs supportpart forumof maya haveredevelopment linksof toWiki, contentthis thatpage ishas nobeen longersimplified, onby thismoving Wikisome page,content soto ifnew thepages. link youOld followedlinks in the support forum, takesthat youwere to herecontent atno thelonger start ofon this page, ratherwill than to a particular section, the informationbring you seek may now be on another pagehere.
 
Content previously on this page has been moved as follows:
= This Wiki page =
* Explanations about Cumulus terminology can be accessed from [[:Category:Terminology]] page
* Each line in this file represents one day, but the start time need not be same for all fields,
** so please see [[Meteorological_day|Meteorological day]], and [[today.ini]] for more details about that
* Advice regarding editing this file has been moved to [[Amending dayfile]] page, this also covers date-separator issues and dealing with missing or corrupted lines
* General advice about Cumulus files with a '''.txt''' extension has been consolidated on [[:Category:Files_with_Comma_Separated_Values]] page
* Advice about correcting any rogue extreme figures inadvertently stored in this file has been consolidated in new [[Correcting Extremes]] page
* If some lines in your file have fewer fields than other lines the advice has been consolidated on [[Calculate Missing Values]] page
** For MX there is a [[Calculate_Missing_Values#CreateMissing.exe|Create Missing Utility]], which checks spot readings in the [[Standard_log_files|MMMyylog.txt]] files, adds any missing derivative spots (e.g. heat index) and uses these figures to recalculate daily derivatives and uses those to replace missing fields/lines in dayfile.txt
** For the legacy Cumulus, there was a [[Amending_dayfile#Create_Missing_on_legacy_dayfile_editor|dayfile editor with create missing option]], that performed equivalent function
** If you import historic data from before you started using Cumulus into [[Standard log files]],
** then see [[Calculate Missing Values]] page for how to generate new lines in dayfile.txt
* Cumulus MX is more fussy, than Cumulus 1, about various formattting issues, see below, also see [[Migrating_from_Cumulus_1_to_MX#dayfile.txt]] section for more advice
 
The content that remains on this page is summarised by the table of contents that follows.
This Wiki page describes one of the [[:Category:Cumulus Files|files]] not included in any release download. This daily summary file (its title is frequently shortened to "day" plus "file") uses a [[:Category:Files_with_Comma_Separated_Values|".txt" extension]] and is created in [[Data folder|data sub-folder]] of your Cumulus installation when Cumulus needs to store its first line in this file.
 
=About this file=
This page is full of links that take you to content previously on this page, the links in the previous paragraph take you to new pages that cover details once repeated on several pages including this page. The names of [[Amending dayfile]] page, and [[Calculate Missing Values]] page, should give you a clue about what has been moved from this page to those new pages.
 
* This Wiki page describes one of the [[:Category:Cumulus Files|files]] not included in any release download.
You might also find it useful to read [[Correcting Extremes]] page which also includes some information that once was on this page about fixing other files where the highest/lowest/total recorded for day has been corrupted by rogue values.
* This daily summary file (Steve Loft shortened that to "day" plus "file") uses a ".txt" extension, and is essentially a plain text file in CSV format.
* The file is created in [[Data folder|data sub-folder]] of your Cumulus installation when Cumulus needs to store its first line in this file.
* Cumulus MX reads the whole of this file when you restart the software,
** So if you move house, it is probably best to archive the old "dayfile.txt", and avoid any discontinuities in any graphs plotted from active file
 
 
([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Ini_Files|.ini files]]).
The content that remains on this page is summarised by the table of contents that follows.{{TOCright}}
 
=About this file=
 
If you import historic data from before you started using Cumulus into [[Standard log files]], then the daily summary for that imported data can be summarised into '''dayfile.txt''' line format (and a dayfile.txt file created/updated) using any of the approaches described on the [[Calculate_Missing_Values|calculate missing values]] page of this Wiki.
 
The single file '''dayfile.txt''' can contain lines created over a long period of time ([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Ini_Files|.ini files]]).
 
==How Cumulus Creates and Updates this file==
 
* Cumulus reads values supplied by your weather station, (either directly while Cumulus running, or for some weather station types can read historic data during catch-up on restarting Cumulus)
Cumulus reads values supplied by your weather station, converts them to the units you prefer, applies any calibration (multiplier and offset) you have set, and then sees if the resulting value implies [[:Category:Ini Files|any extreme records file]] needs to be updated (these hold derived values like totals, highest, and lowest). Two of those files are [[Today.ini#When_Cumulus_is_running|today.ini]] and [[yesterday.ini]] and values from those two files are extracted (see linked Wiki pages for how entries in those files map across to the daily summary file) at the end of a [[meteorological day]] to calculate the necessary [[#List_of_fields|fields]] for a new line to be added to this file.
* Cumulus converts them to the units you prefer,
* Cumulus applies any calibration (multiplier and offset) you have set,
* For a sub-set of those readings (perhaps those every minute if readings are collected every 10 seconds), the spot values of source items like temperature, humidity, wind speed, can be used in calculations of derived items like "wind chill", "dewpoint" and "feel-like" temperature.
* Cumulus then sees if the resulting soource, or derived, value implies [[:Category:Ini Files|any extreme records file]] needs to be updated
** Daily extremes are held in [[today.ini]] and that is main source used when a new line is added to "dayfile.txt"
 
As explained on the "meteorological day" page linked above, if your weather station stores weather data, then Cumulus on restarting can (optionally) read the '''historic data''' and catch-up by processing those past readings (as per description in start of previous paragraph), including activating any end of day actions (see [[Today.ini#Restart_and_Catch-up|today.ini]]), for the period since it was last running.
 
==HowWhy tothis viewfile orshould editbe thisbacked fileup==
 
* This file contains daily extremes, the accuracy of those daily extremes depends on the interval between the spot readings that are used as explained above.
An editor has been included within Cumulus:
* If the file is corrupted, and not regenerated from a back-up, the only source of spot readings is the [[Standard_log_files|MMMyylog.txt]] files, and their interval might be only every half an hour (it has to be an exact fraction of an hour).
* [[File:Badge vMx.png]] From release 3.4.5 (13 Mar 2020) onwards: In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
* As explained [[Monthly_log_files#Enhancement never implemented|here]] minima and maxima between such logging intervals are not recorded by Cumulus software
* Cumulus 2 had no viewing option
* Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,
* [[File:Badge v1.png]] From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
 
Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough for an uncorrupted file:
For detailed information, please see [[Amending dayfile|viewing/editing '''dayfile.txt''']].
* The file is backed up when MX is restarted into a date/time stamped sub-folder of the [[Backup folder|'''backup''' folder]]
*The file is also backed up (to a date/time stamped sub-folder within '''daily''' sub-folder of that backup sub-folder) during the end of day process, depending on the release you are running the back-up copy may, or may not, include the line that is appended in the same end of day
 
Retention of these back ups:
=Reading the file=
* [[File:Badge vMx.png]] MX only keeps the last 9 of the date/time stamped subfolders.
 
* [[File:Badge v1.png]] Legacy Cumulus 1 only keeps up to 8 of the date/time stamped subfolders.
See [[Daily Summary]] page for a full discussion of ways, external to Cumulus, to read this file.
 
Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in various other circumstances depending on the release you are running.
 
[[File:Badge v1.png]] The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], these use several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt". The '''Select a graph''' feature also uses several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt". Thus the legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored, and the end of day action uses a simple "append" instruction.
 
[[File:Badge vMx.png]] From release 3.9.2 - b3097 (7 Dec 2020), MX reads the entire '''dayfile.txt''' file, to drive the '''historic charts functionality'''; both in the [[MX Administrative Interface|admin interface]], and the example included [[New Default Web Site Information|Historic Charts web page]]. Consequently, any error in say the date field (or change of separating character) will stop historic charts working.
 
== Specific issues for MX ==
* MX too has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file. If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, you can add the missing derived fields, please see [[Calculate Missing Values]] page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added.
 
==How to view or edit this file==
=Why this file should be backed up=
 
An editor has been included within Cumulus:
As explained [[Monthly_log_files#Enhancement never implemented|here]] Cumulus only logs periodic spot values as it works through processing the values it processes from your weather station. Therefore the [[Standard log files]] may not include any maximum or minimum values supplied by your weather station. However, on a day-by-day basis these should be preserved in this daily summary file.
* [[File:Badge vMx.png]] From release 3.4.5 (13 Mar 2020) onwards: In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
* [[File:Badge v1.png]] From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
 
For detailed information, please see [[Amending dayfile|viewing/editing '''dayfile.txt''']].
Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,
 
==Reading the file==
Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough for an uncorrupted file:
 
* The file is backed up when MX is restarted into a date/time stamped sub-folder of the [[Backup folder|'''backup''' folder]]
See [[Daily Summary]] page for a full discussion of ways, external to Cumulus, to read this file.
*The file is also backed up (to a date/time stamped sub-folder within '''daily''' sub-folder of that backup sub-folder) during the end of day process, depending on the release you are running the back-up copy may, or may not, include the line that is appended in the same end of day
 
Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in various other circumstances depending on the release you are running.
* [[File:Badge v1.png]] The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], these use several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt". The '''Select a graph''' feature also uses several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt". Thus the legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored, and the end of day action uses a simple "append" instruction.
* [[File:Badge vMx.png]] From release 3.9.2 - b3097 (7 Dec 2020), MX reads the entire '''dayfile.txt''' file, to drive the [[Highcharts_-_Historic|'''historic charts functionality''']]; both in the [[MX Administrative Interface|interface]], and the example included [[New Default Web Site Information|Historic Charts web page]].
** Consequently, any error in say the date field (or change of separating character) will stop historic charts working.
 
Retention of these back ups:
* [[File:Badge vMx.png]] MX only keeps the last 9 of the date/time stamped subfolders.
* [[File:Badge v1.png]] Legacy Cumulus 1 only keeps up to 8 of the date/time stamped subfolders.
 
 
= List of fields in dayfile.txt =
 
'''For your installed build please see ''[[dayfileheader.txt]]'' (stored within the folder that contains your Cumulus executable), as that will list which fields your Cumulus installation uses.'''
 
If you have been using Cumulus for a while, you may wonder which of your log file lines might be shorter, so the table below shows fields grouped by the Cumulus version when those fields were added.
 
==Field numbering==
 
Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering.
== Information shown in the table ==
* The original table below was for Cumulus 1 and then field number '''was''' starting from zero.
 
** Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number where it first found an error (this uniquely identifies the line even if a date is duplicated or a line feed has been deleted so two lines are merged).
** Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file
*The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, the processing code adds a line number in front of the date field on each line it holds.
** The Cumulus MX user may not be aware of this happening as it is within the internal workings, where data from the file is transferred to an array, or data from the array is written back into the file.
**By using line numbers, MX is able to identify which line has been deleted or edited, MX coding treats the date as a fixed bit of text (MX does not allow you to change a date)
 
** Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file
The old numbering from zero had two advantages:
**Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number where it first found an error (this uniquely identifies the line even if a date is duplicated or a line feed has been deleted so two lines are merged).
# It stressed that the date field was different to the rest, all other fields were either values or time-stamps
# Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a Cumulus 1 line was converted to an array, and you needed to address a single field.
 
== Information shown in the table ==
 
* The date '''must''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
* The remaining fields were all either numerical values, or a time paired with (except for first wind field) preceding numerical value.
** Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed).
*The alphabetic column identifiers used by many spreadsheets are shown
** IMPORTANT: Ensure '''all columns are set to "text" format''',
** Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, change the number of decimal places.
* The type of field is shown in the table,
** you must not include a sign for an unsigned field,
** you can not specify a decimal point in an integer field,
** all time fields must use 5 character "HH:mm" format
* The field description is shown, together with references to where that terminology is explained
 
 
== Table listing Fields ==
*The original table below was for Cumulus 1 and then field number '''was''' starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
*#Cumulus 1 stressed that the date field was different to the rest, as it was used as identifier. The date '''must''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
*#The remaining fields were all either numerical values, or a time paired with (except for first wind field) preceding numerical value. Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed).
**Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
*The alphabetic column identifiers used by many spreadsheets are shown, please remember to select options in your spreadsheet to ensure '''all columns are set to "text" format''', you will corrupt this file if you let your spreadsheet:
** recognise content as dates or time,
** or change the number of decimal places.
*The type of field is shown in the table, you must not include a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use 5 character "HH:mm" format
*The field description is shown, together with references to where that terminology is explained
 
=== List of Fields ===
{| class="wikitable" border="1"
|-
|[[Heat/cold degree days and Chill hours | Cooling degree days]]
|-
|colspan="4" style="background:lightgray;"|The next two pairs were added in legacy version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist).
 
Fields listed up to those following here applied to the final legacy Cumulus 1.9.4 and formed the basis for early releases of Cumulus MX.
|-
|43
|Time of high UV Index
|-
|colspan="4" style="background:lightgray;"|The next two pairs were added in versionMX release 3.6.0, 2 more derived values and their times
|-
|47
|Time of low feels like temperature
|-
|colspan="4" style="background:lightgray;"|The next two pairs were added in versionrelease 3.6.12
 
*Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times).
**The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor.
*From versionrelease 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error.
|-
|51
|AZ
|5 characters
| style="background:pink;"|Time of high Humidex
 
'''Bug for releases 3.13.0 to 3.14.2 inclusive''': The major code rewrite for release 3.13.0 replaced the previous code for the processing for all Cumulus files, the new code incorrectly stored ''Time of high feels like temperature'' in this field for all these releases! Corrected in minor code rewrite for 3.14.3 - b3163 25 Jan 2022 (not released to public until 3.15.0 - b3169 Released 31 Jan 2022)
|-
|colspan="4" style="background:lightblue;"|Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in current nor any other version, so have labelled them as error.
|-
| 53 ('''error''')
| BA
|signed decimal
|Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other versionrelease)
|-
| 54 ('''error''')
| BB
| 5 characters
| Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other versionrelease)
|-
|colspan="4" style="background:lightgray;"|The next value was added in release 3.12.0
|-
| 53 (new)
5,838

edits