Cumulus Wiki

Dayfile.txt: Difference between revisions
m (Correcting Typos (mostly missing, but critical, colon characters in links), and mention of some more release 3.20.0 changes)
 
(47 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to both Cumulus flavours. [[Category:Files_with_Comma_Separated_Values]]
<big>'''This article is about the Daily Summary logging file'''</big>


As part of a redevelopment of Wiki, this page has been simplified, by moving some content to new pages. Old links in the support forum, that were to content no longer on this page, will bring you here.
= Introduction =


'''Content previously on this page has been moved as follows''':
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. In all flavours of Cumulus, this file is only updated (with exclusive lock) during the end of meteorological day process. In that process it is also read if the generation of NOAA reports has been requested.
* Explanations about Cumulus terminology can be accessed from [[:Category:Terminology]] page
* Each line in this file represents one day, but the start time need not be same for all fields,
** so please see [[Meteorological_day|Meteorological day]], and [[today.ini]] for more details about that
* Advice regarding editing this file has been moved to [[Amending dayfile]] page, this also covers date-separator issues and dealing with missing or corrupted lines
* General advice about Cumulus files with a '''.txt''' extension has been consolidated on [[:Category:Files_with_Comma_Separated_Values]] page
* Advice about correcting any rogue extreme figures inadvertently stored in this file has been consolidated in new [[Correcting Extremes]] page
* If some lines in your file have fewer fields than other lines the advice has been consolidated on [[Calculate Missing Values]] page
** For MX there is a [[Calculate_Missing_Values#CreateMissing.exe|Create Missing Utility]], which checks spot readings in the [[Standard_log_files|MMMyylog.txt]] files, adds any missing derivative spots (e.g. heat index) and uses these figures to recalculate daily derivatives and uses those to replace missing fields/lines in dayfile.txt
** For the legacy Cumulus, there was a [[Amending_dayfile#Create_Missing_on_legacy_dayfile_editor|dayfile editor with create missing option]], that performed equivalent function
** If you import historic data from before you started using Cumulus into [[Standard log files]],
** then see [[Calculate Missing Values]] page for how to generate new lines in dayfile.txt
* Cumulus MX is more fussy, than Cumulus 1, about various formatting issues, see below, also see [[Migrating_from_Cumulus_1_to_MX#dayfile.txt]] section for more advice


The content that remains on this page is summarised by the table of contents that follows.
In Cumulus 1 only, the figures contained in the file are used for the 'This period' display accessed from the '''View''' menu and to build any graphs based on daily values.


=About this file=
The format of this file is the same for both Cumulus 1 and Cumulus MX, although the number of fields in the file increases in various versions as shown at the end of this article. The file can be ported between flavours only if both are run with exactly the same locale settings, as using a different locale may change the field separator or the symbol used for decimal points.
{{TOCright}}


* This Wiki page describes one of the [[:Category:Cumulus Files|files]] not included in any release download.
* This daily summary file (Steve Loft shortened that to "day" plus "file") uses a ".txt" extension, and is essentially a plain text file in CSV format.
** CSV stands for "Comma Separated Values", but in the Cumulus context the field separator can be other characters including ";" or "|"
* The file is created in [[data folder|data sub-folder]] of your Cumulus installation when Cumulus needs to store its first line in this file.
* Cumulus MX reads the whole of this file into an internal array (stored in RAM) each time you restart the software,
** Please see [[Amending_dayfile#Warning_if_editing_outside_MX|here for implications]]
** Therefore MX users who move to a different location should archive the old "dayfile.txt", to avoid any discontinuities in any graphs plotted from active file


== Changes in different releases of Cumulus ==


([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Ini_Files|.ini files]]).
{{Version badge 1}}


'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.


==How Cumulus Creates and Updates this file==
'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order)


* Cumulus reads values supplied by your weather station, (either directly while Cumulus running, or for some weather station types can read historic data during catch-up on restarting Cumulus)
There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard final release of Cumulus 1 as this section was updated.
* Cumulus converts them to the units you prefer,
* Cumulus applies any calibration (multiplier and offset) you have set,
* For a sub-set of those readings (perhaps those every minute if readings are collected every 10 seconds), the spot values of source items like temperature, humidity, wind speed, can be used in calculations of derived items like "wind chill", "dewpoint" and "feel-like" temperature.
* Cumulus then sees if the resulting soource, or derived, value implies [[:Category:Ini Files|any extreme records file]] needs to be updated
** Daily extremes are held in [[today.ini]] and that is main source used when a new line is added to "dayfile.txt"


[[File:Badge vMx.png]]


==Why this file should be backed up==
'''Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor'''


* This file contains daily extremes, the accuracy of those daily extremes depends on the interval between the spot readings that are used as explained above.
'''Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor'''
* If the file is corrupted, and not regenerated from a back-up, the only source of spot readings is the [[Standard_log_files|MMMyylog.txt]] files, and their interval might be only every half an hour (it has to be an exact fraction of an hour).
* As explained [[Monthly_log_files#Enhancement never implemented|here]] minima and maxima between such logging intervals are not recorded by Cumulus software
* Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,


Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough to find a file that has not been corrupted:
= When Cumulus is left running =
* The file is backed up when MX is restarted into a date/time stamped sub-folder of the [[Backup folder|'''backup''' folder]]
*The file is also backed up (to a date/time stamped sub-folder within '''daily''' sub-folder of that backup sub-folder) during the end of day process, but (depending on the release you are running) the back-up copy may, or may not, include a line for the previous day (the line that is appended in the same end of day)


Retention of these back ups:
* Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day.
* [[File:Badge vMx.png]] MX only keeps the last 9 of the date/time stamped sub-folders.
* There are no updates to dayfile.txt at any other times, but (for Cumulus 1 only) the contents of the file are read and processed for many of the display and edit menu options that can be selected from the main Cumulus 1 screen.
* [[File:Badge v1.png]] Legacy Cumulus 1 only keeps up to 8 of the date/time stamped sub-folders.
* Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in [[Today.ini]], updating that file as required.
**What is stored in today.ini is processed into what get stored as next line in dayfile.txt as described in next section.
*(It also updates [[Alltime.ini]], [[Monthlyalltime.ini]], [[Year.ini]], and [[Month.ini]] when appropriate.
*(Periodically, the weather readings and derived values are stored in [[Standard log files]] which are set up for each month).
* Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day.
* If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original.


== Specific issues for MX ==
== When Cumulus processes the end of the (meteorological) day ==
*A new row is appended to dayfile.txt, the values are prepared from reading "today.ini" file, not all values available in "today.ini" are stored in dayfile.txt.
*Some of this information is also stored in [[yesterday.ini]].
*Back ups of both today.ini, and dayfile.txt, log files ''in their state after the end of day update'' are copied to the 'cumulus\backup\daily' folder, a maximum of only 9 daily sub-folders are retained.


WARNING: It is important to note that some releases of MX are very fussy about consistency in all lines of '''dayfile.txt'''. Earlier releases tended to have better compatibility with the legacy software. Subsequent releases expected every line to be expressed exactly as specified in the locale. MX in its latest release is trying to cope better with inconsistent date separators.
'''Optional'''
* Some people require a copy of the local file to use on their web server. Consequently, after it has been updated they file transfer it to (or if their web server is local, copy it to) their web server. One way of doing this is [[Upload_Dayfile| described here]].
*Some people take a copy of the local file, and use it locally for other purposes. See [[#How you can use the daily log|How you can use the daily log section]] and also the [[[[User Contributions|Cumulusutils]]]] link.
*For some people it is easier to follow an option of converting the file into a database table, and that table having a new row added each time the file gains a new line as described below.
=== Populating a database table ===


Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.
* The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have bee edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using.
*There is a section at [[Cumulus_MX#Optional_Sections|Cumulus MX]] to explain how MX includes the ability to generate SQL for creating the database table, for updating it with past data, and to add a new row at the end of the day for the standard database table version of this daily summary log.


Real number fields: MX uses the '''locale''' to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the '''Region''' settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if ''list separation character'' was comma) was fairly tolerant when reading old data in files. At the time of writing this, it was proposed the next major release will force this file to use decimal points (full stops not commas) regardless of locale. This proposal may or may not happen, the point is that information here might be incorrect for the release that you are using.
==== Using that table ====
In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log.


Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours.
For examples of some of the third party tools (Cumulus [[Category:User_Contributions|user contributions]]) using the database daily summary table see [[Daily Summary#Some_example_Scripts|here]]. Of course there are also a lot of tools written to use a copy of the dayfile.txt log file, and some of these could be adapted to use the database table instead, if you are a programmer.


Number of fields: The number of fields in the file increases in various versions as shown in [[#List_of_fields]].
In my case I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table, i.e. I have one database table column for each of the weather derivatives I show on my web page that show this month's values; it is many more derivatives than are shown on the standard web page, but some are initially hidden. Consequentially, when my daily update script detects from the date that it is processing the last day of a month, it then starts another script that reads all the rows in the daily summary table for that month, and stores the highest/lowest/total (as relevant) in my monthly_summary table (nothing to do with the "monthly" table that MX can generate from its standard log file). This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Just another example of how much you can get from just one log file!
* C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields.
* Each release of MX expects a particular number of fields in each line of file; MX has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an internal (RAM held) array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file.
** If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, there is a utility to run that will create a new dayfile.txt with all fields populated, please see [[Calculate Missing Values]] page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added, and it uses various software libraries contained within .dll files specific to a particular CumulusMX.exe release.


==How to view or edit this file==
==== Alternative schemas ====


The detailed information has been moved to [[Amending dayfile|viewing/editing '''dayfile.txt''' page]].
Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files.


An editor has been included within recent Cumulus releases of each flavour:
{{Version badge 1}}
* [[File:Badge vMx.png]] Available from release 3.4.5 (13 Mar 2020): In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
With Cumulus 1, you would need to be a programmer and write your own script to update the database table with your own schema. You might use the importCumulusFile article to start you off.
* [[File:Badge v1.png]] Available from version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.


Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file.
You might also, as I did, want your script to validate what it reads from the daily summary log to ensure only valid numbers and times are stored in your database table, while any invalid inputs are stored as nulls by your script. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the contents of the Weather Diary, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). When I used Cumulus 1 I wrote a PHP script to find all these additional values, for example it reads the [[today.ini]] and [[month.ini]] log files as stored in the end of day backup (not the ones being updated for new day in data folder), and the [[log.xml|weather diary in log.xml in data folder]].


The MX editor does not let you change the date, you can only change other fields or delete an entire line, so you have no control over the order of the lines in the file.
[[File:Badge vMx.png]]
MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned. I continue to use my Cumulus 1 script (with some changes as for example the weather diary works differently, I am querying SQLite from \CumulusMX\data\diary.db) now I use MX. In the MX standard functionality, you are limited to using web tags for your inputs, and some of those are affected by the end of day process. I tried various content in the custom EOD query, but it did not give me what I need for the scripts that produce my web pages.


MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do.


==Reading the file==
== When Cumulus is restarted after a break in running ==


See [[Daily Summary]] page for a full discussion of ways, external to Cumulus, to read this file.
* It reads the daily log and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|this Cumulus 1 FAQ]])
* Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
* For Cumulus 1, back ups of 8 selected log files including dayfile.txt that are copied to start-up folders in the 'cumulus\backup' folder, the last 8 start-up folders only are retained.
* For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else.


Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in vaCumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error (line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged).rious other circumstances depending on the release you are running.
== How you can use the daily summary log file ==


* [[File:Badge v1.png]] The legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored:
* If you want to run scripts that use the daily summary log file, it is best if you take a copy first, you can ask Cumulus 1 to take a copy after each update by using the '''Daily''' box in the bottom left of the ''Sites/Options'' frame within the ''Internet'' options screen from the '''Configuration''' menu; that will safely take a copy of 'dayfile.txt' after it is updated. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up and so not at usual rollover time according to the computer clock. See Cumulus 1 '''Help''' for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from running the command file I specify in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure.
** The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], to populate these screens Cumulus has to read all of the [[:Category:Files with Comma Separated Values|Files with Comma Separated Values]] as the derivatives shown generally do not match the extremes tracked in [[:Category:Ini Files|.Ini files]].
* Cumulus MX has option to list files to be transferred once a day as part of rollover, so you can use that to generate your extra copy. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up.
** The '''Select a graph''' feature also uses all of the [[:Category:Files with Comma Separated Values|Files with Comma Separated Values]].
* A third party tool "Cumulus Toolbox" can also be used to copy/transfer files at a particular time. Note this cannot tell whether Cumulus has done its rollover at the normal time, or during catch-up.
** The end of day action uses a simple "append" instruction, so it does not need to read the file
* There are other ways to specify that when a file changes it is copied somewhere.
* [[File:Badge vMx.png]] As already mentioned, MX reads the entire '''dayfile.txt''' file when it is first started into an internally held array. That internal copy was introduced in release 3.9.2 - b3097 (7 Dec 2020), it was needed to drive the [[Highcharts_-_Historic|'''historic charts functionality''']]; both in the [[MX Administrative Interface|interface]], and the example included [[New Default Web Site Information|Historic Charts web page]].
*The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file when Cumulus restarts, when Cumulus processes end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, either your external process or the Cumulus process may fail.
** From release 3.20.0 - b3200 (21 Aug 2022) there is a '''This Period Highs and Lows''' web page (accessed from ''Records'' menu) which is similar to the ''screens for viewing data for various periods'' except that the extremes it shows match those tracked in [[:Category:Ini Files|.Ini files]].
*Normally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines.
** From release 3.20.0 - b3200 (21 Aug 2022) there is an option to '''Reload dayfile''' in a ''Utils'' menu, this updates the internal held array by reading the file again; essential if an edit has been made to the file.
*Some third-party tools have to be told what separator you use for dates, before they can read your dayfile.txt.


==Line numbering and Error messages==
= Safely editing the daily summary log =


Both Cumulus 1 and 3 (2 did not) now provide editors where you can see what is in your dayfile.txt log, and if you click on a particular line you can edit the fields in that line, or indeed delete that line.
Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error.


Cumulus MX stores the contents of the file in an internal two dimensional array. The outer dimension is identified by numbering (from zero) what were lines in the file. The inner dimension has that line number as element zero, and the fields taken from file are therefore numbered from one.
If you use these provided editors, then you don't need to worry about the editing rules in the next section. The in-built editors always insert a full line including all fields available in the version of Cumulus you are currently using, so you cannot accidentally insert blank lines. The Cumulus 1 editor also does some validation to ensure the right entry format is used for each field. There is no validation in the MX editor (it was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions), but all lines are passed back via an application programme interface to the MX engine which might raise an error when it can't replace the line in the logging file.


Obviously, line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged in the file. This is why you will see line numbers quoted in any error messages related to this file.
== Important Rules when editing dayfile.txt ==


Some of the editors you may use to view/edit the file may number the lines (in some cases always, in others as an option). If you have seen an error message, then to correct the file you need to use a tool that does number the lines in order to find where the correction is needed. Error messages report just the first error found, so check subsequent lines in case they have same error.
If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it.


= List of fields in dayfile.txt =
=== General Editing Rules ===


For your installed build please see ''[[dayfileheader.txt]]''. This file identifies all the fields in each line of the '''dayfile.txt''' file in the release you are running.
# Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
# Take another copy and use that for your editing, don't edit the actual file being used by the software.
#*This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
#*It also means that you can go back to the last working copy, you can't upset your "revert" copy.
#The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
#* It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used.
#** These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
#*You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
#**If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
#*Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
#Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
#All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
# Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing.
# Don't add a header line to the file, Cumulus expects all lines to be data lines.


That file has changed location from release 3.20.0 - b3200 (21 Aug 2022), it is now in [[data folder]], for Cumulus 1 and all earlier MX releases it was in same directory as the Cumulus executable.
=== File specific Editing Rules ===


As explained earlier, the number of fields in a line has (with one exception when fields were added in error) increased as the Cumulus software has been developed. The table below therefore groups the fields under labels that indicate which version/release introduced them.
[[File:Open office (editing cumulus log files).png| right]]
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
# All rows must ''start with date'' and include at least 14 further fields ''in correct sequence''.
# The (meteorological) date format uses ''two digits for the year''.
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types).
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or on Microsoft Windows '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.
#Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile.
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
#* If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
# Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
#* If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
# Rows can vary in length but only by missing off ''fields at the end''. The minimum number of fields after the date is 14, the maximum varies between different versions.
#Each field has a pre-defined format, and the same format must always be used in that field position.
#No fields will accept letters.
#*Some fields (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
#* Most value fields are in ''real number format'' using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
#Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
#when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a ''lowest or highest value'' and that must be paired with a time-stamp in the next field.
#*Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
#Time stamp fields must always be in ''format HH:mm'' i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
#**Be aware you will have problems if you, or your editing software, add seconds.
#*If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
#* Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
# Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
#* Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps.
#*If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).
#*Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
#Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
#* Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
# The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. Similar rules apply for single character terminators used by other operating systems, don't let windows change a single character terminator into its double character terminator.
#*Problems with terminating characters are intercepted by operating system, before it reaches the software, but may still stop the software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.


==Field numbering==


The table below was first created when the only flavour of Cumulus software available was what we now call the legacy Cumulus 1. Back then the fields '''were''' numbered starting from zero. This reflected the way fields were identified when processed as elements of an single-dimension array variable representing a line.


The old numbering from zero had two advantages:
=== Dealing with errors identified by the software ===
# It stressed that the date field was different to the rest, all other fields were either values or time-stamps
# Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a Cumulus 1 line was converted to an array, and you needed to address a single field.


Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering. As explained [[#Line numbering and Error messages|above]] the fields are now numbered starting from 1 to fit in with an internal array created when the log file is read, the processing code adds a line number in front of the date field on each array 'line' it holds.
If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus 1 will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.


== Information shown in the table ==
If you do have a 'duplicate' error, you need to decide which row to ''delete'', and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.


* The date '''is supposed to''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
=== Editing externally ===
** The dates, as mentioned earlier, should be in ascending order.
** Please see [[amending dayfile]] page for more information on dates
** Both Cumulus 1 and MX may get confused if you restart the software close to rollover time, and in that confused state they may create a second line with same date as a date already in the file.
** If you regularly restart Cumulus close to rollover time, then a whole sequence of dates can be repeated either as a block or as pairs.
* The remaining fields are all either numerical values, or a time paired with (except for first wind field) preceding numerical value.
** Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed). Cumulus MX only validates that each field is of correct type, it does not recognise any relationships between fields.
*The alphabetic column identifiers used by many spreadsheet editors are shown in the table
** IMPORTANT: If you do use a spreadsheet editor, do ensure '''all columns are set to "text" format''',
** Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, or let it change the number of decimal places.
* The type of field is shown in the table:
** Note you must not include a sign for an unsigned field,
** Note you can not specify a decimal comma/point in an integer field,
** Note all fields of type time must use 5 character "HH:mm" format, although Cumulus 1 will accept other separator characters where ":" shown.
* The field description is shown, together with references to where that terminology is explained


See earlier section on this page for the rules and later section for advice on dealing with rogue values.

=== Using the Cumulus 1 editing feature ===

{{Version badge 1}}
'''This section applies to Cumulus 1.x.y only'''. The last command in '''Edit''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu |menu]] is ''dayfile.txt''. ''This is how you view'' the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. Click the ''Help'' button for detailed instructions. Cumulus Help is comprehensive.

You can use this Cumulus 1 editor as follows:
*use '''insert''' key to add one or more missing rows (complete days) manually,
*correct individual values by over-typing,
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message),
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Standard_log_files]] and automatically calculating the best approximations for each field for those missing days.

==== Create Missing ====

The Cumulus 1 editor provides a "Create Missing" option where it will, for any dates for which a line does not exist, create a line if it can from reading the [[Monthly log files|detailed log file]] to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation for each dayfile.txt field, storing the time from the relevant other log file in any time-stamp field in dayfile.txt. If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged. For ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]]. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | work around]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Standard_log_files]] if the whole day (a line starting with that date) is missing in ''dayfile.txt''. Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps.

==== Using Standard logs to deal with shorter (or incomplete) dayfile.txt records for particular dates ====

'''WORKAROUND if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''

Example: '''add extra fields to records created by an earlier version of Cumulus''' [to help you, the versions (not builds) at which fields were added are indicated below].
Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files.

''One method '''for Cumulus 1 only''' is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use ''create missing'' in the [[dayfile.txt]] selection in the '''Edit''' menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).''

'''Cumulus MX'''
Although there is a tentative plan for an executable to be added to the MX package that can read the standard logs and calculate all the fields needed for a line in dayfile.txt, this is only a gleam in the developer's eye!

The only method available at the time of writing involves multiple steps:
#*Either use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table
#*Or if you already have the table and have MX updating the standard logs into that database table
#Then, you can fairly easily update to add any missing values into existing columns in this monthly table, and from it update columns in a daily summary table. See [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] where I describe two PHP scripts designed for this type of task.

=== Importing data not recorded by Cumulus ===

As an alternative to manual line ''insert'', in the Cumulus 1 editor, you may wish to use a procedure for importing, and processing, pre-Cumulus observations into [[Standard_log_files]]. Once there is data for required days in monthly logs, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt.

== Cumulus 2 ==

Does not provide any viewing functionality.

== Cumulus MX ==

There is a dayfile editor within the admin interface to edit this log file.

Only from MX version 3.6.0 has this been able to read the log file if it has some lines that were created using Cumulus 1 (with less than 46 fields). In the same version of MX, the number of fields in this log file was increased by 4 when Feels Like temperature was added. From that version all lines viewed in this editor will have 50 fields. The content of any field that was not in the line when it was created will be an empty string as far as this editor is concerned and any line edited and saved, therefore gains all these empty fields and will be stored as 50 fields until version 3.6.10.

From Emergency Version 3.6.12 (formally released in 3.7.0), all lines have 54 fields. The extras are Canadian Humidity Index (Humidex).

[[File:Badge vMx.png]]For Cumulus MX, when you select a line, both '''Edit''' and '''Delete''' buttons are enabled. There is no way of inserting new lines into dayfile.txt from within MX, nor of changing the dates in the file.



=== Editing inside MX ===

Pick Edit, click that and an editing dialog pops up, that does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), and the right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using).

=== Editing externally ===

See earlier section on this page for the rules and later section for advice on dealing with rogue values. Remember if you edit this file externally, either do not add any missing fields, or if you do then these must remain empty for every extra field in every line that was originally shorter!

MX does not have the capability to recalculate, the values and time-stamp for any entry in dayfile.txt, by reading the [[Monthly log files|detailed log file]] to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation. It can be done in a script if you are a programmer. I have done this, but it was not as easy as I thought, the log file you read from may have duplicate times potentially making the calculation harder, and you cannot edit the file as you go along, you basically rename the old file and create a new file with original name as you work through all days changing individual fields whenever you need to.

== Editing the file or other Manipulation outside Cumulus ==

{{Version badge 1}}Apart from bulk changes, Cumulus 1 made editing this log file quite easy. Although the create missing did not work where a line was present for the date, but an individual field needed to be corrected or populated, there was a [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates |work around]] for this.

[[File:Badge vMx.png]]The beta versions of MX did not include any feature to view or edit dayfile.txt, so any checking or editing had to take place outside the software. As mentioned above, MX does not provide any way to read, or calculate, values from any other file, nor will it do bulk edits. For any of these, you will need to take a copy of the Cumulus file and manipulate using a script or a spreadsheet processor. Do be careful not to change the encoding, the position of line feeds, the format of any field, the locale, or anything else that might stop MX being able to read the file.


The file is <tt>data\dayfile.txt</tt> within the directory holding the Cumulus executable, it can be viewed in a text editor, imported into various database systems, or imported into spreadsheets, to manipulate as you wish. Just remember that Cumulus reads it when it is restarted, and updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating (and possibly doing a rollover of logs), or around the midnight/9am/10am local rollover time when Cumulus is writing a new row.

'''Tip''': Take a copy of the file before you do any edit outside Cumulus, so you can revert to old file.

'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end). (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page).

=== Dealing with rogue measurements ===
Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes.

If you discover a rogue measurement (perhaps the wind affected your tipping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see [[today.ini]] or a [[FAQ]] for further advice. In general, you need to stop Cumulus, edit the monthly log row containing the dodgy values, edit 'today.ini' and possibly other '.ini' files, looking up the logs indicated below that show the updates with previous high or low.

If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time. As your first step you should update the appropriate field in the row for the affected date in dayfile.txt. Once ''dayfile.txt is correct'' the Cumulus editors will allow you to:
* update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. See [[Alltimelog.txt]] for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it.
* update the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.
* update the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago.
* update the highs and lows month-by-month in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows.
* '' '''Note''' in each of above 4 editing screens you can:
*# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp.
*# load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and
*# optionally ''Copy'' (row by row) the logged values (and associated date/time information) into the relevant .ini file.
*# click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''.
*# store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel'').
(Each of these screens is a text editor, and works best when at full screen).''
* create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report.
Ideally, you will subsequently try to edit the rogue data for the particular time it was logged; see [[Standard_log_files#Correcting_any_logged_data_problems]], but correcting the daily summary in dayfile.txt must always be the priority.

== Using the daily summary log on your web-site ==

If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page. You could have a web page that shows a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file.

Search the Cumulus support forum to see (for example) how others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago.

If you use a script to read what is in the daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see [[Daily_Summary]] article for information about ALL of the ways to show values from this database table.

==Viewing summary figures for a month or period==

'''These notes apply to Cumulus 1 only'''

To view a summary of dayfile.txt for a month, calendar year or selected period, use ''This month'' (choose any month, default is month from your computer system date), ''This year'' (choose any year, default is year from your computer system date), or ''This period'' (choose any start and end dates, default is yesterday calculated from your computer system date), within the '''View''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|menu]].
*Remember the daily summary log has its records based on rollover to rollover days.
*In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover.
*If you use 9am or 10am rollover, and choose '''View''' ''This period'' between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank.
*If you use 9am or 10am rollover, and choose '''View''' ''This month'' before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank.

Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying).
*On the screen displayed after selecting ''This month'', you can change the month and year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated.
*On the screen displayed after selecting ''This year'', you can change the year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated.
*On the screen displayed after selecting ''This period'', you can change the start date and end date then click ''Update Display'' to get the equivalent calculations displayed for part of a month or any other period.


Note differences between observation reports on View screens and those available as web tags.
*Date and time stamps:
**The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt;
**A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight).
*Reported statistics example:
**The screen shows total number of dry or wet days in the month;
**The web tags report longest dry or wet period in the month.

= List of fields in dayfile.txt =

== Variation by Cumulus version ==

The dayfile.txt has grown as Cumulus's functionality has been extended, the table below shows fields grouped by the Cumulus version when those fields were added.

If you have been using Cumulus for a while, then some lines will be shorter than others, the number of fields per line growing when a new release adds new fields, Cumulus does not in normal operation modify earlier lines, but both Cumulus 1 and MX provide editors where it is possible to modify any line.

There is information earlier in this article about how you might be able to recalculate values to put in for fields that did not exist when any particular line was created.

'''For your installed build please see ''dayfileheader.txt'' (stored within the folder that contains your Cumulus executable), as that will list which fields are available for you.'''

== Information shown in the table ==


== Table listing Fields ==
*The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, adds a line number in front of the date field.
** The Cumulus MX user may not be aware of this happening as it is within the internal workings.
*The original table below was for Cumulus 1 and then field number '''was''' starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
*#It stressed that the date field was different to the rest. The date '''must''' be a unique identifier, the same date should not be repeated in another line.
*#The remaining fields were all either numerical values or a time paired with preceding numerical value.
**Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
*The alphabetic column identifiers used by many spreadsheets are shown, please see warnings about using spreadsheets for editing earlier on this page
*The type of field is shown, you must not put a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format
*The field description is shown, together with references to where that terminology is explained


=== List of Fields ===
{| class="wikitable" border="1"
{| class="wikitable" border="1"
|-
|-
Line 327: Line 160:
|-
|-
|0
|0
|colspan="3"|For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.
|Colspan="4" style="background: pink;"|For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.


If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1.
If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1.
|-
|-
|colspan="4" style="background:lightgray;"|Those fields included below have been in dayfile.txt from when it was first invented (assumed in first public release 1.0)
|colspan="4" style="background:lightgray;"|Those fields included below have been in dayfile.txt from the start of Cumulus 1 (Version 1.0, the First release on 27th January 2004).
|-
|-
|1
|1
Line 367: Line 200:
|signed decimal
|signed decimal
|Maximum temperature
|Maximum temperature
|-
|Colspan="4" style="background: pink;"| Consistency Note: In some cases Minimum comes before Maximum, in other cases Maximum is before Mimum
|-
|-
|8
|8
Line 410: Line 245:
|colspan="4" style="background:lightblue;"|Above here represents the minimum length for every line, a count of 15 items
|colspan="4" style="background:lightblue;"|Above here represents the minimum length for every line, a count of 15 items
|-
|-
|colspan="4" style="background:pink;"|[There is no record of which version added this next field. The Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the content of each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 as these do not appear in the release history issued by Steve Loft. The first mention of it in the new forum was not until December 2008, but that was not about when it was released. A web tag for this variable was added in Build 978 of 1.9.1 beta, which was obviously long after it was first calculated.
|colspan="4" style="background:gray;"|(The release history between versions 1.2.5 and 1.5.1 was lost by Steve Loft, so there is no record of which version added this next field. It was definitely present in December 2008 (1.8.2), but was probably added a lot earlier.


Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment)
Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment]
|-
|-
|16
|16
Line 438: Line 273:
|Time of Highest Avg. Wind speed
|Time of Highest Avg. Wind speed
|-
|-
|colspan="4" style="background:lightgray;"|(The two pairs of humidity entries were added in a v 1.9.0 beta, the exact build number is now lost)
|colspan="4" style="background:lightgray;"|(The two pairs of humidity entries were added in October 2010, a v 1.9.0 beta, the exact build number is now lost)
|-
|-
|20
|20
Line 471: Line 306:
|unsigned
|unsigned
|Total hours of sunshine (only valid if sunshine sensor connected)
|Total hours of sunshine (only valid if sunshine sensor connected)

'''Important if rollover time is 9 am or 10 am:''' Most fields in this file are updated taking information from [[today.ini]]. For a non-midnight rollover, then the Sunshine hours reported here is from 00:01 on the calendar date corresponding to the date in the first field of this file, to subsequent midnight, and that end time is 9 or 10 hours before when this file is updated. Meanwhile, the sunshine hours count has been reset and so the figure in today.ini is not what is wanted here. For that reason the sunshine hours reported here are taken from [[yesterday.ini]].
|-
|-
|colspan="4" style="background:lightgray;"|(The next 16 entries were added from version 1.9.1 May 2011)
|colspan="4" style="background:lightgray;"|(The next 16 entries were added from version 1.9.1 May 2011)
Line 563: Line 400:
|[[Heat/cold degree days and Chill hours | Cooling degree days]]
|[[Heat/cold degree days and Chill hours | Cooling degree days]]
|-
|-
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist)
|colspan="4" style="background:lightgray;"|The next two pairs were added in legacy version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist).

Fields listed up to those following here applied to the final legacy Cumulus 1.9.4 and formed the basis for early releases of Cumulus MX.
|-
|-
|43
|43
Line 585: Line 424:
|Time of high UV Index
|Time of high UV Index
|-
|-
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 3.6.0, 2 more derived values and their times
|colspan="4" style="background:lightgray;"|The next two pairs were added in MX release 3.6.0, 2 more derived values and their times
|-
|-
|47
|47
Line 607: Line 446:
|Time of low feels like temperature
|Time of low feels like temperature
|-
|-
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 3.6.12
|colspan="4" style="background:lightgray;"|The next two pairs were added in release 3.6.12


*Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times).
*Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times).
**The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor.
**The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor.
*From version 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error.
*From release 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error.
|-
|-
|51
|51
Line 621: Line 460:
|AZ
|AZ
|5 characters
|5 characters
|Time of high Humidex
| style="background:pink;"|Time of high Humidex

'''Bug for releases 3.13.0 to 3.14.2 inclusive''': The major code rewrite for release 3.13.0 replaced the previous code for the processing for all Cumulus files, the new code incorrectly stored ''Time of high feels like temperature'' in this field for all these releases! Corrected in minor code rewrite for 3.14.3 - b3163 25 Jan 2022 (not released to public until 3.15.0 - b3169 Released 31 Jan 2022)
|-
|-
|colspan="4" style="background:lightblue;"|Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in current nor any other version, so have labelled them as error.
|colspan="4" style="background:lightblue;"|Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in any other version, so have labelled them as error.
|-
|-
|53 ('''error''')
| 53 ('''error''')
|BA
| BA
|signed decimal
| signed decimal
|Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other version)
| Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other release)
|-
|-
|54 ('''error''')
| 54 ('''error''')
|BB
| BB
|5 characters
| 5 characters
|Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other version)
| Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other release)
|-
|colspan="4" style="background:lightgray;"|The next value was added in release 3.12.0
|-
| 53 (new)
| BA
| unsigned decimal
| [[Heat/cold_degree_days_and_Chill_hours#Chill_Hours_and.2For_Air_Frost|Cumulative Chill Hours]] since start of season
|-
|colspan="4" style="background:lightgray;"|The next value and time fields were added in release 3.20.0
|-
| 54
| BB
| unsigned decimal
| Highest rainfall total in (an approximate, as it depends upon timing of earlier entries in [[Recent history]]) 24 hour period ending in [[Meteorological day]]
|-
| 55
| BC
| 5 characters
| Time when (approximate) 24 hour period for rainfall total above ends
|}
|}


==Example of the file==
==Example of the file==


An extract of a few lines of the dayfile.txt
An extract of a few lines of a dayfile.txt


<pre>
<pre>
Line 655: Line 515:
07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7
07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7
</pre>
</pre>

[[Category:Log Files]]

Latest revision as of 07:14, 27 September 2022

Cumulus Version MX SpecificCumulus Version 1 SpecificThis Wiki page applies to both Cumulus flavours.

As part of a redevelopment of Wiki, this page has been simplified, by moving some content to new pages. Old links in the support forum, that were to content no longer on this page, will bring you here.

Content previously on this page has been moved as follows:

The content that remains on this page is summarised by the table of contents that follows.

About this file

  • This Wiki page describes one of the files not included in any release download.
  • This daily summary file (Steve Loft shortened that to "day" plus "file") uses a ".txt" extension, and is essentially a plain text file in CSV format.
    • CSV stands for "Comma Separated Values", but in the Cumulus context the field separator can be other characters including ";" or "|"
  • The file is created in data sub-folder of your Cumulus installation when Cumulus needs to store its first line in this file.
  • Cumulus MX reads the whole of this file into an internal array (stored in RAM) each time you restart the software,
    • Please see here for implications
    • Therefore MX users who move to a different location should archive the old "dayfile.txt", to avoid any discontinuities in any graphs plotted from active file


(speciallog.txt is another log file that contains all dates in a single file, as do all the .ini files).


How Cumulus Creates and Updates this file

  • Cumulus reads values supplied by your weather station, (either directly while Cumulus running, or for some weather station types can read historic data during catch-up on restarting Cumulus)
  • Cumulus converts them to the units you prefer,
  • Cumulus applies any calibration (multiplier and offset) you have set,
  • For a sub-set of those readings (perhaps those every minute if readings are collected every 10 seconds), the spot values of source items like temperature, humidity, wind speed, can be used in calculations of derived items like "wind chill", "dewpoint" and "feel-like" temperature.
  • Cumulus then sees if the resulting soource, or derived, value implies any extreme records file needs to be updated
    • Daily extremes are held in today.ini and that is main source used when a new line is added to "dayfile.txt"


Why this file should be backed up

  • This file contains daily extremes, the accuracy of those daily extremes depends on the interval between the spot readings that are used as explained above.
  • If the file is corrupted, and not regenerated from a back-up, the only source of spot readings is the MMMyylog.txt files, and their interval might be only every half an hour (it has to be an exact fraction of an hour).
  • As explained here minima and maxima between such logging intervals are not recorded by Cumulus software
  • Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,

Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough to find a file that has not been corrupted:

  • The file is backed up when MX is restarted into a date/time stamped sub-folder of the backup folder
  • The file is also backed up (to a date/time stamped sub-folder within daily sub-folder of that backup sub-folder) during the end of day process, but (depending on the release you are running) the back-up copy may, or may not, include a line for the previous day (the line that is appended in the same end of day)

Retention of these back ups:

  •   MX only keeps the last 9 of the date/time stamped sub-folders.
  •   Legacy Cumulus 1 only keeps up to 8 of the date/time stamped sub-folders.

Specific issues for MX

WARNING: It is important to note that some releases of MX are very fussy about consistency in all lines of dayfile.txt. Earlier releases tended to have better compatibility with the legacy software. Subsequent releases expected every line to be expressed exactly as specified in the locale. MX in its latest release is trying to cope better with inconsistent date separators.

Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.

Real number fields: MX uses the locale to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the Region settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if list separation character was comma) was fairly tolerant when reading old data in files. At the time of writing this, it was proposed the next major release will force this file to use decimal points (full stops not commas) regardless of locale. This proposal may or may not happen, the point is that information here might be incorrect for the release that you are using.

Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours.

Number of fields: The number of fields in the file increases in various versions as shown in #List_of_fields.

  • C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields.
  • Each release of MX expects a particular number of fields in each line of file; MX has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an internal (RAM held) array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file.
    • If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, there is a utility to run that will create a new dayfile.txt with all fields populated, please see Calculate Missing Values page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added, and it uses various software libraries contained within .dll files specific to a particular CumulusMX.exe release.

How to view or edit this file

The detailed information has been moved to viewing/editing dayfile.txt page.

An editor has been included within recent Cumulus releases of each flavour:

  •   Available from release 3.4.5 (13 Mar 2020): In the interface go to Data logs menu and select Dayfile
  •   Available from version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from Edit menu select Dayfile.txt
    • Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
    • Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
    • There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.

Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file.

The MX editor does not let you change the date, you can only change other fields or delete an entire line, so you have no control over the order of the lines in the file.


Reading the file

See Daily Summary page for a full discussion of ways, external to Cumulus, to read this file.

Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in vaCumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error (line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged).rious other circumstances depending on the release you are running.

  •   The legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored:
  •   As already mentioned, MX reads the entire dayfile.txt file when it is first started into an internally held array. That internal copy was introduced in release 3.9.2 - b3097 (7 Dec 2020), it was needed to drive the historic charts functionality; both in the interface, and the example included Historic Charts web page.
    • From release 3.20.0 - b3200 (21 Aug 2022) there is a This Period Highs and Lows web page (accessed from Records menu) which is similar to the screens for viewing data for various periods except that the extremes it shows match those tracked in .Ini files.
    • From release 3.20.0 - b3200 (21 Aug 2022) there is an option to Reload dayfile in a Utils menu, this updates the internal held array by reading the file again; essential if an edit has been made to the file.

Line numbering and Error messages

Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error.

Cumulus MX stores the contents of the file in an internal two dimensional array. The outer dimension is identified by numbering (from zero) what were lines in the file. The inner dimension has that line number as element zero, and the fields taken from file are therefore numbered from one.

Obviously, line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged in the file. This is why you will see line numbers quoted in any error messages related to this file.

Some of the editors you may use to view/edit the file may number the lines (in some cases always, in others as an option). If you have seen an error message, then to correct the file you need to use a tool that does number the lines in order to find where the correction is needed. Error messages report just the first error found, so check subsequent lines in case they have same error.

List of fields in dayfile.txt

For your installed build please see dayfileheader.txt. This file identifies all the fields in each line of the dayfile.txt file in the release you are running.

That file has changed location from release 3.20.0 - b3200 (21 Aug 2022), it is now in data folder, for Cumulus 1 and all earlier MX releases it was in same directory as the Cumulus executable.

As explained earlier, the number of fields in a line has (with one exception when fields were added in error) increased as the Cumulus software has been developed. The table below therefore groups the fields under labels that indicate which version/release introduced them.

Field numbering

The table below was first created when the only flavour of Cumulus software available was what we now call the legacy Cumulus 1. Back then the fields were numbered starting from zero. This reflected the way fields were identified when processed as elements of an single-dimension array variable representing a line.

The old numbering from zero had two advantages:

  1. It stressed that the date field was different to the rest, all other fields were either values or time-stamps
  2. Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown was the number to quote in any scripts where a Cumulus 1 line was converted to an array, and you needed to address a single field.

Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering. As explained above the fields are now numbered starting from 1 to fit in with an internal array created when the log file is read, the processing code adds a line number in front of the date field on each array 'line' it holds.

Information shown in the table

  • The date is supposed to be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
    • The dates, as mentioned earlier, should be in ascending order.
    • Please see amending dayfile page for more information on dates
    • Both Cumulus 1 and MX may get confused if you restart the software close to rollover time, and in that confused state they may create a second line with same date as a date already in the file.
    • If you regularly restart Cumulus close to rollover time, then a whole sequence of dates can be repeated either as a block or as pairs.
  • The remaining fields are all either numerical values, or a time paired with (except for first wind field) preceding numerical value.
    • Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed). Cumulus MX only validates that each field is of correct type, it does not recognise any relationships between fields.
  • The alphabetic column identifiers used by many spreadsheet editors are shown in the table
    • IMPORTANT: If you do use a spreadsheet editor, do ensure all columns are set to "text" format,
    • Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, or let it change the number of decimal places.
  • The type of field is shown in the table:
    • Note you must not include a sign for an unsigned field,
    • Note you can not specify a decimal comma/point in an integer field,
    • Note all fields of type time must use 5 character "HH:mm" format, although Cumulus 1 will accept other separator characters where ":" shown.
  • The field description is shown, together with references to where that terminology is explained


Table listing Fields

Field number Spreadsheet column Field type Description
0 For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.

If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1.

Those fields included below have been in dayfile.txt from the start of Cumulus 1 (Version 1.0, the First release on 27th January 2004).
1 A 8 characters Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup)
2 B Unsigned number Highest wind gust speed
3 C unsigned integer Bearing of highest wind gust
4 D 5 characters Time of highest wind gust
5 E signed decimal Minimum temperature
6 F 5 characters Time of minimum temperature
7 G signed decimal Maximum temperature
Consistency Note: In some cases Minimum comes before Maximum, in other cases Maximum is before Mimum
8 H 5 characters Time of maximum temperature
9 I Unsigned number Minimum sea level pressure
10 J 5 characters Time of minimum pressure
11 K Unsigned number Maximum sea level pressure
12 L 5 characters Time of maximum pressure
13 M unsigned number Maximum rainfall rate
14 N 5 characters Time of maximum rainfall rate
15 O unsigned number Total rainfall for the day
Above here represents the minimum length for every line, a count of 15 items
[There is no record of which version added this next field. The Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the content of each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 as these do not appear in the release history issued by Steve Loft. The first mention of it in the new forum was not until December 2008, but that was not about when it was released. A web tag for this variable was added in Build 978 of 1.9.1 beta, which was obviously long after it was first calculated.

Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment]

16 P signed decimal Average temperature for the day
(Wind run was added from version 1.8.4)
17 Q unsigned number Daily wind run
(The next pair of entries were added from version 1.8.9 build 907 (June 2010) as part of a total redesign of how dayfile.txt was implemented in Cumulus 1)
18 R unsigned number Highest Average Wind Speed
19 S 5 characters Time of Highest Avg. Wind speed
(The two pairs of humidity entries were added in October 2010, a v 1.9.0 beta, the exact build number is now lost)
20 T unsigned integer Lowest humidity
21 U 5 characters Time of lowest humidity
22 V unsigned integer Highest humidity
23 W 5 characters Time of highest humidity
(The next two entries were added from version 1.9.0)
24 X (not documented) Total evapotranspiration (Only valid for Davis stations, shows zero otherwise)
25 Y unsigned Total hours of sunshine (only valid if sunshine sensor connected)

Important if rollover time is 9 am or 10 am: Most fields in this file are updated taking information from today.ini. For a non-midnight rollover, then the Sunshine hours reported here is from 00:01 on the calendar date corresponding to the date in the first field of this file, to subsequent midnight, and that end time is 9 or 10 hours before when this file is updated. Meanwhile, the sunshine hours count has been reset and so the figure in today.ini is not what is wanted here. For that reason the sunshine hours reported here are taken from yesterday.ini.

(The next 16 entries were added from version 1.9.1 May 2011)
26 Z signed decimal High Heat index (added to Cumulus in 1.7.11 only as spot value, not stored)
27 AA 5 characters Time of high heat index
28 AB Signed decimal High Apparent temperature
29 AC 5 characters Time of high apparent temperature
30 AD signed decimal Low apparent temperature
31 AE 5 characters Time of low apparent temperature
32 AF unsigned number High hourly rain
33 AG 5 characters Time of high hourly rain
34 AH) signed decimal Greatest wind chill (high wind speed, low temperature) (calculated since version 1.8.3 as spot value, not stored)
35 AI 5 characters Time of greatest wind chill
(The next two pairs for dew point were added in version 1.9.2 beta build)
36 AJ signed decimal High dew point
37 AK 5 characters Time of high dew point
38 AL signed decimal Low dew point
39 AM) 5 characters Time of low dew point
(The next three entries were added in version 1.9.2 Build 1004)
40 AN unsigned integer Today's dominant/average wind direction
41 AO unsigned decimal Heating degree days
42 AP unsigned decimal Cooling degree days
The next two pairs were added in legacy version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist).

Fields listed up to those following here applied to the final legacy Cumulus 1.9.4 and formed the basis for early releases of Cumulus MX.

43 AQ unsigned decimal High solar radiation
44 AR 5 characters Time of high solar radiation
45 AS unsigned decimal High UV Index
46 AT 5 characters Time of high UV Index
The next two pairs were added in MX release 3.6.0, 2 more derived values and their times
47 AU signed decimal High Feels Like temperature
48 AV 5 characters Time of high feels like temperature
49 AW signed decimal Low Feels Like temperature
50 AX 5 characters Time of low feels like temperature
The next two pairs were added in release 3.6.12
  • Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times).
    • The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor.
  • From release 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error.
51 AY signed decimal High Canadian Humidity Index or Humidex
52 AZ 5 characters Time of high Humidex

Bug for releases 3.13.0 to 3.14.2 inclusive: The major code rewrite for release 3.13.0 replaced the previous code for the processing for all Cumulus files, the new code incorrectly stored Time of high feels like temperature in this field for all these releases! Corrected in minor code rewrite for 3.14.3 - b3163 25 Jan 2022 (not released to public until 3.15.0 - b3169 Released 31 Jan 2022)

Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in any other version, so have labelled them as error.
53 (error) BA signed decimal Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other release)
54 (error) BB 5 characters Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other release)
The next value was added in release 3.12.0
53 (new) BA unsigned decimal Cumulative Chill Hours since start of season
The next value and time fields were added in release 3.20.0
54 BB unsigned decimal Highest rainfall total in (an approximate, as it depends upon timing of earlier entries in Recent history) 24 hour period ending in Meteorological day
55 BC 5 characters Time when (approximate) 24 hour period for rainfall total above ends

Example of the file

An extract of a few lines of a dayfile.txt 

01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5

02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7

03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9

04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9

05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3

06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3

07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7
Retrieved from "https://www.cumuluswiki.org/index.php?title=Dayfile.txt&oldid=10741"