Cumulus Wiki

Dayfile.txt: Difference between revisions

Dayfile.txt (view source)

Revision as of 10:01, 22 July 2021

5,744 bytes removed ,  10:01, 22 July 2021
m
Cut some text, past to daily summary page
m (→‎List of Fields: Sunshine hours note)
m (Cut some text, past to daily summary page)
Line 1: Line 1:
<big>'''This article is about the Daily Summary logging file'''</big>
{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to both Cumulus flavours.


= Introduction =
= Introduction =


Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. In all flavours of Cumulus, this file is only updated (with exclusive lock) during the end of meteorological day process. In that process it is also read, if the generation of NOAA reports has been requested.
This Wiki page describes one of the [[Category:Cumulus Files|files]] not included in any release download.  This daily summary file (its formal title is shortened to "day" plus "file") uses a [[Category:MX_txt_Files|text extension]] and is created in [[Data folder|data sub-folder]] of your Cumulus installation when Cumulus needs to store its first line in this file.
 
{{TOCright}}
Both Cumulus 1 and MX have ways to [[Amending dayfile|edit '''dayfile.txt''']] while Cumulus is running. Cumulus 2 does not even allow you to view this log file.  
==Creating dayfile.txt==
 
This file is not included in any release distribution of Cumulus.


Cumulus software creates the file, if the file does not exist, when the software does its first end of day action.
=About this file=


Whenever Cumulus does an end of day action (either in normal processing or in catch-up on a restart of Cumulus while processing archive data), Cumulus will create a new line in '''dayfile.txt''' from what it has stored in [[Today.ini#When_Cumulus_is_running|today.ini]] and at that link you can read how the parameters in '''today.ini''' map across to fields in '''dayfile.txt'''.
If you import historic data from before you started using Cumulus into [[Standard log files]], then the daily summary for that imported data can be summarised into '''dayfile.txt''' line format (and a dayfile.txt file created/updated) using any of the approaches described on the [[Calculate_Missing_Values|calculate missing values]] page of this Wiki.


If you import historic data from before you started using Cumulus, and have this in [[Standard log files]], then the daily summary for imported data can be converted into dayfile.txt line format (and a dayfile.txt file created/updated) using any of the approaches described on [[Calculate_Missing_Values|calculate missing values]] page.
The single file '''dayfile.txt''' can contain lines created over a long period of time ([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Ini_Files|.ini files]]).  


==How Cumulus Creates and Updates this file==


==[[File:Badge vMx.png]] WARNING==
Cumulus reads values supplied by your weather station, converts them to the units you prefer, applies any calibration (multiplier and offset) you have set, and then sees if the resulting value implies [[Category:Ini Files|any extreme records file]] needs to be updated (these hold derived values like totals, highest, and lowest).  Two of those files are [[Today.ini#When_Cumulus_is_running|today.ini]] and [[yesterday.ini]] and values from those two files are exacted (see linked Wiki pages for how entries in those files map across to the daily summary file) at the end of a [[meteorological day]] to calculate the necessary [[#List_of_fields|fields]] for a new line to be added to this file.


It is important to note that MX is very fussy about consistency in all lines of '''dayfile.txt'''.
As explained on the "meteorological day" page linked above, if your weather station stores weather data, then Cumulus on restarting can (optionally) read the '''historic data''' and catch-up by processing those past readings (as per description in start of previous paragraph), including activating any end of day actions (see [[Today.ini#Restart_and_Catch-up|today.ini]]), for the period since it was last running.


MX reads the entire '''dayfile.txt''' file, to drive the  '''historic charts functionality'''; both in the [[MX Administrative Interface|admin interface]], and the example included [[New Default Web Site Information|web page]]. Consequently, any error in say the date field (or change of separating character) will stop historic charts working.
==How to view or edit this file==


Date field: Cumulus 1 did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.  
An editor has been included within Cumulus (look up the linked pages for screen shots):
* [[File:Badge vMx.png]] From release 3.4.5 (13 Mar 2020) onwards:  In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy  or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
* Cumulus 2 has no viewing option
* [[File:Badge v1.png]] From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''


Real number fields: MX uses the locale to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator.
For detailed information, please see [[Amending dayfile|viewing/editing '''dayfile.txt''']].


Number of fields: The number of fields in the file increases in various versions as shown at the end of this article.  C1 can cope with 15 to 45 fields in each line. MX (except for the early releases) assumes 48 fields in each line. If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than 48 fields, you can add the missing derived fields, please see [[Calculate Missing Values]] page.
You might also find it useful to read [[Correcting Extremes]] page for information about fixing other files where the highest/lowest/total recorded for day has been corrupted by rogue values.  


Please see [[Amending dayfile]] page for information about how to edit this file.
=Reading the file=


Please see [[Correcting Extremes]] page for information about fixing where the highest/lowest/total recorded for day has been corrupted by rogue values.  
See [[Daily Summary]] page for a full discussion of ways, external to Cumulus, to read this file.  


Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in various other circumstances depending on the release you are running.


[[File:Badge v1.png]] The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], these use several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt".  The '''Select a graph''' feature also uses several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt".  Thus the legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored, and the end of day action uses a simple "append" instruction.


{{TOCright}}
[[File:Badge vMx.png]] From release 3.9.2 - b3097 (7 Dec 2020), MX reads the entire '''dayfile.txt''' file, to drive the  '''historic charts functionality'''; both in the [[MX Administrative Interface|admin interface]], and the example included [[New Default Web Site Information|Historic Charts web page]]. Consequently, any error in say the date field (or change of separating character) will stop historic charts working.


== Specific issues for MX ==


WARNING: It is important to note that MX is very fussy about consistency in all lines of '''dayfile.txt'''.


== Changes in different releases of Cumulus ==
Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.


Be aware that this article was written for Cumulus 1, amended to also cover MX 3.0.0.  Whilst the list of fields has been kept updated for more recent MX versions, other parts of this article may not be applicable to latest MX version, check for edits by developer.
Real number fields: MX uses the '''locale''' to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the '''Region''' settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if ''list separation character'' was comma) was fairly tolerant when reading old data in files.


=== Notes for Legacy Cumulus===
Number of fields: The number of fields in the file increases in various versions as shown in [[#List_of_fields]]. 
{{Version badge 1}}
* C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields.
* MX too has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file.  If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, you can add the missing derived fields, please see [[Calculate Missing Values]] page.  Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added.


In Cumulus 1 only, the figures contained in the file are used for the 'This period' display (and the month/year displays) accessed from the '''View''' menu; and to build any graphs based on daily values.
=Why this file should be backed up=
*The original Cumulus software reads the contents of '''dayfile.txt''' for many of the display and edit menu options that can be selected from the main Cumulus 1 screen.
**As the file is read each time you select to display a period, or update for a new period, you will see the results of any edit of dayfile.txt, whether with in-built editor or an external editor.


*'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
As explained [[Monthly_log_files#Enhancement never implemented|here]] Cumulus only logs periodic spot values as it works through processing the values it processes from your weather station. Therefore the [[Standard log files]]  may not include any maximum or minimum values supplied by your weather station. However, on a day-by-day basis these should be preserved in this daily summary file.
*'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy  or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
*There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.


== When Cumulus processes the end of the (meteorological) day ==
Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,
*A new row is appended to dayfile.txt, the values are prepared from reading "today.ini" file, not all values available in "today.ini" are stored in dayfile.txt.
*Some of this information is also stored in [[yesterday.ini]].
*Back ups of both today.ini, and dayfile.txt, log files ''in their once a day state (either '''before''' or '''after''' changes in the end of day update'', depending on which version you have running) are copied to [[Backup folder]].


===Options for reading dayfile.txt for other uses===
Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and so your main file is corrupted you might not be able to go far back enough for an uncorrupted file:
* The file is backed up when MX is restarted into a date/time stamped sub-folder of the [[Backup folder|'''backup''' folder]]
*The file is also backed up (to a date/time stamped sub-folder within '''daily''' sub-folder of that backup sub-folder) during the end of day process, depending on the release you are running the back up copy may, or may not, include the line that is appended in the same end of day


See [[Daily Summary]] page for a full discussion.
Retention of these back ups:
*Some people take a copy of the local dayfile.txt, and use it locally for other purposes.
* [[File:Badge vMx.png]] MX only keeps the last 9 of the date/time stamped subfolders.
*A number of third-party routines that make use of this file (and in some cases, others) are listed at [[:Category:WebTools]].
* [[File:Badge v1.png]] Legacy Cumulus 1 only keeps up to 8 of the date/time stamped subfolders.
** Some people use [[Upload_Dayfile|the method described here]] or [[Toolbox|Cumulus Toolbox]] to get a copy of the local file to use on their web server. (If your web server is local, you can simply copy the file to that server, once a day).
** Search in the Cumulus support forum for examples of third-part JavaScript projects that read the web copy, for example to insert data for one year ago, or to enable extremes for each day in a week to be included in a web page.  
**Dayfile.txt (and other files!) are used as described at the [[CumulusUtils]] link.
*Other people load the contents of their dayfile.txt into a database table, allowing SQL enquiries to efficiently display, or summarise, the contents of the file.
**Cumulus MX provides optional functionality to create a database table, and to automatically insert a new row each time a new line is added to the local log file.


=== Populating a database table ===
*{{Version badge 1}} The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have been edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using.
*[[File:Badge vMx.png]] Please see [[MX_Administrative_Interface#Standard_Daily_Summary_Table]] section for details of how '''CumulusMX.exe''' has an optional feature with a standard option to insert a new row into a database table holding columns relating to the dayfile.txt fields.  Either (for all early MX releases) '''ExportMySql.exe''', or (for all later MX releases) '''ExportToMySQL.exe''', can update the database table with past rows.
==== Using a database table ====
In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log.
For examples of some of the third party tools (Cumulus [[Category:User_Contributions|user contributions]]) using the database daily summary table see [[Daily Summary#Some_example_Scripts|here]].
In my case, I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table, i.e. I have one database table column for each of the weather derivatives I show on my web page that show this month's values; it is many more derivatives than are shown on the standard web page, but some are initially hidden. Consequentially, when my daily update script detects from the date that it is processing the last day of a month, it then starts another script that reads all the rows in the daily summary table for that month, and stores the highest/lowest/total (as relevant) in my monthly_summary table (nothing to do with the "monthly" table that MX can generate from its standard log file). This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Just another example of how much you can get from just one (daily summary) log file!
==== Alternative schemas ====
Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files.
*{{Version badge 1}}
**With Cumulus 1, you would need to be a programmer and write your own script to update the database table with your own schema. You might use the importCumulusFile article to start you off.
**You might also, as I did, want your script to validate what it reads from the daily summary log to ensure only valid numbers and times are stored in your database table, while any invalid inputs are stored as nulls by your script.  In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the contents of the Weather Diary, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). When I used Cumulus 1 I wrote a PHP script to find all these additional values, for example it reads the [[today.ini]] and [[month.ini]] log files as stored in the end of day backup (not the ones being updated for new day in data folder), and the [[log.xml|weather diary in log.xml in data folder]].
*[[File:Badge vMx.png]]
**MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned.
**MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do.
== When Cumulus is restarted after a break in running  ==
* It reads the daily summary log (dayfile.txt) and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|this Cumulus 1 FAQ]])
* Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
* For Cumulus 1, back ups of 8 selected log files including dayfile.txt that are copied to start-up folders in the 'cumulus\backup' folder, the last 8 start-up folders only are retained.
* For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are  copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else.
* For MX, except early releases which did not have this functionality, the whole of dayfile.txt is read when MX starts as it provides the daily data for the historic charts (both in admin interface charts and default web site chart data).
If your weather station includes it own data logger, then Cumulus does a catch-up during which Cumulus will read "archived records" from that weather station log, and when it detects the archive log has moved to a new meteorological day, Cumulus will run the "end of day" process that moves what is in [[Today.ini#Restart_and_Catch-up|today.ini]] into a new line in dayfile.txt as described in that linked section.


= List of fields in dayfile.txt =
= List of fields in dayfile.txt =
==The number of fields in a line==
The single file '''dayfile.txt''' can contain lines created over a long period of time ([[Speciallog.txt|speciallog.txt]] is another log file that contains all dates in a single file, as do all the [[:Category:Log Files|.ini files]]).
For some people, the file was first created  by Cumulus 1 (at various versions over time), and then moved to MX.  Such files may have different numbers of fields depending on when each line was either created or last updated. Cumulus does not in normal operation modify earlier lines, but both Cumulus 1 and MX provide editors where it is possible to modify any line, and any line modified will from then onwards have the number of fields defined for the Cumulus version being used..
There is advice on editing the file on [[Calculate_Missing_Values#Missing_fields_.28or_missing_lines.29_in_dayfile.txt|Calculate Missing Values page]], using the Cumulus in-built editor. Should you want to edit the file outside Cumulus, a "Comma Separated Value" editor has advantage of being able to maintain the differing existing lengths for each line.  One example of such a tool in the Microsoft Windows environment is [https://csved.sjfrancke.nl/ freeware CSV editor software by Sam Francke for Windows].  If you do edit the file (within Cumulus or outside), there are [[Calculate_Missing_Values#Rules_when_editing_daily_summary_log|Rules to follow]].
=== Variation by Cumulus version ===
The number of fields in '''dayfile.txt''' has grown as time has gone by; simply because Cumulus's functionality has been extended both to cover more sensors and to calculate more derivatives. The only way that Cumulus can retain a daily summary for past days is by using this file. 


'''For your installed build please see ''dayfileheader.txt'' (stored within the folder that contains your Cumulus executable), as that will list which fields your Cumulus installation uses.'''
'''For your installed build please see ''dayfileheader.txt'' (stored within the folder that contains your Cumulus executable), as that will list which fields your Cumulus installation uses.'''
Line 125: Line 75:
If you have been using Cumulus for a while, you may wonder which of your log file lines might be shorter, so the table below shows fields grouped by the Cumulus version when those fields were added.
If you have been using Cumulus for a while, you may wonder which of your log file lines might be shorter, so the table below shows fields grouped by the Cumulus version when those fields were added.


For the original Cumulus software, each line of this file can contain anything from 15 to 45 fields, and having some lines shorter than others does not matter.
For Cumulus MX, several of the earliest releases supported only the same 45 fields as used by 1.9.4, a few early releases had 50 fields, just one particular release supported 54 fields (in error), then several releases supported exactly 52 fields, but from release 3.12.0 MX expects exactly 53 fields.
If you have any lines with 15 to 52 fields then MX can still read that line, but if you edit that line using the editor in MX, it will show all fields when you edit it, and when you save that line.
Please note that if you use MX, then there is a utility program that can insert missing fields for you. Please see [[Calculate_Missing_Values#CreateMissing.exe|here]].


== Information shown in the table ==
== Information shown in the table ==
Line 141: Line 84:
*The original table below was for Cumulus 1 and then field number '''was''' starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
*The original table below was for Cumulus 1 and then field number '''was''' starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
*#Cumulus 1 stressed that the date field was different to the rest, as it was used as identifier. The date '''must''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
*#Cumulus 1 stressed that the date field was different to the rest, as it was used as identifier. The date '''must''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
*#The remaining fields were all either numerical values, or a time paired with preceding numerical value. Cumulus 1 actually forces this pairing.
*#The remaining fields were all either numerical values, or a time paired with preceding numerical value. Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed).
**Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
**Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
*The alphabetic column identifiers used by many spreadsheets are shown, please see warnings about using spreadsheets for editing earlier on this page
*The alphabetic column identifiers used by many spreadsheets are shown, please remember to select options in your spreadsheet to ensure '''all columns are set to "text" format''', you will corrupt this file if you let your spreadsheet recognise content as dates or time, or change the number of decimal places.
*The type of field is shown, you must not put a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format
*The type of field is shown in the table, you must not include a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format
*The field description is shown, together with references to where that terminology is explained
*The field description is shown, together with references to where that terminology is explained


Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/9513"