Dayfile.txt: Difference between revisions

886 bytes added ,  19:18, 3 June 2015
→‎Manipulation outside Cumulus: added illustration & slightly changed text order
(→‎Manipulation outside Cumulus: Expansion of 'NULL' section)
(→‎Manipulation outside Cumulus: added illustration & slightly changed text order)
Line 94: Line 94:


'''Important Rules''':  
'''Important Rules''':  
* The (meteorological) date format uses ''two digits for the year''. Edit this file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed ''not'' to recognise special field types).  For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within logfiles).
* The (meteorological) date format uses ''two digits for the year''. Edit this file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed ''not'' to recognise special field types).  For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within logfiles). [[File:Open office (editing cumulus log files).png]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected. 
* Times are in ''format hh:mm''
* Times are in ''format hh:mm'' (Be aware you will have problems if you, or your editing software, add seconds).
* Most value fields are in ''real number format x.y'' using your system decimal notation, a few (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field.
* Most value fields are in ''real number format x.y'' using your system decimal notation, a few (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field.
* If you insert a ''lowest or highest value'' into a day, where there was none before, insert a ''timestamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if a value and any related timestamp are either both present or both absent. (Use a timestamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time).
* If you insert a ''lowest or highest value'' for a new day, where there was no record before, insert a ''time-stamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if each value has any related time-stamp are either both present or both absent. (Use a time-stamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time).
* Nulls ',,' are not allowed in the line (except during editing after all available valid parameters inserted, they may be added at end of lines inserted by 'Create Missing' (or your external editing software) as all lines end up with same number of fields), so if you do not know the value for a particular field within the line, then type in a zero for integer values and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for decimal format (replace the full stops with your decimal separator).
* The row terminator normally expected is ''CRLF'', ensure any external editor does not change the terminator.
* The fields are separated using the Windows list separator (e.g. a comma or semi-colon)
* Nulls (2 field separators without something between them ',,') are not allowed within the part of the line with values and time-stamps, so if you do not know the value for a particular field within the line, then type in a zero for integer values and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for decimal format (replace the full stops with your decimal separator).
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes.  This can make editing by picking values in logs harder.
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes.  This can make editing by picking values in logs harder.
**Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide.
* Rows can vary in length but only by missing off ''fields at the end''.
**Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). My suggestion is use the 'External Program' facility to create a copy of ''dayfile.txt'' and make any such changes only on that copy.  Alternatively, do as I do, upload the file into a database, with validation code checking for the -999.9 etc, and store the '''NULL''' value if that validation finds such an obviously invalid figure.
* Shorter lines can have multiple field separators added at end of row (during editing after all available valid parameters inserted, they may be added at end of lines inserted by 'Create Missing' (or your external editing software) as all lines end up with same number of fields)
* Make sure that any editing does not create any ''blank lines'' in the file.
**Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
**Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). My suggestion is use the 'External Program' facility to create a copy of ''dayfile.txt'' and make any such changes only on that copy, and set the third-party script to read this copy.  Alternatively, do as I do, upload the file into a database, with validation code checking for the -999.9 etc, and store a '''NULL''' value as default in the database if that validation finds such an obviously invalid figure.
* Make sure that any editing does not create any ''blank lines'' in the file (can have 1 empty line at end).
* All rows must ''start with date'' and include some of the parameters listed ''in correct sequence''.
* All rows must ''start with date'' and include some of the parameters listed ''in correct sequence''.
* Rows can vary in length but only by missing off ''fields at the end''.
* The row terminator normally expected is ''CRLF'', ensure any external editor does not change the terminator.
* The fields are separated using the Windows list separator (e.g. a comma)


==List of fields in the file==
==List of fields in the file==
5,838

edits