5,838
edits
Dayfile.txt: Difference between revisions
m
→Important Rules when editing dayfile.txt
| Line 92: | Line 92: | ||
== Important Rules when editing dayfile.txt == | == Important Rules when editing dayfile.txt == | ||
If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it. | |||
=== General Editing Rules === | |||
# Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use. | |||
# Take another copy and use that for your editing, don't edit the actual file being used by the software. | |||
#*This prevents any conflicts between access by the software and access by your script or tool being used to modify the file. | |||
#*It also means that you can go back to the last working copy, you can't upset your "revert" copy. | |||
#The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values. | |||
#* It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used. | |||
#*You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus. | |||
=== File specific Editing Rules === | |||
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM. | |||
# All rows must ''start with date'' and include at least 14 further fields ''in correct sequence''. | |||
# The (meteorological) date format uses ''two digits for the year''. | |||
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types). | |||
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or on Microsoft Windows '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file. | |||
#Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile. | |||
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space. | |||
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus. | |||
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits. | |||
#* If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent. | |||
* The fields are separated using the Windows (or whatever operating system you are using for MX) list separator (e.g. a comma or semi-colon)[[File:Open office (editing cumulus log files).png]] If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. | |||
* Rows can vary in length but only by missing off ''fields at the end''. | |||
* Most value fields are in ''real number format x.y'' using your system decimal notation, a few (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field. | |||
* If you insert a ''lowest or highest value'' for a new day, where there was no record before, insert a ''time-stamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if each value has any related time-stamp. (Use a time-stamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time). If you are using a 9am or 10am rollover time, create missing in Cumulus inserts 00:00 for null time-stamps, but normally Cumulus uses the rollover time for null time-stamps. | |||
* Times appearing for some of the fields must always be in ''format HH:mm'' i.e. 2 digit hour, followed by a colon, then 2 digit minutes (Be aware you will have problems if you, or your editing software, add seconds). Except for wind gust (start of line), each time field will immediately follow the value for that parameter. | |||
* Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool. (during editing after all available valid parameters inserted, extra field separators may be added at end of shorter lines inserted by 'Create Missing' or by a spreadsheet as these make all lines end up with same number of fields) | |||
* Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps. If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator). | |||
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder. | |||
**Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day. | |||
*** Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases. | |||
**Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). You must never do this in the dayfile.txt that Cumulus processes. My suggestion is use the 'External Program' facility to create a copy of ''dayfile.txt'', make any such changes only on that copy, and set the third-party script to read this copy. | |||
*** Alternatively, do as I do, clean your data as you upload the file into a database. I use a script with validation code checking for rogue values and recognising the -999.9 etc., in all cases my script will store a '''NULL''' value as default in the database table if that validation finds any obviously invalid figure. | |||
* The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. Similar rules apply for terminators used by other operating systems, so be careful if you edit the file on a different device to that running Cumulus. | |||
* Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing. | |||
* Don't add a header line to the file, Cumulus expects all lines to be data lines. | |||
== Cumulus 1 == | == Cumulus 1 == | ||