232
edits
Dayfile.txt: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
changed sandaysoft.com references
(→Manipulation outside Cumulus: added illustration & slightly changed text order) |
SaratogaWX (talk | contribs) m (changed sandaysoft.com references) |
||
| (11 intermediate revisions by one other user not shown) | |||
| Line 3: | Line 3: | ||
== How Cumulus uses the daily log == | == How Cumulus uses the daily log == | ||
*Cumulus is frequently reading observations from your weather station | *Cumulus is frequently reading observations from your weather station. | ||
*It tracks the highs and lows in those observations by comparing read values against those it has stored in [[Today.ini]]. | *It tracks the highs and lows in those observations by comparing read values against those it has stored in [[Today.ini]], updating that file as required. | ||
*(It also updates [[Alltime.ini]], [[Monthlyalltime.ini]], [[Year.ini]], and [[Month.ini]] when appropriate. | *(It also updates [[Alltime.ini]], [[Monthlyalltime.ini]], [[Year.ini]], and [[Month.ini]] when appropriate. | ||
*When Cumulus processes the end of the (meteorological) day, the highs, lows and other information in 'today.ini' are used to | See the Wiki pages for these files or the Cumulus Help for more information, including the details of where the previous values are logged for updates to these highs and lows). | ||
*When Cumulus processes the end of the (meteorological) day, the highs, lows and other information in 'today.ini' are used to append a new line into dayfile.txt, then a new 'today.ini' is created initialised with the observations just read from the weather station. | |||
*When Cumulus is restarted after a break in running, | |||
*#it uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|FAQ]]) | |||
*#depending on the make of weather station: | |||
*#*read observations from the station's log into the Cumulus monthly log, updating today.ini as it processes the station log; | |||
*#*otherwise the observations in the station's current measurements holder at the Cumulus logging interval are what gets written into the monthly log, and update today.ini. | |||
*Back ups of all the files mentioned here are copied to the 'cumulus\backup\daily' folder. | *Back ups of all the files mentioned here are copied to the 'cumulus\backup\daily' folder. | ||
*There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options selected from the main Cumulus screen. | *There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options when selected from the main Cumulus 1 screen. | ||
*The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file at end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, one or other may fail. You can | |||
== How you can use the daily log == | |||
*If you want to run scripts that use the daily log, it is best if you take a copy first, I take a copy that is put onto my web server by using the '''Daily''' box in the bottom left of the ''Sites/Options'' frame within the ''Internet'' options screen from the '''Configuration''' menu to safely take a copy of 'dayfile.txt' after it is updated. See Cumulus '''Help''' for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from my "T:\Cumulus\daily_batch_all.cmd" in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure. My Cumulus 1 batch command also runs PHP to create SQL to update a database table version of the log, but my database table version also contains some additional information from the daily backup '[[today.ini]]' log. | |||
*The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file at end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, one or other may fail. | |||
*You can find examples of code that uses the daily log elsewhere in this Wiki and on the support forum. | |||
*Cumulus MX includes the ability to generate SQL to update a database table version of the log. Using that database table avoids any clash with the Cumulus MX update of the daily summary log. | |||
== Viewing or Editing the daily summary log == | |||
=== Editing the file or other Manipulation outside Cumulus === | |||
The file is <tt>data\dayfile.txt</tt> within the directory holding the Cumulus excutable, it can be viewed in a text editor, imported into various database systems, or imported into spreadsheets, to manipulate as you wish. Just remember that Cumulus updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating (and possibly doing a rollover of logs), or around the midnight/9am/10am local rollover time when Cumulus is writing a new row. | |||
'''Tip''': Take a copy of the file before you work on it outside Cumulus, so you can revert to old file. | |||
'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end). (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page). | |||
'''Important Rules''': | |||
These notes were written for Cumulus 1, see the MX part of the forum for any differences introduced in the future. | |||
* The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values. | |||
* Use either a specialised "Comma Separated Value" file editor or a text editor. | |||
* All rows must ''start with date'' and include some of the parameters listed ''in correct sequence''. | |||
* The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM. | |||
* The (meteorological) date format uses ''two digits for the year''. Edit this file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed ''not'' to recognise special field types). For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. | |||
* (Remember the month must be the middle figure in the date, USA convention cannot apply within logfiles). The separator between the three parts of the date will be a hyphen or a slash, it cannot be a space. | |||
* The fields are separated using the Windows list separator (e.g. a comma or semi-colon)[[File:Open office (editing cumulus log files).png]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected. | |||
* Times are in ''format hh:mm'' (Be aware you will have problems if you, or your editing software, add seconds). | |||
* Rows can vary in length but only by missing off ''fields at the end''. | |||
* Shorter lines can have multiple field separators added at end of row (during editing after all available valid parameters inserted, extra field separators may be added at end of shorter lines inserted by 'Create Missing' or by a spreadsheet as these make all lines end up with same number of fields) | |||
* Most value fields are in ''real number format x.y'' using your system decimal notation, a few (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field. | |||
* If you insert a ''lowest or highest value'' for a new day, where there was no record before, insert a ''time-stamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if each value has any related time-stamp. (Use a time-stamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time). If you are using a 9am or 10am rollover time, create missing in Cumulus inserts 00:00 for null time-stamps, but normally Cumulus uses the rollover time for null time-stamps. | |||
* Nulls (2 field separators without something between them ',,') are not allowed within the part of the line with values and time-stamps, so if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator). | |||
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder. | |||
**Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day. | |||
**Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). My suggestion is use the 'External Program' facility to create a copy of ''dayfile.txt'' and make any such changes only on that copy, and set the third-party script to read this copy. Alternatively, do as I do, clean your data as you upload the file into a database, with validation code checking for the -999.9 etc, and store a '''NULL''' value as default in the database if that validation finds such an obviously invalid figure. | |||
* The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. | |||
* Make sure that any editing does not create any ''blank lines'' in the file. | |||
=== Editing daily summary in Cumulus === | |||
'''This section applies to Cumulus 1.x.y only - Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor''' | |||
The last command in '''Edit''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu |menu]] is ''dayfile.txt''. ''This is how you view'' the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. | The last command in '''Edit''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu |menu]] is ''dayfile.txt''. ''This is how you view'' the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. | ||
Click the ''Help'' button for detailed instructions. Cumulus Help is comprehensive. | Click the ''Help'' button for detailed instructions. Cumulus Help is comprehensive. | ||
If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an exta row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have | If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an exta row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged. | ||
If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be. | If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be. | ||
| Line 23: | Line 64: | ||
You can use this editor as follows: | You can use this editor as follows: | ||
*use '''insert''' key to add one or more missing rows (complete days) manually, | *use '''insert''' key to add one or more missing rows (complete days) manually, | ||
*correct individual values by | *correct individual values by over-typing, | ||
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message), | *use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message), | ||
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Monthly_log_files]] and automatically calculating the best approximations for each field for those missing days. | *use '''Create missing''' button to insert missing rows (complete days) by reading from [[Monthly_log_files]] and automatically calculating the best approximations for each field for those missing days. | ||
| Line 39: | Line 80: | ||
'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order) | '''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order) | ||
There are no known bugs in version 1.9.4 | There are no known bugs in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard release as this section was updated. | ||
=== Dealing with rogue measurements === | === Dealing with rogue measurements === | ||
Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes. | Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes. | ||
| Line 62: | Line 103: | ||
== Using the daily summary log on your web-site == | == Using the daily summary log on your web-site == | ||
If you upload the file to your web site | If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page. I once had a web page that showed a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file. Search the Cumulus support forum to see (for example) howt others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago. | ||
If you use a script to read what is in the daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see [[Daily_Summary]] article for information about ALL of the ways to show values from this database table. | |||
==Viewing summary figures for a month or period== | ==Viewing summary figures for a month or period== | ||
| Line 85: | Line 128: | ||
**The web tags report longest dry or wet period in the month. | **The web tags report longest dry or wet period in the month. | ||
==List of fields in the file== | ==List of fields in the file== | ||
| Line 128: | Line 148: | ||
* 13 (N): Time of maximum rainfall rate | * 13 (N): Time of maximum rainfall rate | ||
* 14 (O): Total rainfall for the day | * 14 (O): Total rainfall for the day | ||
(This point represents the minimum length of all records) | |||
* 15 (P): [[Average temperature]] for the day | * 15 (P): [[Average temperature]] for the day | ||
* 16 (Q): Total [[Windrun | wind run]] | * 16 (Q): Total [[Windrun | wind run]] | ||
(The last two entries were added from v 1.8.9 build 907) | |||
* 17 (R): Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]] | * 17 (R): Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]] | ||
* 18 (S): Time of Highest Avg. Wind speed | * 18 (S): Time of Highest Avg. Wind speed | ||
(Those two entries were added from v 1.9.0) | |||
* 19 (T): Lowest [[Temperature_(and_humidity)_measurement | humidity]] (integer) | * 19 (T): Lowest [[Temperature_(and_humidity)_measurement | humidity]] (integer) | ||
* 20 (U): Time of lowest humidity | * 20 (U): Time of lowest humidity | ||
| Line 149: | Line 171: | ||
* 33 (AH): Low [[wind chill]] | * 33 (AH): Low [[wind chill]] | ||
* 34 (AI): Time of low wind chill | * 34 (AI): Time of low wind chill | ||
(The last 16 entries were added in version 1.9.1) | |||
* 35 (AJ): High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]] | * 35 (AJ): High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]] | ||
* 36 (AK): Time of high dew point | * 36 (AK): Time of high dew point | ||
* 37 (AL): Low dew point | * 37 (AL): Low dew point | ||
* 38 (AM): Time of low dew point | * 38 (AM): Time of low dew point | ||
(Last 4 entries added in version 1.9.2 beta build) | |||
* 39 (AN): Today's dominant/average wind direction (integer) | * 39 (AN): Today's dominant/average wind direction (integer) | ||
* 40 (AO): [[Heat/cold degree days and Chill hours | Heating degree days]] | * 40 (AO): [[Heat/cold degree days and Chill hours | Heating degree days]] | ||
* 41 (AP): [[Heat/cold degree days and Chill hours | Cooling degree days]] | * 41 (AP): [[Heat/cold degree days and Chill hours | Cooling degree days]] | ||
(These last three entries were added from 1.9.2 Build 1004) | |||
Finally, added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist) | |||
* 42 (AQ): High solar radiation | * 42 (AQ): High solar radiation | ||
* 43 (AR): Time of high solar radiation | * 43 (AR): Time of high solar radiation | ||
* 44 (AS): High UV Index | * 44 (AS): High UV Index | ||
* 45 (AT): Time of high UV Index | * 45 (AT): Time of high UV Index | ||
(Cumulus MX uses all fields as listed above) | |||
==Example of the file== | ==Example of the file== | ||
