232
edits
Monthly log files: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
Text replacement - "http://sandaysoft.com/forum/" to "https://cumulus.hosiene.co.uk/"
m (→Manipulation outside Cumulus: added comment reflecting comment by Steve on Forum) |
SaratogaWX (talk | contribs) m (Text replacement - "http://sandaysoft.com/forum/" to "https://cumulus.hosiene.co.uk/") |
||
| (5 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
This article is about the log file that contains the values at whatever interval you have set for Cumulus to keep detailed logging. The main files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called <tt>Aug09log.txt</tt>); an entry is made to the current month's file every ''ten'' (default value; you can change this on the station settings screen) minutes, recording the current sensor values. The file is in comma-separated format with one line per entry. | |||
Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). All data is logged in the units which which have been selected by the user on the station configuration screen. Note that 'comma-separated' does not necessarily mean that a comma is used to separate the values! If your PC uses a semicolon for the list separator, that will be used in these files. Similarly, decimals use the Windows decimal separator. | See [[speciallog.txt]] for the equivalent log file for detailed logging of internal temperature and humidity. | ||
Times in these files are in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). All data is logged in the units which which have been selected by the user on the station configuration screen. Note that 'comma-separated' does not necessarily mean that a comma is used to separate the values! If your PC uses a semicolon for the list separator, that will be used in these files. Similarly, decimals use the Windows decimal separator. | |||
For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. | For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. | ||
==Viewing monthly log files== | ==Viewing monthly log files== | ||
As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus applies an exclusive lock, and (as explained further below) conflicts can happen if another process seeks to access this file. | |||
In the '''View''' menu, select ''Data logs'', click '''Load...'''. The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. Select the required monthly log, click '''Open''' and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen. | |||
Also note these log files do not include a header line, but the appropriate headers can be found for the installed version of Cumulus by looking at "Cumulus\monthlyfileheader.txt", i.e. in the directory above the 'data' directory containing the monthly log. The viewing method described next automatically includes the header line. | |||
In the '''View''' menu of Cumulus 1, select ''Data logs'', click '''Load...'''. The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. Select the required monthly log, click '''Open''' and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen. | |||
If no logs seen, see [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]]. Also note, this '''Data logs''' viewer is only for viewing monthly log files (not any other files in the same ''data'' directory), the column headings for standard monthly log files will always be shown at the top of the screen. | If no logs seen, see [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]]. Also note, this '''Data logs''' viewer is only for viewing monthly log files (not any other files in the same ''data'' directory), the column headings for standard monthly log files will always be shown at the top of the screen. | ||
| Line 13: | Line 18: | ||
==How can Monthly log files be used within Cumulus?== | ==How can Monthly log files be used within Cumulus?== | ||
===Uploading current log to your website=== | ===Uploading current log to your website=== | ||
The current monthly log can be updated at your usual website update interval using ''<currentlogfile>'' on the '''Files''' tab of the ''Internet Settings'' screen off the '''Configuration''' menu (or by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. It is recommended that your remote file is local so the updating can use copy and you do not have FTP ticked to avoid conflict with logging - see [ | The current monthly log can be updated at your usual website update interval using ''<currentlogfile>'' on the '''Files''' tab of the ''Internet Settings'' screen off the '''Configuration''' menu (or by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. It is recommended that your remote file is local so the updating can use copy and you do not have FTP ticked to avoid conflict with logging - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for a suggested method. | ||
A script can then read the file into a database (see [[ImportCumulusFile]]), or an array, for further processing. | A script can then read the file into a database (see [[ImportCumulusFile]]), or an array, for further processing. | ||
CAUTION: Be aware if you tick FTP to upload this file to a remote server, there can be conflict between the thread that Cumulus uses to run FTP and the independent thread that Cumulus uses to log the latest observations into this file. There is a danger that a FTP in progress still has exclusive access and the logging therefore fails e.g. if the interval between FTP is too small or, it is not near the start of the month so, the file is big; see [[FAQ#I_am_getting_I.2FO_error_32_or_I.2FO_error_103]]. An alterative approach is to upload this file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus. | CAUTION: Be aware if you tick FTP to upload this file to a remote server, there can be conflict between the thread that Cumulus uses to run FTP and the independent thread that Cumulus uses to log the latest observations into this file. There is a danger that a FTP in progress still has exclusive access and the logging therefore fails e.g. if the interval between FTP is too small or, it is not near the start of the month so, the file is big; see [[FAQ#I_am_getting_I.2FO_error_32_or_I.2FO_error_103]]. An alterative approach is to upload this file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus. | ||
== Editing the log file == | |||
* You can't edit the current log file while Cumulus is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors. | |||
* You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand. | |||
* You can use either a specialised comma separated value file editor or a text editor. | |||
* Text editors designed for programmers will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM). | |||
* If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected. Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ..."). | |||
* If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer). | |||
* Ensure no blank lines, and if you have added in the column headings then remove that heading line when saving ready for Cumulus. | |||
You can see [[#Manipulation outside Cumulus]] below for more information. | |||
===After you have edited (or created any missing) Monthly log files, you can:=== | ===After you have edited (or created any missing) Monthly log files, you can:=== | ||
| Line 35: | Line 52: | ||
** see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as webtags). | ** see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as webtags). | ||
** use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below). | ** use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below). | ||
** obtain (by [[Monthly_log_files#Viewing_monthly_log_files | viewing the created/edited log file]] figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]]. | ** obtain (by [[Monthly_log_files#Viewing_monthly_log_files | viewing the created/edited log file]]) figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]]. | ||
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. | ** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. | ||
| Line 52: | Line 69: | ||
'''Tips''' -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the ''not to be touched copy'' a filename of "<Month><Year>log.csv"). | '''Tips''' -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the ''not to be touched copy'' a filename of "<Month><Year>log.csv"). | ||
Edit the original file using an editor that treats all fields as text [either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date) types]. | Edit the original file using an editor that treats all fields as text [use either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date and time) types]. [[File:Open office (editing cumulus log files).png | left]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected (as normal default for this software). Best not to use 'Microsoft Excel' as that normally has opposite default, and may change without any warning the format of all dates so Cumulus can no longer recognise the first two fields: | ||
'''Important Rules''' | '''Important Rules''' | ||
*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept | *Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end). | ||
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within | * The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files''). | ||
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]]. | * Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]]. | ||
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points. | *See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points. | ||
| Line 158: | Line 175: | ||
| 11(L) | | 11(L) | ||
|215.2 | |215.2 | ||
| Total rainfall counter (content depends on weather station type, might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]) | | Total rainfall counter (content depends on weather station type, might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]. This figure is not intended to be useful in itself, it may vary for no apparent reason. It is used internally by Cumulus.) | ||
|- | |- | ||
| 12(M) | | 12(M) | ||
