232
edits
Monthly log files: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
Text replacement - "http://sandaysoft.com/forum/" to "https://cumulus.hosiene.co.uk/"
m (→After you have edited (or created any missing) Monthly log files, you can:: Complete previous incomplete correction) |
SaratogaWX (talk | contribs) m (Text replacement - "http://sandaysoft.com/forum/" to "https://cumulus.hosiene.co.uk/") |
||
| (19 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
This article is about the log file that contains the values at whatever interval you have set for Cumulus to keep detailed logging. The main files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called <tt>Aug09log.txt</tt>); an entry is made to the current month's file every ''ten'' (default value; you can change this on the station settings screen) minutes, recording the current sensor values. The file is in comma-separated format with one line per entry. | |||
Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). All data is logged in the units which which have been selected by the user on the station configuration screen. Note that 'comma-separated' does not necessarily mean that a comma is used to separate the values! If your PC uses a semicolon for the list separator, that will be used in these files. | See [[speciallog.txt]] for the equivalent log file for detailed logging of internal temperature and humidity. | ||
Times in these files are in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). All data is logged in the units which which have been selected by the user on the station configuration screen. Note that 'comma-separated' does not necessarily mean that a comma is used to separate the values! If your PC uses a semicolon for the list separator, that will be used in these files. Similarly, decimals use the Windows decimal separator. | |||
For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. | For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. | ||
==Viewing monthly log files== | ==Viewing monthly log files== | ||
As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus applies an exclusive lock, and (as explained further below) conflicts can happen if another process seeks to access this file. | |||
Also note these log files do not include a header line, but the appropriate headers can be found for the installed version of Cumulus by looking at "Cumulus\monthlyfileheader.txt", i.e. in the directory above the 'data' directory containing the monthly log. The viewing method described next automatically includes the header line. | |||
In the '''View''' menu of Cumulus 1, select ''Data logs'', click '''Load...'''. The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. Select the required monthly log, click '''Open''' and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen. | |||
If no logs seen, see [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]]. Also note, this '''Data logs''' viewer is only for viewing monthly log files (not any other files in the same ''data'' directory), the column headings for standard monthly log files will always be shown at the top of the screen. | |||
==How can Monthly log files be used within Cumulus?== | |||
===Uploading current log to your website=== | |||
The current monthly log can be updated at your usual website update interval using ''<currentlogfile>'' on the '''Files''' tab of the ''Internet Settings'' screen off the '''Configuration''' menu (or by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. It is recommended that your remote file is local so the updating can use copy and you do not have FTP ticked to avoid conflict with logging - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for a suggested method. | |||
A script can then read the file into a database (see [[ImportCumulusFile]]), or an array, for further processing. | |||
CAUTION: Be aware if you tick FTP to upload this file to a remote server, there can be conflict between the thread that Cumulus uses to run FTP and the independent thread that Cumulus uses to log the latest observations into this file. There is a danger that a FTP in progress still has exclusive access and the logging therefore fails e.g. if the interval between FTP is too small or, it is not near the start of the month so, the file is big; see [[FAQ#I_am_getting_I.2FO_error_32_or_I.2FO_error_103]]. An alterative approach is to upload this file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus. | |||
== Editing the log file == | |||
* You can't edit the current log file while Cumulus is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors. | |||
* You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand. | |||
* You can use either a specialised comma separated value file editor or a text editor. | |||
* Text editors designed for programmers will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM). | |||
* If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected. Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ..."). | |||
* If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer). | |||
* Ensure no blank lines, and if you have added in the column headings then remove that heading line when saving ready for Cumulus. | |||
You can see [[#Manipulation outside Cumulus]] below for more information. | |||
===After you have edited (or created any missing) Monthly log files, you can:=== | ===After you have edited (or created any missing) Monthly log files, you can:=== | ||
* CORRECT HIGHS AND LOWS | * CORRECT HIGHS AND LOWS | ||
| Line 41: | Line 52: | ||
** see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as webtags). | ** see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as webtags). | ||
** use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below). | ** use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below). | ||
** obtain (by [[Monthly_log_files#Viewing_monthly_log_files | viewing the created/edited log file]] figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]]. | ** obtain (by [[Monthly_log_files#Viewing_monthly_log_files | viewing the created/edited log file]]) figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]]. | ||
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. | ** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. | ||
| Line 47: | Line 58: | ||
==Correcting any logged data problems== | ==Correcting any logged data problems== | ||
The 'Monthly Data Logs Viewer' screen accessed by selecting ''Data logs'' on the '''View''' menu does not allow editing of values, but you can use it to look and see if you are happy with what has been recorded. | |||
Some Weather stations may occasionally supply corrupted or rogue values to Cumulus. Cumulus provides via '''Configuration''' menu [[Cumulus.ini#Section:_Offsets|''Calibration'']] screen the ability to screen out spikes (i.e. abnormal differences between two successive readings from weather station) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with future spikes. Some rogue values may be obviously invalid (e.g. a zero pressure), others (e.g. wind shake on a tipping bucket rain gauge on a fine day) may be within the accepted all-time range (so cannot be prevented by spike settings), but you know the value is not right for that particular time. | Some Weather stations may occasionally supply corrupted or rogue values to Cumulus. Cumulus provides via '''Configuration''' menu [[Cumulus.ini#Section:_Offsets|''Calibration'']] screen the ability to screen out spikes (i.e. abnormal differences between two successive readings from weather station) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with future spikes. Some rogue values may be obviously invalid (e.g. a zero pressure), others (e.g. wind shake on a tipping bucket rain gauge on a fine day) may be within the accepted all-time range (so cannot be prevented by spike settings), but you know the value is not right for that particular time. | ||
Frequently, the graphical views ('''View''' menu, ''Graphs'' or ''Select-A-Graph'' screens) are the best way to spot many rogue values and to help you guess by interpolation a more reasonable value. Another possiblity is to examine the relevant values in previous versions of the log files stored in the ''backup'' or ''backup/daily'' subdirectories of where your cumulus executable is stored. You may choose to just edit the rogue value into a reasonable value, or (if you are unable to decide on a better value) to delete that particular entire row in the monthly log. As explained above once the log is corrected, new highs and lows in other logs are easily rectified | Frequently, the graphical views ('''View''' menu, ''Graphs'' or ''Select-A-Graph'' screens) are the best way to spot many rogue values and to help you guess by interpolation a more reasonable value. Another possiblity is to examine the relevant values in previous versions of the log files stored in the ''backup'' or ''backup/daily'' subdirectories of where your cumulus executable is stored. You may choose to just edit the rogue value into a reasonable value, or (if you are unable to decide on a better value) to delete that particular entire row in the monthly log. As explained above once the log is corrected, new highs and lows in other logs are easily rectified. | ||
===Manipulation outside Cumulus=== | ===Manipulation outside Cumulus=== | ||
Any log files for ''previous'' months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The current month log file can also be edited outside Cumulus, but you must stop Cumulus first. | Any log files for ''previous'' months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The current month log file can also be edited outside Cumulus, but as it is being updated frequently you must stop Cumulus first (that explains why there is no edit option within Cumulus!). | ||
'''Tips''' -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the ''not to be touched copy'' a filename of "<Month><Year>log.csv"). | '''Tips''' -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the ''not to be touched copy'' a filename of "<Month><Year>log.csv"). | ||
Edit the original file using an editor that treats all fields as text [either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date) types]. | Edit the original file using an editor that treats all fields as text [use either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date and time) types]. [[File:Open office (editing cumulus log files).png | left]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected (as normal default for this software). Best not to use 'Microsoft Excel' as that normally has opposite default, and may change without any warning the format of all dates so Cumulus can no longer recognise the first two fields: | ||
'''Important Rules''' | '''Important Rules''' | ||
*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept | *Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end). | ||
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within | * The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files''). | ||
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]]. | * Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]]. | ||
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points. | *See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points. | ||
*Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360. | *Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360. | ||
*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or | *Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can. | ||
* Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers. | * Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers. | ||
| Line 112: | Line 120: | ||
The table is split by Cumulus version for all changes since version 1.8.5; it shows: | The table is split by Cumulus version for all changes since version 1.8.5; it shows: | ||
*The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column. | *The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column. | ||
*The second column shows an example, | *The second column shows an example, and the values shown are sometimes integers and sometimes floating point numbers, and this depends on a number of factors, including your station type. If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures, and floating point for the others (add a ".0" if necessary). Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero. | ||
*The third column describes the observation often with a link to where there is more information. | *The third column describes the observation often with a link to where there is more information. | ||
===Version 1.8.5:=== | ===Version 1.8.5:=== | ||
| Line 142: | Line 150: | ||
|- | |- | ||
| 05(F) | | 05(F) | ||
|13 | |13.5 | ||
| Cumulus 'Average' wind speed (See [[#Cumulus_Wind_Speed_Terminology]]) | | Cumulus 'Average' wind speed (See [[#Cumulus_Wind_Speed_Terminology]]) | ||
|- | |- | ||
| 06(G) | | 06(G) | ||
|20 | |20.3 | ||
| Cumulus 'Gust' wind speed | | Cumulus 'Gust' wind speed | ||
|- | |- | ||
| Line 159: | Line 167: | ||
| 09(J) | | 09(J) | ||
|5.4 | |5.4 | ||
| Total rainfall today so far (i.e. resets to zero at rollover) | | Total rainfall today so far (i.e. resets to zero at daily rollover) | ||
|- | |- | ||
| 10(K) | | 10(K) | ||
| Line 167: | Line 175: | ||
| 11(L) | | 11(L) | ||
|215.2 | |215.2 | ||
| Total rainfall counter (content depends on weather station type, might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]) | | Total rainfall counter (content depends on weather station type, might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]. This figure is not intended to be useful in itself, it may vary for no apparent reason. It is used internally by Cumulus.) | ||
|- | |- | ||
| 12(M) | | 12(M) | ||
| Line 178: | Line 186: | ||
|- | |- | ||
| 14(O) | | 14(O) | ||
|17 | |17.6 | ||
| Cumulus 'Latest' gust (See [[#Cumulus_Wind_Speed_Terminology]]) | | Cumulus 'Latest' gust (See [[#Cumulus_Wind_Speed_Terminology]]) | ||
|- | |- | ||
| Line 197: | Line 205: | ||
| Solar Radiation (only valid if solar sensor on weather station) | | Solar Radiation (only valid if solar sensor on weather station) | ||
|} | |} | ||
===Added in 1.9.1=== | ===Added in 1.9.1=== | ||
{| border="1" class="wikitable" | {| border="1" class="wikitable" | ||
| Line 218: | Line 227: | ||
| 22(W) | | 22(W) | ||
|663 | |663 | ||
| Current theoretical max solar radiation ( | | Current theoretical max solar radiation (see [[FAQ#What_is_this_.22Solar_Max.22_value_which_Cumulus_is_displaying.3F | the FAQ]]) | ||
|- | |- | ||
| 23(X) | | 23(X) | ||
| Line 224: | Line 233: | ||
| Hours of sunshine so far today (only valid if solar sensor on weather station) | | Hours of sunshine so far today (only valid if solar sensor on weather station) | ||
|} | |} | ||
===Added in 1.9.2=== | ===Added in 1.9.2=== | ||
{| border="1" class="wikitable" | {| border="1" class="wikitable" | ||
