Monthly log files: Difference between revisions

This article is about the log file that contains the values at whatever interval you have set for Cumulus to keep detailed logging. The main files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called <tt>Aug09log.txt</tt>); an entry is made to the current month's file every ''ten'' (default value; you can change this on the station settings screen) minutes, recording the current sensor values. The file is in comma-separated format with one line per entry.

 

Times in these files are in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time).  All data is logged in the units which which have been selected by the user on the station configuration screen. Note that 'comma-separated' does not necessarily mean that a comma is used to separate the values! If your PC uses a semicolon for the list separator, that will be used in these files. Similarly, decimals use the Windows decimal separator.  

If no logs seen, see  [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]]. Also note, this '''Data logs''' viewer is only for viewing monthly log files (not any other files in the same ''data'' directory), the column headings for standard monthly log files will always be shown at the top of the screen.

The current monthly log can be updated at your usual website update interval using ''<currentlogfile>'' on the '''Files''' tab of the ''Internet Settings'' screen off the '''Configuration''' menu (or by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''.  It is recommended that your remote file is local so the updating can use copy and you do not have FTP ticked to avoid conflict with logging - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for a suggested method.

 

A script can then read the file into a database (see [[ImportCumulusFile]]), or an array, for further processing.

 

CAUTION: Be aware if you tick FTP to upload this file to a remote server, there can be conflict between the thread that Cumulus uses to run FTP and the independent thread that Cumulus uses to log the latest observations into this file.  There is a danger that a FTP in progress still has exclusive access and the logging therefore fails e.g. if the interval between FTP is too small or, it is not near the start of the month so, the file is big; see [[FAQ#I_am_getting_I.2FO_error_32_or_I.2FO_error_103]]. An alterative approach is to upload this file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus.

* You can't edit the current log file while Cumulus is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors.

* You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand.

* Text editors designed for programmers will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM).

* If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected.  Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ...").

* If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer).

* Ensure no blank lines, and if you have added in the column headings then remove that heading line when saving ready for Cumulus.

** update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. See [[Alltimelog.txt]] for current and previous values.

** update (if the created/edited monthly log is within the current year) the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.

** update (if the created/edited monthly log is for the current month) the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values.

** update (from version 1.9.3) the highs and lows in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows.

** see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These  figures are not stored anywhere, nor available as webtags).

** use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu;  This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created).  Click the ''Help'' button for more information.  In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below).

** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors.  Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated.  Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report.

Frequently, the graphical views ('''View''' menu, ''Graphs'' or ''Select-A-Graph'' screens) are the best way to spot many rogue values and to help you guess by interpolation a more reasonable value. Another possiblity is to examine the relevant values in previous versions of the log files stored in the ''backup'' or ''backup/daily'' subdirectories of where your cumulus executable is stored. You may choose to just edit the rogue value into a reasonable value, or (if you are unable to decide on a better value) to delete that particular entire row in the monthly log. As explained above once the log is corrected, new highs and lows in other logs are easily rectified.

Any log files for ''previous'' months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The current month log file can also be edited outside Cumulus, but as it is being updated frequently you must stop Cumulus first (that explains why there is no edit option within Cumulus!).

Edit the original file using an editor that treats all fields as text [use either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date and time) types]. [[File:Open office (editing cumulus log files).png | left]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected (as normal default for this software). Best not to use 'Microsoft Excel' as that normally has opposite default, and may change without any warning the format of all dates so Cumulus can no longer recognise the first two fields:

*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end).

* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').

*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet).  In other words put in an obviously strange number as Cumulus does not have a defined null!  This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can.

*The second column shows an example, and the values shown are sometimes integers and sometimes floating point numbers, and this depends on a number of factors, including your station type. If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures, and floating point for the others (add a ".0" if necessary).  Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero.

|13.5

|20.3

| Total rainfall today so far (i.e. resets to zero at daily rollover)

| Total rainfall counter (content depends on weather station type, might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]. This figure is not intended to be useful in itself, it may vary for no apparent reason. It is used internally by Cumulus.)

|17.6

| Current theoretical max solar radiation (see [[FAQ#What_is_this_.22Solar_Max.22_value_which_Cumulus_is_displaying.3F | the FAQ]])