Standard log files: Difference between revisions

m
Release 3.9.0 field 30 section corrected as change delayed
m (Update category)
m (Release 3.9.0 field 30 section corrected as change delayed)
 
(11 intermediate revisions by the same user not shown)
{{Template:WorkInProgressBannerVersion badge Mx}}{{Template:Version badge Mx1}}[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:MX txt Files]][[Category:Cumulus 1Files_with_Comma_Separated_Values]]
This page applies to all flavours.
 
= Introduction =
 
Cumulus has ability to log data on a periodic basis, see [[Monthly log files]]:
Cumulus has ability to create a number of data logs, most ('''[[Speciallog.txt]]''' , [[Extra_Sensor_Files]], and [[Air_Link_Log.txt]]) are optional, in that settings determine whether they are created or not. However, this page documents the data logs that are not optional (hence termed "standard"). These store the full set of current (read and derived) values at whatever interval you have set for Cumulus to keep detailed logging. The default time interval between adding data logging lines is every 10 minutes, but you can change this:
* This page is about the standard logging file, that is always created, with a new file in each new month
*Cumulus 1: From Configuration menu, select the [[Cumulus_Screenshots#Station|station settings screen]], '''Data log interval''' box on extreme RH side.
* Steve Loft started work, for Cumulus 2 on a replacement for these files using a database, and planned [[Cumulusmx.db]] to hold such data for MX 3.0.0 beta, although his work on this was never completed.
*Cumulus MX: From settings tab, select the '''Station settings''' screen, and find '''Log interval''' after the ''Ecowitt GW1000 Settings'' section.
* Other files ('''[[Speciallog.txt]]''' , [[Extra_Sensor_Files]], and [[Air_Link_Log.txt]]) are optional, in that settings determine whether they are created or not.
 
In fact, in Cumulus MX, the processing of this extra sensor logging file shares a lot of the code with the processing of the standard logging file, as they both are updated at same interval, and they both take values processed following a read from your weather station. The applications programming interface (api) that sends values between the MX engine and the MX admin interface is also identical (with a parameter "Extra" that swaps between true and false depending on the log file involved).
These files are automatically created with names in the form <tt><Month><Year>log.txt</tt> (for example, the file for August 2009 is often called <tt>Aug09log.txt</tt>), but the month can be in another language or a different abbreviation depending on your locale. There is no setting to change this format.
 
The logging file, [[speciallog.txt]], for detailed logging of internal temperature and humidity is not split into separate files for each month. Although this exists in Cumulus 1, it appears this functionality will remain missing from MX.
The standard logging file is in comma-separated format with one line per entry. Being comma separated does not mean the fields in this log are always separated by commas, they might use semi-colon (;) for example.
 
The logging file for [[Air Link Log.txt|air link]] sensors, is only available in MX.
There is an equivalent logging file for [[Extra_Sensor_Files|extra sensor]] values. In fact, in Cumulus MX, the processing of this extra sensor logging file shares a lot of the code with the processing of the standard logging file, as they both are updated at same interval, and they both take values processed following a read from your weather station. The applications programming interface (api) that sends values between the MX engine and the MX admin interface is also identical (with a parameter "Extra" that swaps between true and false depending on the log file involved).
 
=How often is data stored in the standard file?=
There is another logging file, [[speciallog.txt]], for detailed logging of internal temperature and humidity, but it is not split into separate files for each month. Although this exists in Cumulus 1, it appears this functionality will remain missing from MX.
 
These store the full set of current (read and derived) values at whatever interval you have set for Cumulus to keep detailed logging.
Finally, MX development has added another data log, this stores the output from [[Air Link Log.txt|air link]] sensors.
* The interval has to be a fraction of an hour, so the least often storing is every half hour.
* For some weather station types, you MUST match the interval at which the weather station logs data internally
* The default time interval between adding data logging lines is every 10 minutes, but you can change this:
** Legacy Cumulus 1: On main screen, select '''Configuration''' menu, select the [[Cumulus_Screenshots#Station|'''station settings''' screen]], look for '''Data log interval''' box on extreme RH side.
** Cumulus MX: From [[MX Administrative Interface|Interface]], find settings menu, select the '''Station settings''' page, select '''General Settings''', and find '''Log interval''' where there is a drop down that offers the various timings that are permitted.
 
=What name is used for the standard file?=
{{TOCright}}
 
These files are automatically created with names in the form <tt><Month><Year>log.txt</tt>.
 
In UK English locale, the file for August 2009 is called <code>Aug09log.txt</code>.
 
The month can be in another language, or a different abbreviation, depending on your locale.
 
The same naming format must be used for all files,if Cumulus is to be able to find them. For Cumulus MX only, any file lines for dates before the Cumulus start date will be ignored.
 
= File content =
 
All data is logged in the units as selected by the user:
*Cumulus 1: FromOn main screen, select '''Configuration''' menu, select the [[Cumulus_Screenshots#Station|'''station settings''' screen]], '''Units''' box is just below the centre of that screen.
*Cumulus MX: From [[MX Administrative Interface|Interface]], find settings tabmenu, select the '''Station settings''' screenpage, and find '''Units''' section at the start of that screen.
** Advanced settings in MX allow you to vary the number of decimal places for the various stored values
 
Each line has a number of fields separated by whatever symbol is defined as the list separator on your device, typically a comma (,) or a semi-colon (;).
* The various fields are all listed later inafter this articlesection, but they can be of various types:
** Date, this is always the first field, and it always uses a fixed format, ignoring the format defined in your locale.
** Time, this always the second field, and is always in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). However, if you use the feature in MX to upload the equivalent of this file to a database table, the automatic update upload will add ":00" to include seconds.
For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from [[Software#Resources]] page in this Wiki.
 
=List of fields in the file=
= Accessing the Standard Log File =
 
Please see [[Calculate Missing Values]] and [[Correcting Extremes]] for full details on relevant functionality in Cumulus software.
 
== Access to current log by other processes ==
 
As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus 1 applies an exclusive lock, and [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 conflicts] can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while Cumulus is running. Whilst MX does not apply such locks, it would be unwise to externally edit a file that is being frequently updated while MX is running.
Also note these log files do not include a header line, and should not be edited to include it.
 
== Additional ideas for how any Standard Log File can be used within Cumulus ==
 
=== Uploading current log to your web site ===
 
The current standard log can be updated at your usual web site update interval using ''<currentlogfile>'' as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced.
 
#*For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini_(Cumulus_1)#Section:_FTP_Site]]).
#*For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column.
#You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. You will also need to specify a path in the remote column for MX or the 'ExtraRemote[0 to 99]=...' for Cumulus 1. You might of course want to specify a local path, and leave the FTP column unticked (or set to 0 for Cumulus 1). You can then use a separate FTP process to upload this copy.
 
*It is recommended that you do not try to FTP the actual standard log file (instead FTP a copy) because the FTP process is relatively slow, particularly as the file size increases as the month progresses, so there is an increasing chance of conflict with logging the next line of values - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for more information.
*An alternative approach is to upload the standard log file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus.
 
=== Cumulus 1: transferring the contents of the standard log file into a database table ===
 
*A script can be used to read the your copy of the standard log file into either a database table (see [[ImportCumulusFile]]), or into an array, for further processing.
*It is, as advised above, best to work with a copy of the file that Cumulus is actually updating, because there can be a conflict of the locks, Cumulus needs to have exclusive use when it does its normal update.
*But this script can safely be used with older standard log files, or a copy of the latest standard log file.
 
=== Cumulus MX: transferring the contents of the standard log file into a database table ===
 
{{Template:WorkInProgressBanner}}
 
PLEASE UPDATE THIS SECTION - '''ExportMySQL.exe''' HAS BEEN REPLACED BY [https://github.com/cumulusmx/ExportToMySQL ExportToMySQL] - see [https://cumulus.hosiene.co.uk/viewtopic.php?p=154206#p154206 forum] for an example
 
Although the '''ImportCumulusFile''' script can be used with MX too, MX provides 2 other ways to transfer the contents of a Standard Log File into a database table.
 
#The first, uploads ALL past data, using '''ExportMySQL.exe''' found in the top level folder where ''CumulusMX.exe'' is found. It can be run using the same way as you run the main program, but you need to specify "monthly" as the first parameter (this indicates the table name that is to be updated, it does not work for any other table name).
#The second, which only uploads the line most recently added to the standard log file is found in the '''My SQL''' settings section of the admin interface. Find more information at [[Cumulus_MX#MySQL_settings|MySQL settings]]. In this option the default table name is "monthly", but it can be changed.
 
===Additional Functionality in Cumulus===
 
*Additional derived values in original Cumulus software
** You can see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the standard log files and the daily summary log.
** You can use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu.
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''.
***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from values read from the standard log files) for period in question. Finally press ''Save'' button to store the new or amended report.
 
*The last of these is also in MX
**In Cumulus MX, use the '''Reports''' tab, then choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. A date selector allows you to choose the exact report you want, click "Load" button next to it. You can alternatively click another button to generate a missing report, or to regenerate an existing report after any corrections you have made to any standard log file relevant to that report.
 
=Rules to follow if editing a standard data log=
 
==Important Rules for Cumulus 1==
 
*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end).
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]].
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points.
*Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360.
*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can.
* Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers.
 
==Equivalent rules for MX ==
 
*All fields in all lines must contain the same character for separating one field from next and it must be consistent with whatever locale you use when you run MX (it might be set using a parameter to the command running MX, it might be set in Mono, or in Windows control panel).
* Each line must contain all the fields '''defined in the version of MX that you are using''' in correct sequence
* ''Feels like'' was added onto the end of this log file in version 3.5.4. Any line in any log file produced before you started using MX version 3.5.4 or later will end with ''Rain since midnight''. However, the way that MX was modified for version 3.5.4 means that those earlier lines will be modified by the application process interface used to transfer log lines between the MX engine and the admin interface to include the full number of fields per row and to contain a space character for that additional field. Consequently, any external edit also has to add a field separator and a space to every line where the original lacked feel like.
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using a colon as separator, the 24 hour clock and local time (system time).
* If you are editing a log file that was produced by Cumulus 1 and it used a separator other than a colon (:) then the old separator must be replaced by colon.
* For MX the rules for fields containing values are different to those for Cumulus 1, which makes it difficult to successfully edit any pre MX log files. For MX if a field can take a decimal place, it must contain a decimal figure, and it must use the correct decimal separator as defined in locale. If the field only takes integers (see [[#List_of_fields_in_the_file]]) then you cannot edit in any decimal places.
 
 
==Importing pre-Cumulus data==
 
If you were recording weather data manually, or with different weather station software, you might create files, using the standard data log files format, holding that pre-Cumulus data. Cumulus can treat these files just like any where it created the data log entries, and therefore you can apply any of the techniques explained in [[Calculate Missing Values]] and [[Correcting Extremes]].
 
 
The table is split by Cumulus version for all changes since version 1.7.x (Steve Loft did not retain his list of changes in 1.4, his '''changes.txt''' is rather sparse in details for many releases, and the support forum excludes any posts before 2 Jun 2008, so earlier changes are not now documented).
The original Cumulus 1 software takes no notice of the <#recordsbegandate> see [[Webtags#Records]], so it is quite happy to process standard data logs with lines with earlier dates. The same applies to '''ExportMySQL.exe'''. However, by default, '''CumulusMX.exe''' will ignore any data log lines dated before the [[Cumulus.ini_(Cumulus_1)#Section:_Station|start date]] it recorded in the configuration file.
 
===Potential issues when trying to match formats===
 
 
'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[Standard_log_files#List_of_fields_in_the_file|List of fields in the file]] below with the fields you have available in your source.
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number).
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings').
* Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions).
* Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]]
* Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures.
* Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated by following advice in [[Calculate Missing Values]].
 
==== Cumulus Wind Speed Terminology ====
 
'''This section was written for the original Cumulus software'''. There are some differences in how MX reads information from weather stations, particularly those by Davis, but that should not affect content below.
 
How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus:
#Gust - the highest wind speed over a particular period (this period defaults to 10 minutes),
#Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes),
#Latest - the most recent (wind speed or gust speed) measurement.
[[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Davis_stations.3F|FAQ: Wind speeds in Cumulus with Davis stations]], the ''EasyWeather'' differences in [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F|FAQ: Wind speeds in Cumulus with Fine Offset stations]] and [[FAQ#How_do_I_set_up_recording_wind_speeds_in_Cumulus.2C_with_Oregon_Scientific_and_La_Crosse_stations| FAQ: Wind speeds with Oregon Scientific and La Crosse stations]]
 
When Cumulus is running normally, it actually stores (in the fifth item ''after'' the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record (assuming you have a) kept default averaging interval to 10 minutes, and b) have also adjusted the logging interval from the default 15 minutes to be 10 minutes).
 
If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day.
 
Cumulus calculates the monthly/annual/period ''average wind speed'' (as used for 'This ...' screens in '''View''' menu and the NOAA report) by trawling through the relevant standard logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your standard log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval.
 
[[Windrun]] is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger. Again, if Cumulus is not left running the calculated value will be skewed.
 
=List of fields in the file=
 
The table shows:
The table is split by Cumulus version for all changes since version 1.8.5; it shows:
#The field number
#*When this list was first produced, only the original Cumulus software was available and the decision then was to start with date in field zero to be consistent with index used for arrays in programming languages like JavaScript.
#The final column describes the observation often with a link to where there is more information.
 
==Number of fields per line varies by release==
===Version 1.8.5:===
 
The fields that are available in the release you are running are listed in [[Monthlyfileheader.txt]] (found in folder level above the data sub-folder).
 
If you have some early files with fewer fields than the latest files, there are limited possibilities for calculating correct values for those missing fields
 
[[File:Badge v1.png]] For the legacy Cumulus software, the number of fields in the file increased as the software was developed. The Cumulus author, Steve Loft, did not provide any utility for populating fields missing from earlier versions of his software. A third party did produce a [[Calculate_Missing_Values#Using_a_PHP_script_on_your_web_server|PHP script]] that could:
* add Heat Index to lines created prior to 1.7.11
* add Apparent Temperature (not other fields, see below) to lines created before 1.9.1
* add Feels Like temperature (not other fields, see below) to lines created before 3.6.0
The fields that ''cannot'' be calculated by this script are (set them all to zero when editing file manually):
* UV Index
* Solar Radiation
* Evapotranspiration (spot value and annual cumulative total)
* Current theoretical max solar radiation
* Hours of sunshine so far today
* Current Wind bearing
* RG-11 rain today
* Total Rainfall since midnight (this is for 0900/1000 'rollover' users; normally same as rain today for 'midnight rollover' users)
 
[[File:Badge vMx.png]] As MX has developed, further fields have been added to the file at 3.6.0 and 3.6.12:
* Feels Like temperature
* Canadian Humidity Index (Humidex)
The main MX developer, Mark Crossley, has written a [[Calculate_Missing_Values#CreateMissing.exe|Create Missing utility]], this is dependent on having a significantly recent MX release, but it can calculate the following derived values (coping with cases where any of these are missing:
* [[Wind chill|Wind Chill]]
* [[Apparent_temperature|Australian Apparent Temperature]]
* [[Feels Like|Feels Like temperature]]
* [[Heat index|North American Heat Index]]
* [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
* [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
* Rain since midnight)
''Please note the developer does not mention any of this at [https://github.com/cumulusmx/CreateMissing/blob/master/README.md his utility details explanation] so the author of this Wiki update cannot guarantee the above list is correct''
 
 
You will need to check in MX release announcements for any update, at time of typing this, the developer has written a routine within CumulusMX.exe to calculate Evapotranspiration, but that is not in the CreateMissing.exe Utility.
 
===Before Version 1.7.10:===
 
{| border="1" class="wikitable"
|4.8
| [[wind chill]] (content depends on weather station type and Cumulus station settings)
|}
 
===Added in 1.7.11===
 
{| border="1" class="wikitable"
|-
! style="width:60px" | Field #
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 17(Q)
|8.1
| [[Heat index]]
|- }
 
 
===Added in 1.8.4===
 
{| border="1" class="wikitable"
|-
! style="width:60px" | Field #
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 18(R)
|unsigned number
|Unsigned integer
|158
| Current Wind bearing (See 7average wind bearing in field 8)
|}
''For Cumulus MX editing a file created by Cumulus 1'', any of the preceding fields that are not present will be '''defaulted to an empty string''', but there will always be 50 fields (unless you are using MX version before 3.6.0, it which case MX cannot read the file.
|-
| 28(AB)
|signed number (or space)
|13.7
| [[Feels Like]] temperature
 
NOTE: MX adds a space here, if a line (created before 3.6.0 without a valid value in this field) is edited.
|}
 
|}
 
==Example= linesAdded fromin the3.nn?.0 file===
 
This new field makes it easier to work out a time in UTC, and identifies uniquely those lines created during a switch from Daylight Saving Time to Winter time, that would otherwise have duplicate times for an hour at time-zone changeover.
 
The following change was initially planned for release of MX 3.9.0 (b3095), but this section has been amended as the introduction of this new field has been postponed, because of implications of other changes planned at same time (see below table).
 
The developer has said it is still his plan to add this field during a major release in 2022, but he cannot yet say when it will actually be available to public.
 
 
{| border="1" class="wikitable"
|-
! style="width:60px" | Field #
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 30(AD)
|datetime integer
| 1579564800000
| UNIX time-stamp
|}
 
''The proposed "Updates.txt" entry read <pre>
- Major changes to how Cumulus MX handles it's data files
- Dayfile.txt
- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility)
- The decimal point is now fixed at "."
- The list separator is now fixed at ","
- Monthly log files
- The file names are fixed at "yyyyMMlog.txt"
- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility), any lines using "dd/MM/yy" are converted to new separator
- The time field format is now fixed at "HH:mm"
- An extra field is added that is the UTC date-time integer
- The decimal point is now fixed at "."
- The list separator is now fixed at ","
- Extra Data log files
- No change in file names
- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility), any lines using "dd/MM/yy" are converted to new separator
- The time field format is now fixed at "HH:mm"
- An extra field is added that is the UTC date-time integer
- The decimal point is now fixed at "."
- The list separator is now fixed at ","
</pre>''
 
= Accessing the Standard Log File =
 
Please see [[Calculate Missing Values]] and [[Correcting Extremes]] for full details on relevant functionality in Cumulus software.
 
== Access to current log by other processes ==
 
As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus 1 applies an exclusive lock, and [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 conflicts] can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while Cumulus is running. Whilst MX does not apply such locks, it would be unwise to externally edit a file that is being frequently updated while MX is running.
Also note these log files do not include a header line, and should not be edited to include it.
 
== Additional ideas for how any Standard Log File can be used within Cumulus ==
 
=== Uploading current log to your web site ===
 
The current standard log can be updated at your usual web site update interval using ''<currentlogfile>'' as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced.
 
#*For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini_(Cumulus_1)#Section:_FTP_Site]]).
#*For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column.
#You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. You will also need to specify a path in the remote column for MX or the 'ExtraRemote[0 to 99]=...' for Cumulus 1. You might of course want to specify a local path, and leave the FTP column unticked (or set to 0 for Cumulus 1). You can then use a separate FTP process to upload this copy.
 
*It is recommended that you do not try to FTP the actual standard log file (instead FTP a copy) because the FTP process is relatively slow, particularly as the file size increases as the month progresses, so there is an increasing chance of conflict with logging the next line of values - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for more information.
*An alternative approach is to upload the standard log file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus.
 
=== Cumulus 1: transferring the contents of the standard log file into a database table ===
 
*A script can be used to read the your copy of the standard log file into either a database table (see [[ImportCumulusFile]]), or into an array, for further processing.
*It is, as advised above, best to work with a copy of the file that Cumulus is actually updating, because there can be a conflict of the locks, Cumulus needs to have exclusive use when it does its normal update.
*But this script can safely be used with older standard log files, or a copy of the latest standard log file.
 
=== Cumulus MX: transferring the contents of the standard log file into a database table ===
 
{{Template:WorkInProgressBanner}}
 
PLEASE UPDATE THIS SECTION - '''ExportMySQL.exe''' HAS BEEN REPLACED BY [https://github.com/cumulusmx/ExportToMySQL ExportToMySQL] - see [https://cumulus.hosiene.co.uk/viewtopic.php?p=154206#p154206 forum] for an example
 
Although the '''ImportCumulusFile''' script can be used with MX too, MX provides 2 other ways to transfer the contents of a Standard Log File into a database table.
 
#The first, uploads ALL past data, using '''ExportMySQL.exe''' found in the top level folder where ''CumulusMX.exe'' is found. It can be run using the same way as you run the main program, but you need to specify "monthly" as the first parameter (this indicates the table name that is to be updated, it does not work for any other table name).
#The second, which only uploads the line most recently added to the standard log file is found in the '''My SQL''' settings section of the admin interface. Find more information at [[Cumulus_MX#MySQL_settings|MySQL settings]]. In this option the default table name is "monthly", but it can be changed.
 
===Additional Functionality in Cumulus===
 
*Additional derived values in original Cumulus software
** You can see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the standard log files and the daily summary log.
** You can use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu.
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''.
***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from values read from the standard log files) for period in question. Finally press ''Save'' button to store the new or amended report.
 
*The last of these is also in MX
**In Cumulus MX, use the '''Reports''' tab, then choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. A date selector allows you to choose the exact report you want, click "Load" button next to it. You can alternatively click another button to generate a missing report, or to regenerate an existing report after any corrections you have made to any standard log file relevant to that report.
 
=Rules to follow if editing a standard data log=
 
==Important Rules for Cumulus 1==
 
*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end).
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]].
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points.
*Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360.
*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can.
* Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers.
 
==Equivalent rules for MX ==
 
*All fields in all lines must contain the same character for separating one field from next and it must be consistent with whatever locale you use when you run MX (it might be set using a parameter to the command running MX, it might be set in Mono, or in Windows control panel).
* Each line must contain all the fields '''defined in the version of MX that you are using''' in correct sequence
* ''Feels like'' was added onto the end of this log file in version 3.5.4. Any line in any log file produced before you started using MX version 3.5.4 or later will end with ''Rain since midnight''. However, the way that MX was modified for version 3.5.4 means that those earlier lines will be modified by the application process interface used to transfer log lines between the MX engine and the admin interface to include the full number of fields per row and to contain a space character for that additional field. Consequently, any external edit also has to add a field separator and a space to every line where the original lacked feel like.
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using a colon as separator, the 24 hour clock and local time (system time).
* If you are editing a log file that was produced by Cumulus 1 and it used a separator other than a colon (:) then the old separator must be replaced by colon.
* For MX the rules for fields containing values are different to those for Cumulus 1, which makes it difficult to successfully edit any pre MX log files. For MX if a field can take a decimal place, it must contain a decimal figure, and it must use the correct decimal separator as defined in locale. If the field only takes integers (see [[#List_of_fields_in_the_file]]) then you cannot edit in any decimal places.
 
 
==Importing pre-Cumulus data==
 
If you were recording weather data manually, or with different weather station software, you might create files, using the standard data log files format, holding that pre-Cumulus data. Cumulus can treat these files just like any where it created the data log entries, and therefore you can apply any of the techniques explained in [[Calculate Missing Values]] and [[Correcting Extremes]].
 
In the configuration file, Cumulus.ini, there is a parameter '''StartDate=''', this is assigned a value when you first start using Cumulus of the current date, in the format specified by the '''local''' you are using.
* The original Cumulus 1 software takes no notice of this date, other than to use it to populate the <#recordsbegandate> see [[Webtags#Records]], so it is quite happy to process standard data logs with lines with earlier dates.
* The '''ExportMySQL.exe''' which was written by Steve Loft and included with the beta releases of MX, and subsequent early MX release distributions from Mark Crossley, also ignores the date in the configuration file.
* All releases containing '''CumulusMX.exe''', will ignore any data log lines dated before the date in the configuration file.
* The other Mark Crossley utility packages linked from [[Software#Other_utilities|Software page utilities section]] will also generally ignore any data log lines dated before the date in the configuration file.
** [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] will use the earliest date in [[dayfile.txt]] if that is earlier than the date in the configuration file.
** '''ExportToMySQL.exe''' (not currently documented in this Wiki) will normally ignore any data log lines dated before the date in the configuration file.
 
Don't worry if you are using MX and your pre-cumulus data does not have all the fields listed at [[#List_of_fields_in_the_file]]. [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] can be used to calculate (at lease some of, maybe all of) those missing fields. That utility will also create [[dayfile.txt]] entries for the days in the pre-Cumulus data.
 
If you are not using MX, you may still be able to populate missing fields (see [[Calculate_Missing_Values|calculate missing values]] page), and create lines in dayfile.txt (see [[Amending dayfile]] page).
 
===Potential issues when trying to match formats===
 
'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[#List_of_fields_in_the_file|List of fields in the file]] below with the fields you have available in your source.
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number).
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings').
* Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions).
* Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]]
* Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures.
* Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated by following advice in [[Calculate Missing Values]].
 
==== Cumulus Wind Speed Terminology ====
 
'''This section was written for the original Cumulus software'''. There are some differences in how MX reads information from weather stations, particularly those by Davis, but that should not affect content below.
 
How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus:
#Gust - the highest wind speed over a particular period (this period defaults to 10 minutes),
#Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes),
#Latest - the most recent (wind speed or gust speed) measurement.
[[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Davis_stations.3F|FAQ: Wind speeds in Cumulus with Davis stations]], the ''EasyWeather'' differences in [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F|FAQ: Wind speeds in Cumulus with Fine Offset stations]] and [[FAQ#How_do_I_set_up_recording_wind_speeds_in_Cumulus.2C_with_Oregon_Scientific_and_La_Crosse_stations| FAQ: Wind speeds with Oregon Scientific and La Crosse stations]]
 
When Cumulus is running normally, it actually stores (in the fifth item ''after'' the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record (assuming you have a) kept default averaging interval to 10 minutes, and b) have also adjusted the logging interval from the default 15 minutes to be 10 minutes).
 
If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day.
 
Cumulus calculates the monthly/annual/period ''average wind speed'' (as used for 'This ...' screens in '''View''' menu and the NOAA report) by trawling through the relevant standard logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your standard log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval.
 
[[Windrun]] is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger. Again, if Cumulus is not left running the calculated value will be skewed.
 
 
=Example lines from the file=
 
Note that your field delimiters may be different, and your date delimiter too.
5,838

edits