5,838
edits
Standard log files: Difference between revisions
m
no edit summary
mNo edit summary |
|||
| Line 34: | Line 34: | ||
For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from [[Software#Resources]] page in this Wiki. | For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from [[Software#Resources]] page in this Wiki. | ||
= Accessing the | = Accessing the Standard Log File = | ||
== Access to current log by other processes == | == Access to current log by other processes == | ||
| Line 52: | Line 52: | ||
The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. | The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. | ||
Select the required | Select the required month for which to display the standard log, click '''Open''' and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen. Cumulus 1 does not provide an editing facility for this file. | ||
=== Problems === | === Problems === | ||
| Line 61: | Line 61: | ||
=== Editing for Cumulus 1 users === | === Editing for Cumulus 1 users === | ||
Cumulus 1 allows you to view any | Cumulus 1 allows you to view any standard log file, but not to edit the contents. Consequently for Cumulus 1, you need to edit standard log files outside Cumulus 1, subject to the following rules: | ||
* You can't edit the current log file while Cumulus 1 is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors. | * You can't edit the current log file while Cumulus 1 is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors. | ||
* You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand. | * You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand. | ||
| Line 74: | Line 74: | ||
In the admin interface, there is a "Data Logs" tab, from it select ''Data logs''. | In the admin interface, there is a "Data Logs" tab, from it select ''Data logs''. | ||
=== The viewing and editing web page === | |||
*On this screen,you see a box for selecting the log you want to display. The default month and year (shown on loading the web page) is taken from yesterday's date. No log is loaded at start up. You can '''either''' type in a period (one or two digit representing month, then hyphen, then four digits representing year) '''or''' select year then month in the data picker (this is produced by the bootstrap software MX uses, it is not the date picker provided by your browser), when you click on the month the box is updated with selected period. | *On this screen,you see a box for selecting the log you want to display. The default month and year (shown on loading the web page) is taken from yesterday's date. No log is loaded at start up. You can '''either''' type in a period (one or two digit representing month, then hyphen, then four digits representing year) '''or''' select year then month in the data picker (this is produced by the bootstrap software MX uses, it is not the date picker provided by your browser), when you click on the month the box is updated with selected period. | ||
* Next to it is a "Load" button. Press it and the Cumulus MX engine will generate an application programming interface (api) table that will appear on the web page using datatables software. There is a "Refresh" button, for when you want that api to refresh the web page (without reloading the whole page) | * Next to it is a "Load" button. Press it and the Cumulus MX engine will generate an application programming interface (api) table that will appear on the web page using datatables software. There is a "Refresh" button, for when you want that api to refresh the web page (without reloading the whole page). | ||
When you click on any single line, two extra buttons (for editing that line and for deleting that line) are enabled. The Delete a line button brings up what the | ==== Pagination ==== | ||
The table generated is paginated, it shows ten lines at a time, and there are arrow icons to move to next, and back to previous page. | |||
==== Deleting a line ==== | |||
When you click on any single line, two extra buttons (for editing that line and for deleting that line) are enabled. The '''Delete a line''' button brings up what the editing software calls a modal, a pop up dialog where you see the fields on that row listed vertically and have two more buttons (Cancel and Delete) with obvious consequences. | |||
==== How to edit a line ==== | |||
How to edit the log using this functionality has changed already for different MX versions and is planned to change again, so I am not describing it further for now. | How to edit the log using this functionality has changed already for different MX versions and is planned to change again, so I am not describing it further for now. | ||
| Line 90: | Line 100: | ||
=== Uploading current log to your web site === | === Uploading current log to your web site === | ||
The current | The current standard log can be updated at your usual web site update interval using ''<currentlogfile>'' as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced. | ||
For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column. | For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column. | ||
| Line 123: | Line 133: | ||
*HOW TO USE CORRECTED INFORMATION | *HOW TO USE CORRECTED INFORMATION | ||
** (In cumulus 1 only) see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the monthly logs and the daily summary log. | ** (In cumulus 1 only) see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the monthly logs and the daily summary log. | ||
** (In cumulus 1 only) use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[ | ** (In cumulus 1 only) use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below). | ||
** obtain (in Cumulus 1 by [[ | ** obtain (in Cumulus 1 by [[Standard_log_files#Viewing_standard_log_files | viewing the created/edited log file]]) figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]]. | ||
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. | ** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. | ||
***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. | ***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. | ||
| Line 181: | Line 191: | ||
Given that monthly log files are used as input for '''updating the all-time, month-by-month alltime (from version 1.9.3), and this year record extremes''', for '''creating missing dayfile.txt entries''', and for '''creating NOAA style reports''', you may have some observations from your weather station recorded manually or electronically for a period before you first starting using Cumulus and want to create new monthly logs to feed into Cumulus. | Given that monthly log files are used as input for '''updating the all-time, month-by-month alltime (from version 1.9.3), and this year record extremes''', for '''creating missing dayfile.txt entries''', and for '''creating NOAA style reports''', you may have some observations from your weather station recorded manually or electronically for a period before you first starting using Cumulus and want to create new monthly logs to feed into Cumulus. | ||
'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[ | '''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[Standard_log_files#List_of_fields_in_the_file|List of fields in the file]] below with the fields you have available in your source. | ||
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to | * Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number). | ||
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings'). | * Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings'). | ||
* Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions). | * Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions). | ||
* Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]] | * Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]] | ||
* Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[ | * Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures. | ||
* Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated from the [[ | * Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated from the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] if unavailable from your station. The daily total in field 9 is since rollover time, and available weather station outputs may need some processing in your spreadsheet to derive this. Some stations output a count of total rainfall, others the total annual rainfall, either of these may be used for field 11. Rainfall processing issues are frequently discussed in the Support Forum, so you may find the solution there or ask there for assistance. | ||
'''The date that Cumulus first started tracking all-time records''' (<#recordsbegandate> see [[Webtags#Records]]), '''does ''not'' need to be updated for Cumulus 1''' to recognise earlier monthly log files in its ''data'' subfolder and to use any monthly logs found in the various '''View''' and '''Edit''' options available from the Cumulus main screen. | '''The date that Cumulus first started tracking all-time records''' (<#recordsbegandate> see [[Webtags#Records]]), '''does ''not'' need to be updated for Cumulus 1''' to recognise earlier monthly log files in its ''data'' subfolder and to use any monthly logs found in the various '''View''' and '''Edit''' options available from the Cumulus main screen. | ||
| Line 222: | Line 232: | ||
==List of fields in the file== | ==List of fields in the file== | ||
The table is split by Cumulus version for all changes since version 1.8.5; it shows: | The table is split by Cumulus version for all changes since version 1.8.5; it shows: | ||
#The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column. | #The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column. | ||
| Line 229: | Line 240: | ||
#The third column shows an example, and the values shown are sometimes integers and sometimes floating point numbers, and this depends on a number of factors, including your station type. If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures, and floating point for the others (add a ".0" if necessary). Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero. | #The third column shows an example, and the values shown are sometimes integers and sometimes floating point numbers, and this depends on a number of factors, including your station type. If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures, and floating point for the others (add a ".0" if necessary). Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero. | ||
#The final column describes the observation often with a link to where there is more information. | #The final column describes the observation often with a link to where there is more information. | ||
===Version 1.8.5:=== | ===Version 1.8.5:=== | ||
{| border="1" class="wikitable" | {| border="1" class="wikitable" | ||
|- | |- | ||
| Line 368: | Line 381: | ||
===Added in 1.9.2=== | ===Added in 1.9.2=== | ||
{| border="1" class="wikitable" | {| border="1" class="wikitable" | ||
|- | |- | ||