5,838
edits
Standard log files: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
→Importing pre-Cumulus data: add links
m (→Introduction) |
m (→Importing pre-Cumulus data: add links) |
||
| Line 116: | Line 116: | ||
If you were recording weather data manually, or with different weather station software, you might create files, using the standard data log files format, holding that pre-Cumulus data. Cumulus can treat these files just like any where it created the data log entries, and therefore you can apply any of the techniques explained in [[Calculate Missing Values]] and [[Correcting Extremes]]. | If you were recording weather data manually, or with different weather station software, you might create files, using the standard data log files format, holding that pre-Cumulus data. Cumulus can treat these files just like any where it created the data log entries, and therefore you can apply any of the techniques explained in [[Calculate Missing Values]] and [[Correcting Extremes]]. | ||
In the configuration file, Cumulus.ini, there is a parameter '''StartDate=''', this is assigned a value when you first start using Cumulus of the current date, in the format specified by the '''local''' you are using. | |||
* The original Cumulus 1 software takes no notice of this date, other than to use it to populate the <#recordsbegandate> see [[Webtags#Records]], so it is quite happy to process standard data logs with lines with earlier dates. | |||
* The '''ExportMySQL.exe''' which was written by Steve Loft and included with the beta releases of MX, and subsequent early MX release distributions from Mark Crossley, also ignores the date in the configuration file. | |||
* All releases containing '''CumulusMX.exe''', will ignore any data log lines dated before the date in the configuration file. | |||
* The other Mark Crossley utility packages linked from [[Software#Other_utilities|Software page utilities section]] will also generally ignore any data log lines dated before the date in the configuration file. | |||
** [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] will use the earliest date in [[dayfile.txt]] if that is earlier than the date in the configuration file. | |||
** '''ExportToMySQL.exe''' (not currently documented in this Wiki) will normally ignore any data log lines dated before the date in the configuration file. | |||
Don't worry if you are using MX and your pre-cumulus data does not have all the fields listed at [[#List_of_fields_in_the_file]]. [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] can be used to calculate (at lease some of, maybe all of) those missing fields. That utility will also create [[dayfile.txt]] entries for the days in the pre-Cumulus data. | |||
If you are not using MX, you may still be able to populate missing fields (see [[Calculate_Missing_Values|calculate missing values]] page), and create lines in dayfile.txt (see [[Amending dayfile]] page). | |||
===Potential issues when trying to match formats=== | ===Potential issues when trying to match formats=== | ||
'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[#List_of_fields_in_the_file|List of fields in the file]] below with the fields you have available in your source. | |||
'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[ | |||
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number). | * Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number). | ||
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings'). | * Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings'). | ||
