5,838
edits
Calculate Missing Values: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
→Missing fields (or missing lines) in dayfile.txt
mNo edit summary |
|||
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
<div style="background: LemonChiffon;padding:5px; margin:2px;"> | |||
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest! | |||
<big>Mark Crossley has now written a utility (download at https://github.com/cumulusmx/CreateMissing) that can update dayfile.txt.</big> | |||
</div> | |||
=Introduction= | =Introduction= | ||
| Line 6: | Line 14: | ||
For these, and other reasons, this page explains various ways in which you can edit log files, (and for MX users, database tables) so all lines contain all the fields that can be | For these, and other reasons, this page explains various ways in which you can edit log files, (and for MX users, database tables) so all lines contain all the fields that can be calculated for the weather station you are using. | ||
[[Category:Log Files]] | [[Category:Log Files]] | ||
| Line 29: | Line 37: | ||
# Current air pressure (absolute or sea-level) | # Current air pressure (absolute or sea-level) | ||
By default [[Cumulus. | By default [[Cumulus.ini_(Cumulus_1)#Read-only_parameters_in_the_Station_section|see here]], Cumulus will stop processing any readings from a weather station unless the above 4 source values are being supplied and reveal they are being updated (failure is set is after a total of 6 unsuccessful consecutive attempts to read each of these). | ||
Cumulus also expects that your weather station can provide: | Cumulus also expects that your weather station can provide: | ||
| Line 45: | Line 53: | ||
* Air pollution measurement | * Air pollution measurement | ||
There are some configuration settings [[Cumulus. | There are some configuration settings [[Cumulus.ini_(Cumulus_1)#Read-Write_parameters_in_the_Station_section|here]] where you can decide whether to use a weather station supplied dewpoint temperature and whether to use a weather station supplied wind chill temperature. | ||
===How Cumulus might transform a Source Value=== | ===How Cumulus might transform a Source Value=== | ||
| Line 51: | Line 59: | ||
*For some weather data processed by Cumulus the value sent by the weather station and that used by Cumulus will be different: | *For some weather data processed by Cumulus the value sent by the weather station and that used by Cumulus will be different: | ||
** This can be because the units used by the weather station, and the units required by the Cumulus user are different (and Cumulus allows the user to select the units in which values will be stored) | ** This can be because the units used by the weather station, and the units required by the Cumulus user are different (and Cumulus allows the user to select the units in which values will be stored) | ||
** This can be because the user has entered some calibration settings. The exact settings available vary for different weather data, but in some cases a second order multiplier is available (the square of the input value is multiplied by the second order multiplier), a first order multiplier is available (the input value is multiplied by the first order multiplier), and an offset is available (added to the input value). If you want to understand how this works, please see [[Cumulus. | ** This can be because the user has entered some calibration settings. The exact settings available vary for different weather data, but in some cases a second order multiplier is available (the square of the input value is multiplied by the second order multiplier), a first order multiplier is available (the input value is multiplied by the first order multiplier), and an offset is available (added to the input value). If you want to understand how this works, please see [[Cumulus.ini_(Cumulus_1)#Offsets]] for the details | ||
**A weather station might use a rainfall counter, Cumulus converts it to a rainfall rate (based on difference between two counter readings scaled up to represent amount per hour) | **A weather station might use a rainfall counter, Cumulus converts it to a rainfall rate (based on difference between two counter readings scaled up to represent amount per hour) | ||
| Line 148: | Line 156: | ||
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records. | However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records. | ||
=Missing fields(or missing lines) in dayfile.txt= | =Missing fields (or missing lines) in dayfile.txt= | ||
'''Please see note at top of page, for a utility that works with MX log files, and updates dayfile.txt''' | |||
==Missing or Corrupted lines== | ==Missing or Corrupted lines== | ||
| Line 194: | Line 204: | ||
In the '''Edit''' menu, select '''Dayfile.txt''' screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines. If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor. | In the '''Edit''' menu, select '''Dayfile.txt''' screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines. If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor. | ||
For the daily summary log, the original Cumulus software includes functionality to '''Create Missing''' in its daily summary log editor, which | For the daily summary log, the original Cumulus software includes functionality to '''Create Missing''' in its daily summary log editor, which was described earlier. This functionality creates an entire missing log line and will insert highs and lows whether they are extremes of source values, or extremes of derived values. | ||
This functionality is not designed to work out individual missing extreme figures, only to insert complete missing lines. However, we can workaround that constraint: | |||
'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt''' | '''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt''' | ||
There are two common reasons for needing to use this, Cumulus 1 only, workaround: | There are two common reasons for needing to use this, Cumulus 1 only, workaround: | ||
# To add extra fields to | # To add extra derived fields now calculated by Cumulus, to past lines created by an earlier version of Cumulus that did not calculate those derived fields at the time | ||
# If you are importing into Cumulus records from other software, and that other software does not provide derived extremes (apparent temperature, heat index, rain rate, wind run) | # If you are importing into Cumulus records from other software, and that other software does not provide derived extremes (e.g. dew point, apparent temperature, heat index, rain rate, wind run) so your insert in the daily summary log file has only populated a sub-set of the fields. | ||
The steps you need to take, to implement this Cumulus 1 only workaround: | The steps you need to take, to implement this Cumulus 1 only workaround: | ||
| Line 229: | Line 241: | ||
===Correcting individual fields in multiple lines of Daily Summary Log in Cumulus MX=== | ===Correcting individual fields in multiple lines of Daily Summary Log in Cumulus MX=== | ||
'''This section was written before Mark Crossley released the utility that works with MX log files, and updates dayfile.txt, that can be downloaded from https://github.com/cumulusmx/CreateMissing.''' | |||
Without that utility, the only method, involves using database functionality and multiple steps: | |||
#The first step depends on whether you use the option in MX to create and update database tables... | #The first step depends on whether you use the option in MX to create and update database tables... | ||
#*...If you don't use that option yet, use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table, | #*...If you don't use that option yet, use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table, | ||
