5,838
edits
Calculate Missing Values: Difference between revisions
m
→Missing fields (or missing lines) in dayfile.txt
m (→Derived spot values: Insert links) |
|||
| (5 intermediate revisions by the same user not shown) | |||
|
<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!
<big>Mark Crossley has now written a utility (download at https://github.com/cumulusmx/CreateMissing) that can update dayfile.txt.</big>
</div>
=Introduction=
For these, and other reasons, this page explains various ways in which you can edit log files, (and for MX users, database tables) so all lines contain all the fields that can be
[[Category:Log Files]]
# Current air pressure (absolute or sea-level)
By default [[Cumulus.
Cumulus also expects that your weather station can provide:
* Air pollution measurement
There are some configuration settings [[Cumulus.
===How Cumulus might transform a Source Value===
*For some weather data processed by Cumulus the value sent by the weather station and that used by Cumulus will be different:
** This can be because the units used by the weather station, and the units required by the Cumulus user are different (and Cumulus allows the user to select the units in which values will be stored)
** This can be because the user has entered some calibration settings. The exact settings available vary for different weather data, but in some cases a second order multiplier is available (the square of the input value is multiplied by the second order multiplier), a first order multiplier is available (the input value is multiplied by the first order multiplier), and an offset is available (added to the input value). If you want to understand how this works, please see [[Cumulus.
**A weather station might use a rainfall counter, Cumulus converts it to a rainfall rate (based on difference between two counter readings scaled up to represent amount per hour)
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.
=Missing fields (or missing lines) in dayfile.txt=
'''Please see note at top of page, for a utility that works with MX log files, and updates dayfile.txt'''
==Missing or Corrupted lines==
If you accidentally corrupt a line in the [[dayfile.txt|daily summary log]], then both the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|original Cumulus software]] and [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX's admin interface]] have editors for viewing and modifying content.
If you have one, or more, dates missing in your dayfile.txt file, then the next question is, '''has the line been deleted by accident?'''
If a line for a particular date was present before, but is now corrupted or missing:
#See if you have a back-up of dayfile.txt with the line present and correct
#*If it is a recent date, then Cumulus makes a backup of dayfile.txt every time it is restarted and after every end-of-day rollover
#*If it is an older date, then maybe you took a back-up onto a separate drive or separate device
#If you have a suitable backup available, take a copy of that file
#Append onto the copy, any dates after when that copy ends, taking the extra lines from the current dayfile.txt
#Rename the current dayfile.txt to say dayfile.old
#Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
#Cumulus will now use the file with all days correct
If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1. That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
#Only use the editor when Cumulus is '''not''' doing catch-up or end-of-day, or any other action that could create an editing conflict.
#With the contents of the file loaded into the editing table, click the '''Create Missing''' button
#Cumulus 1 will now work through every line from start to end
#If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
#It will read from the [[Standard log files]] all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
#*Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)
''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''
'''Another lesson here, is to periodically take a backup, stored away from your Cumulus running environment in case you ever corrupt an old line'''
Although there exists a tentative plan for an executable, to be added to the MX package, that can read the standard data logs and calculate all the fields needed for a
==Correcting individual fields in existing lines==
The "Create Missing" functionality, described in previous sub-section, is designed for when an entire line is missing for a prior date. Often people want to correct an individual field, in all lines within a particular period. This might be because you have suddenly discovered your pressure high and low are wrong by a certain offset, because you used the wrong setting for converting absolute pressure to the sea level equivalent you want to store in the file. Or it might be that you realise that MX stores 52 fields while you have old lines (perhaps from Cumulus 1) with fewer fields.
Neither Cumulus 1 nor MX provide any functionality to make such multi-line editing easy. However, the first possibility can be resolved by opening dayfile.txt in a spreadsheet (carefully ensuring each column is treated as either text (date and time fields) or number (the value fields) and that your field separator is set in the filter conditions to be a column separator. Spreadsheets have a "Paste Special" function that allows you to add/subtract a constant to every cell in a column. The extra fields in MX all store derived values, i.e. they are calculated from source values (see [[Calculate_Missing_Values#Some_definitions|Some definitions]] section earlier. If you have the correct spot derived values for the equivalent source fields in your standard data logs, then there is a work-around that can read standard log and calculate what is required to store in our daily summary file. The next two sub-sections describe how for Cumulus 1 and MX respectively.
===Correcting individual fields in multiple lines of Daily summary Log in Cumulus 1===
In the '''Edit''' menu, select '''Dayfile.txt''' screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines. If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor.
For the daily summary log, the original Cumulus software includes functionality to '''Create Missing''' in its daily summary log editor, which
This functionality is not designed to work out individual missing extreme figures, only to insert complete missing lines. However, we can workaround that constraint:
'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
There are two common reasons for needing to use this, Cumulus 1 only, workaround:
# To add extra derived fields now calculated by Cumulus, to
# If you are importing into Cumulus records from other software, and that other software does not provide derived extremes (e.g. dew point, apparent temperature, heat index, rain rate, wind run)
The steps you need to take, to implement this Cumulus 1 only workaround:
*Cumulus 1 does not have the functionality to retrospectively calculate derived values when it is reading source values in the standard data log line.
===Correcting individual fields in multiple lines of Daily Summary Log in Cumulus MX===
'''This section was written before Mark Crossley released the utility that works with MX log files, and updates dayfile.txt, that can be downloaded from https://github.com/cumulusmx/CreateMissing.'''
▲Although there exists a tentative plan for an executable, to be added to the MX package, that can read the standard data logs and calculate all the fields needed for a line in dayfile.txt, this is only a gleam in the developer's eye! At the moment, the 'Create Missing' functionality and workaround described above is not available to MX users.
#The first step depends on whether you use the option in MX to create and update database tables...
#*...If you don't use that option yet, use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table,
| |||