Amending dayfile

This text was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!

Mark Crossley has now written a utility (download at https://github.com/cumulusmx/CreateMissing) that can update dayfile.txt.

Safely editing the daily summary log

At the end of each meteorological day, Cumulus uses information held in today.ini to write a new line into dayfile.txt, before it resets today.ini ready for the new day. As discussed in Correcting_Extremes, it is possible for rogue values to be read from a weather station and propagate into various log files, so this article will help you make any necessary corrections.

The fields contained in the daily summary log are listed in the Dayfile.txt#List_of_Fields. That page contains a lot of information, and to avoid confusion information about editing this log has been moved here. Both Cumulus 1 and 3 (2 did not) now provide editors where you can see what is in your dayfile.txt log, and if you click on a particular line you can edit the fields in that line, or indeed delete that line. The functionality available to you, depends on the flavour of Cumulus you are running (and which release of that flavour you have installed).

The most common problems with dayfile.txt are:

• (MX only) inconsistencies between lines i.e. one of following changes from one line to another
  • either the date separator (character or characters between day of month, month, and year parts)
  • or the decimal point or decimal comma (that separates the integer and decimal parts of real numbers)
• people see Cumulus giving error messages saying there is an error in a particular line (Cumulus only reports first error, there may be more)
• plots only show data for a subset of the lines in the file, because different lines have different formats for the dates
• a Cumulus user (or an electrical fault) stops or starts the software close to rollover time, a Cumulus user upgrades the software close to rollover time, or another process interferes with Cumulus causing it to twice copy information from today.ini to dayfile.txt resulting in a duplicate line with same date (but incorrect content in one of the lines)

All errors generally relate to

• Either using the same file, but moving between computers, without checking the locale settings
• Or a mistake in the way a Cumulus user has edited the file, perhaps because of editing the file using Microsoft Excel (and formatting the date column as a date not as text)!

Validation by in-built editors

Both the legacy editor and the MX editor will ensure that the correct number of fields is stored (as defined at the release where you do editing) (one common error in an external editor is to accidentally add/delete a field).

The legacy editor does also validate individual fields for appropriate content (integer, real number, time-stamp).

Unfortunately, the editor provided with MX does not validate any fields. Thus you can use the wrong separator (between date elements, between hour and minute for time-stamps, between integer and decimal parts in any real number) in the MX editor and the incorrect line will be accepted.

Correcting date separator errors

The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date. Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".

MX is fussy, the same separator must be used in every date in the file. An error will be reported in diagnostics, and the historic graphs will stop working if the separator changes from one line to another.

1. So look at the last line in the file, that is the character MX expects
2. If any earlier line, or lines, uses a different character, then that line, or lines, must be edited, it is best to use an editor designed for programmers
   • If the last line has hyphen "-", but some earlier lines use slash "/", it is simple to do an edit
     • Use "Replace All" option in your chosen editor, to find "/", and replace with "-". This works because "/" does not appear anywhere else in the file
   • If the last line has "/", but some earlier lines have "." or "-" as separator, then the correction is more complex, because you don't want to edit any minus signs or decimal points in any other fields
     • See if your editor accepts "Regular Expressions", then you can look for two matching separators in same field, see this forum post.

See this forum thread for a fuller discussion.

Importing data not recorded by Cumulus

You might have been using your weather station with some other weather software before you installed Cumulus. If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus dayfile.txt file using a script or spreadsheet package. All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in Dayfile.txt#List_of_Fields. There is more guidance later on this page about the rules you must obey for this file.

If you have imported the data from the other weather software into the Standard_log_files format, then in the Cumulus 1 editor, Create missing can insert the new rows for those days previously missing in dayfile.txt, as explained below.

Using the Cumulus 1 editing feature

This section applies to Cumulus 1.x.y only. The last command in Edit menu is dayfile.txt.

This is how you view the dayfile.txt from within Cumulus. Click the Help button for detailed instructions. Cumulus Help is concise but comprehensive.

It is a text editor, and it works best when at full screen:

• correct individual values by over-typing new values over those currently displayed,
• you can use insert key to add one or more missing rows (complete days) manually typing in values for all fields,
• use delete key to remove an entire day (e.g. if you get a 'duplicate' error message) after ensuring all fields are correct in the line that will remain,
• use Create missing button to insert missing rows (complete days) by reading from Standard_log_files and automatically calculating the best approximations for each field for those missing days.

Create Missing

The Cumulus 1 editor provides a "Create Missing" option where it will, for any dates for which a line does not exist, create a line if it can from reading the detailed log file to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation for each dayfile.txt field, storing the time from the relevant other log file in any time-stamp field in dayfile.txt. If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged. For Create missing a list of inserted records is produced in dayfileeditlog.txt. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a work around as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the Standard_log_files if the whole day (a line starting with that date) is missing in dayfile.txt. Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps.

Dealing with errors identified by the software

If there is an error in dayfile.txt, then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus 1 will illuminate its Error light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.

If you do have a 'duplicate' error, you need to decide which row to delete, and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.

Using the Cumulus MX editing functionality

• Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor
• Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor
• From MX release 3.9.2 - b3097 historic charts are added to admin interface
  • If you use the in-built editor to update dayfile.txt, your change will be reflected in the historic graphs
  • If you edit dayfile.txt outside Cumulus MX, historic graphs will not be updated until Cumulus MX is restarted

There is a dayfile.txt editor within the admin interface to edit this log file. In the Edit menu, choose the Dayfile option.

If lines in your daily summary log were created by a variety of Cumulus 1 releases (so some have less than 46 fields), you need to use Cumulus MX version 3.7.0, or later, to be sure that the provided editor will cope. The code was actually amended to be able to read lines with fewer fields at version 3.6.0, when 4 fields were added for feels like, so the total number of fields became 50.

It is recommended that nobody uses any 3.6.x version (see Updating_MX_to_new_version#If_using_a_3.5.x_release. (For historical interest only, Emergency Version 3.6.12 changed the number of fields to 54 in error)

From formal release 3.7.0, MX's dayfile.txt reverted to 52 fields. The extras are Canadian Humidity Index (Humidex). At the time of typing this, it has been said this file structure will not be changed again.

Whatever MX release you use, any line that is edited (even if it originally had fewer or more fields) will be saved with the same number of fields that release uses when it creates a line from the contents of today.ini.

Editing inside MX

The editor in MX can be found in the administrative interface by selecting Data Logs menu and Dayfile page. The Dayfile viewer/editor will display up to 10 lines at a time. When the page is first loaded, the oldest lines will be read from the file by the Cumulus MX engine, and via an application programming interface' (api) transferred to the web page where the lines are displayed using some software called datatables. That software generates a navigation section where you can navigate to First, Previous, Next, and Last, with (for a longer file) up to 6 page numbers (each containing up to 10 lines) that you can select directly.

The editor does not provide any way of inserting new lines into dayfile.txt (so you cannot correct an error when MX end of day failed and a line was not created), nor is there any way of changing the dates used by lines in the file (a very common problem reported by Cumulus users is that MX is giving them problems because not all lines in this file use the same format for the date, but this editor cannot resolve that).

There is a Refresh button that sends a application programming interface instruction asking the MX engine to resend the lines on the currently selected page via the api.

When you select a line in the provided editor, both Edit and Delete buttons are enabled.

Pick Edit, click that, and an editing dialog pops up (MX uses altEditor software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.

There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions:

• some fields can only accept integers, other expect decimals,
• and some fields can accept negatives, others don't accept signed numbers
• some fields have a minimum and/or maximum acceptable value

As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error. There was a third-party proposal to add simple validation into a replacement editor, that would hav

Pick Delete, click that, and a simple dialog pops up (MX uses altEditor software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted. Despite the MX engine getting a copy of the fields that are to be deleted, it does not validate the fields, just the line number. The button labelled 'Close' or 'Cancel' does the same effect as clicking the "x" in the top right corner, it prevents the api sending any deletion message back to the MX engine.

Important Rules when editing dayfile.txt

If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it.

General Editing Rules

1. Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
2. Take another copy and use that for your editing, don't edit the actual file being used by the software.
   • This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
   • It also means that you can go back to the last working copy, you can't upset your "revert" copy.
3. The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
   • It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used.
     • These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
   • You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
     • If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
   • Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
4. Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
5. All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
6. Make sure that any editing does not create any blank lines in the file. Cumulus assumes an empty line means end of processing.
7. Don't add a header line to the file, Cumulus expects all lines to be data lines.

File specific Editing Rules

1. The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
2. All rows must start with date and include at least 14 further fields in correct sequence.
3. The (meteorological) date format uses two digits for the year.
   • This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed not to recognise special field types).
   • For spreadsheet tools (e.g. Calc in Libre Office, or Microsoft's Excel) avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.
4. Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile.
5. The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
   • Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.
   • Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
   • If you move your software (any flavour) to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
6. Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
   • If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
7. Rows can vary in length but only by missing off fields at the end. The minimum number of fields after the date is 14, the maximum varies between different versions.
8. Each field has a pre-defined format, and the same format must always be used in that field position.
9. No fields will accept letters.
   • Some fields (e.g. bearings, solar, humidity) are integers (see #List_of_fields_in_the_file) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
   • Most value fields are in real number format using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
10. Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
11. when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a lowest or highest value and that must be paired with a time-stamp in the next field.
    • Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
12. Time stamp fields must always be in format HH:mm i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
    • • Be aware you will have problems if you, or your editing software, add seconds.
    • If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
    • Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
13. Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
    • Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps.
    • If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).
    • Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
14. Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
    • Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
15. The row terminator for Windows is CR LF (a combination of Carriage Return represented in many character set encodings by the binary equivalent of decimal 13), and Line Feed represented in many character set encodings by the binary equivalent of decimal 10), ensure any external editor does not change the two character terminator into a single character. Similar rules apply for single character terminators used by other operating systems (Linux uses LF only, Mac uses CR only), don't let any editor you use change a single character terminator into its double character terminator.
    • Problems with terminating characters are intercepted by operating system, before it reaches the software, but may still stop the software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.