Amending dayfile: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
8,415 bytes added ,  05:55, 22 August 2022
m
 
(7 intermediate revisions by the same user not shown)
Line 1: Line 1:
Cumulus uses a daily summary log file, the fields in that file are listed at [[Dayfile.txt#List_of_Fields]].  The information about amending the file that was originally on the same page has been moved to this page.
Cumulus uses a daily summary log file, the fields in that file are listed at [[Dayfile.txt#List_of_Fields]].  The information about amending the file that was originally on the same page has been moved to this page. {{Version badge 1}}When the text was first created (on the other page) it was for the (legacy) Cumulus 1 software. {{Template:Version badge Mx}}As MX was developed, the text here has been amended to keep up, it currently applies up to release 3.12.0.
 
[[Category:Files_with_Comma_Separated_Values]]
[[Category:Cumulus 1]]
[[Category:Cumulus Files]]


This page was created with text for the (legacy) Cumulus 1 software, but it has been amended to cover MX as well.
{{Version badge 1}}{{Template:Version badge Mx}}[[Category:Cumulus Files]][[Category:MX txt Files]]


=How Cumulus uses the daily summary log=
=How Cumulus uses the daily summary log=
Line 12: Line 14:
For MX release 3.9.2 - build 3097 onwards, when the software first starts, the whole of '''dayfile.txt''' is read, the contents are used to drive the [[Highcharts_-_Historic]] functionality.
For MX release 3.9.2 - build 3097 onwards, when the software first starts, the whole of '''dayfile.txt''' is read, the contents are used to drive the [[Highcharts_-_Historic]] functionality.


The file is also read, if you are using the editor provided in Cumulus.
The file is also read, if you are using the editor provided in either Cumulus flavour.


==Writing a new line to the daily summary log file==
==Writing a new line to the daily summary log file==
Line 27: Line 29:
As discussed in [[Correcting_Extremes]], it is possible for rogue values to be read from a weather station, and propagate into various log files. An error in many of those files, corrupts a particular extreme record (or more than one), but generally does not stop Cumulus working.
As discussed in [[Correcting_Extremes]], it is possible for rogue values to be read from a weather station, and propagate into various log files. An error in many of those files, corrupts a particular extreme record (or more than one), but generally does not stop Cumulus working.


Since MX now reads the entire daily summary log when you start it up, an error in '''dayfile.txt''' can cause a more significant problem, most likely for historic charts, but it can affect other functionality!
Since release 3.9.2 (build 3097) MX now reads the entire daily summary log when you start it up. This implies that an error in '''dayfile.txt''' is now always picked up when MX starts, error messages will be added to the latest file in [[MXdiags folder]], but most Cumulus users will not realise there is a problem until they use historic charts (either in the local admin interface, or on a external web server).  There is other functionality that uses recent lines in the '''dayfile.txt''' for certain calculations.
 
== Thoughts required ==
 
To correct '''dayfile.txt''', you might think the best approach is to look in your [[standard log files|log file]] covering the relevant date, or to use "Create Missing" (which works differently for the legacy software and MX as explained later). Often that is not a good idea, your standard log file might be corrupted as well, and since these log files only record spot values, they miss any extremes occurring between the log entries. 
 
Equally, to stop Cumulus, and make it rewind, may worsen the problem, because you throw away the good data you have on other derivatives just to try to resolve one rogue value. 
 
If you discover the corruption within a few days of it happening, you can make use of an earlier [[Backup folder|back-up]] as explained later.




Line 55: Line 65:
| Moving from one device to another (and not ensuring same locale on both devices)
| Moving from one device to another (and not ensuring same locale on both devices)
| Correct "locale" and/or use an external editor that offers "Replace all" (see [[#Validation by in-built editors]])
| Correct "locale" and/or use an external editor that offers "Replace all" (see [[#Validation by in-built editors]])
| Use '''Control Panel''' to correct region. Correct "locale" and/or use an external editor that offers "Replace all"
| Use '''Control Panel''' to correct region (that defines decimal symbol). To edit the file, use an external editor that offers "Replace all"
|-
|-
| Inconsistencies in list separator (what comes between fields)
| Inconsistencies in list separator (what comes between fields)
| Moving from one device to another (and not ensuring same locale on both devices)
| Moving from one device to another (and not ensuring same locale on both devices)
| Correct "locale" and/or use an external editor that offers "Replace all"
| Correct "locale" and/or use an external editor that offers "Replace all"
| Use '''Control Panel''' to correct region. See [[#Using the Cumulus 1 editing feature]] to check file is now consistent.  
| Use '''Control Panel''' to correct region (that defines list separator). See [[#Using the Cumulus 1 editing feature]] to check file is now consistent.  
|-
|-
| Duplication of dates between lines (either consecutive lines, or non-adjacient lines)
| Duplication of dates between lines (either consecutive lines, or non-adjacient lines)
Line 72: Line 82:
# Restarting Cumulus after a crash, either manually using "Rewind" approach or Cumulus is confused by a corrupted file
# Restarting Cumulus after a crash, either manually using "Rewind" approach or Cumulus is confused by a corrupted file
| Manual editing outside MX in a text editor, if particular dates appear twice, see [[#Dates restart/repeat]]
| Manual editing outside MX in a text editor, if particular dates appear twice, see [[#Dates restart/repeat]]
| Manual editing outside Cumulus in a text editor, if particular dates appear twice, see [[#Dates restart/repeat]]
| The legacy editor can resequence lines for you, if no duplicates. However, if particular dates appear twice, see [[#Dates restart/repeat]]
|-
|-
| Some lines with fewer fields than others
| Some lines with fewer fields than others
| As explained at [[Dayfile.txt#List_of_Fields]], as Cumulus has developed, more fields have been added
| As explained at [[Dayfile.txt#List_of_Fields]], as Cumulus has developed, more fields have been added
| Use [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]]
| Use [[Calculate_Missing_Values#CreateMissing.exe|separate CreateMissing.exe utility]]
| Use workaround described at [[#Legacy Workaround]]
| Use workaround described at [[#Legacy Workaround]]
|-
|-
Line 86: Line 96:
| Use [[#'''Create Missing''' on legacy dayfile editor]]
| Use [[#'''Create Missing''' on legacy dayfile editor]]
|}
|}
== Editors built into Cumulus ==
If you need to view, or edit, a line in the [[dayfile.txt|daily summary log]:
* MX: Use [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|Edit menu in MX's admin interface]]
* C1: Use [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|Edit menu on Main Screen]]
===Validation by in-built editors===
Both  the legacy editor and the MX editor will ensure that the correct number of fields is stored (as defined at the release where you do editing) (one common error in an external editor is to accidentally add/delete a field).
The legacy editor will allow you to edit the date field, the MX editor cannot change the date field.  The MX editor, reads the file into an array, it uses the array index for all actions on a particular line, and it then writes the array back to the file when you finish editing.
The legacy editor will validate any edit you make to individual fields; it checks for appropriate content (integer, real number, time-stamp).
Unfortunately, '''the editor provided with MX does not validate any fields'''.  In MX, the editor will save an edited line, even if there are errors in individual fields:
* you can put inappropriate content in a particular field  (integer, real number, time-stamp)
* you can use the wrong separator in fields you do edit (i.e. between hour and minute for time-stamps,  or between integer and decimal parts in any real number)


==Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software==
==Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software==


If you accidentally corrupt a line in the [[dayfile.txt|daily summary log]], then view the resulting file:
* MX: Use [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX's admin interface]]
* C1: Use [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|original Cumulus software]]


Note that both flavours have editors that can be used to check the file contents, even if you don't need to edit any line.
If you have one, or more, dates missing in your dayfile.txt file, then the first question is:
* '''Has the line been deleted by accident?'''
* ''Is the line missing because it was never saved into the file?''


If you have one, or more, dates missing in your dayfile.txt file, then the next question is, '''has the line been deleted by accident?'''
* If a line for a particular date was present before, but is now corrupted or missing:
 
*# See if you have a back-up of '''dayfile.txt''' with the line present, and correct
If a line for a particular date was present before, but is now corrupted or missing:
*#* If the missing/corrupted line is for a recent date, then Cumulus makes '''a backup of dayfile.txt every time it is restarted and after every end-of-day rollover'''
#See if you have a back-up of dayfile.txt with the line present and correct
*#* If the missing/corrupted line is for an older date, then ''maybe you took a back-up onto a separate drive or separate device''
#*If it is a recent date, then Cumulus makes a backup of dayfile.txt every time it is restarted and after every end-of-day rollover
*# If you have a suitable backup available, '''take a copy of that back-up file'''
#*If it is an older date, then maybe you took a back-up onto a separate drive or separate device
*# Append onto the copy of the backup, '''any dates after when that copy ends''', taking the extra lines from the current dayfile.txt
#If you have a suitable backup available, take a copy of that file
*# Rename the current dayfile.txt to say dayfile.old
#Append onto the backup, any dates after when that copy ends, taking the extra lines from the current dayfile.txt
*# Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
#Rename the current dayfile.txt to say dayfile.old
*# Cumulus will now use the file with all days correct
#Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
* If Cumulus never saved the line in the file in the first place
#Cumulus will now use the file with all days correct
*# The missing line will not be in any back-up
*# If it is the last line on the file that is missing (i.e. last rollover failed), take a copy of the whole [[data folder]], and keep that copy in a safe place
*# Now look in [[backup folder]], and open the '''daily''' sub-folder
*# If there is a subfolder within "daily" that was successfully created during the rollover that failed, rewind Cumulus by overwriting the contents of '''data'' folder with files from the backup in '''daily''' sub-folder. Restart Cumulus, and let it create a new dayfile.txt line.  Stop Cumulus again, restore the original files (except "dayfile.txt) from the copy you put in safe place
*# If the rollover failure meant a backup was not created in the "daily" sub-folder, this will usually be the case, you need to follow instructions for the MX, or legacy, Create Missing




''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''
''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''


'''Another lesson here, is to periodically take a backup, stored away from your Cumulus running environment in case you ever corrupt an old line'''
'''Another lesson here, is to periodically take your own backup, stored away from your Cumulus running environment in case you ever corrupt an old line'''


==Dates restart/repeat==
==Dates restart/repeat==


If you have a Cumulus crash (either because the connection between the weather station and Cumulus fails; or because there is a untrapped error in the Cumulus code; or because there is a power issue), then when it restarts it is possible that Cumulus will think it is doing catch-up from an earlier day, and the ''dayfile.txt'' may end up with consecutive lines being in date ascending order until the problem, then jumping back to an earlier date, before continuing in date asceneding order (but repeating one, or more, dates already in file).
If you have a Cumulus crash (either because the connection between the weather station and Cumulus fails; or because there is a unmanaged error in the Cumulus code; or because there is a power issue), then when it restarts it is possible that Cumulus will think it is doing catch-up from an earlier day, and the ''dayfile.txt'' may end up with consecutive lines being in date ascending order until the problem, then jumping back to an earlier date, before continuing in date ascending order (but repeating one, or more, dates already in file).


If Cumulus was working correctly before the problem, then the lines stored before the problem should be okay, just delete the lines that repeat earlier dates, so the file ends up being date ascending order with no dupicates. Similarly, if Cumulus was working correctly after the problem, but there was an issue before the restart, then keep the lines that repeat the dates, but delete the earlier lines with same dates, so the file ends up being date ascending order with no dupicates.
If Cumulus was working correctly before the problem, then the lines stored before the problem should be okay, just delete the lines that repeat earlier dates, so the file ends up being date ascending order with no duplicates. Similarly, if Cumulus was working correctly after the problem, but there was an issue before the restart, then keep the lines that repeat the dates, but delete the earlier lines with same dates, so the file ends up being date ascending order with no duplicates.


If there are two lines with the date when the problem occured, then it is likely you will manually have to edit the two lines into one line.  Any field with a time-stamp before the problem will be kept unless it is obvious that extreme was correctly broken (i.e. not rogue restart value) after the problem.
If there are two lines with the date when the problem occurred, then it is likely you will manually have to edit the two lines into one line.  Any field with a time-stamp before the problem will be kept unless it is obvious that extreme was correctly broken (i.e. not rogue restart value) after the problem.


If the dates restart in your daily summary log file, because you manually stopped Cumulus close to a rollover time, or you corrupted a file perhaps by regressing to an older release and back; then it is likely you will need to merge the two lines with same date, deciding for each value field which is more likely to be right, and matching it with correct time-stamp. A rainfall (or wind run) total might require summing totals in the two individual lines, or discarding a rouge value and accepting the other, your judgement!
If the dates restart in your daily summary log file, because you manually stopped Cumulus close to a rollover time, or you corrupted a file perhaps by regressing to an older release and back; then it is likely you will need to merge the two lines with same date, deciding for each value field which is more likely to be right, and matching it with correct time-stamp. A rainfall (or wind run) total might require summing totals in the two individual lines, or discarding a rouge value and accepting the other, your judgement!


==Validation by in-built editors==


Both  the legacy editor and the MX editor will ensure that the correct number of fields is stored (as defined at the release where you do editing) (one common error in an external editor is to accidentally add/delete a field).
==Correcting date separator errors==


The legacy editor does also validate individual fields for appropriate content (integer, real number, time-stamp).  
Cumulus uses a format of "day of month in 2 digits", separator, "month number as 2 digits", separator, "last 2 digits of year" in the first field of every line stored in '''dayfile.txt'''. It might be expected that "separator" is consistent throughout the file, because the device/locale when the line is created determines the separator.  The problem comes when a user moves their Cumulus installation to a new device, and forgets to ensure that new device uses same (Microsoft Windows PC) region, or (Unix, Linux, and other operating systems) locale, settings.


Unfortunately, '''the editor provided with MX does not validate any fields'''.  Thus you can use the wrong separator (between date elements, between hour and minute for time-stamps, between integer and decimal parts in any real number) in the MX editor and the incorrect line will be accepted.
Historic notes:
*In Cumulus 1 builds, any symbol (including ;, :, /, &, -, and .) character (excluding what is defined as list separator) within the first field of each line was treated as part of separator characters when parsing for date.  Thus Cumulus 1 was reasonably tolerant when someone moved to a new device.
*For builds 3000 to build 3049, MX used fixed offsets to find day, month, year in date field. This meant MX could process any file that the legacy software had accepted. The issue was this did not work for locales that used two characters for separator, and with MX now working with far more locales, this proved to be a serious problem.
*From Build 3050 MX uses the separators, defined by the locale, to split the values (so allowing for multi-character separators).  The same character(s) had to be used in every line of the file, or this gave a new problem.


==Correcting date separator errors==
Some [[Daily Summary|third party routines]] for reading dayfile.txt take a different approach, they check the first field for the first character that is not a "0 to 9" digit, take that as start of separator, then look for the next "0 to 9" digit, that belongs to next part of the date, so the separator ends at previous character.  Other third party routines, ask user what they use as date separator


=== Date Separator in MX===
=== Date Separator in MX===
Line 146: Line 179:
The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date.  Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".
The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date.  Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".


Historic note: In Cumulus 1 builds, any symbol (including ;, :, /, &, -, and .) character (excluding what is defined as list separator) within the first field of each line was treated as part of separator characters when parsing for date. For builds 3000 to build 3049, MX used fixed offsets to find day, month, year. From Build 3050 MX uses the separators defined by the OS to split the values (so allowing for multi-character separators), but it does mean it will be sensitive to the separator value changing.


= Importing data not recorded by Cumulus =
= Importing data not recorded by Cumulus =
Line 152: Line 184:
You might have been using your weather station with some other weather software before you installed Cumulus.  If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus '''dayfile.txt''' file using a script or spreadsheet package.  All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in [[Dayfile.txt#List_of_Fields]].  There is more guidance later on this page about the rules you must obey for this file.
You might have been using your weather station with some other weather software before you installed Cumulus.  If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus '''dayfile.txt''' file using a script or spreadsheet package.  All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in [[Dayfile.txt#List_of_Fields]].  There is more guidance later on this page about the rules you must obey for this file.


If you have imported the data from the other weather software into the [[Standard_log_files]] format, then in the Cumulus 1 editor, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt, as explained below.  
If you have imported the data from the other weather software into the [[Standard_log_files]] format, then
* a separate utility "CreateMissing.exe" can create the new rows, for MX users, for those days previously missing in dayfile.txt, as explained below.
* in the Cumulus 1 editor, ''Create missing'' can insert the new rows, for those days previously missing, in dayfile.txt, as explained below.  


= Using CumulusMX.exe editing functionality =
= Using CumulusMX.exe editing functionality =
Line 169: Line 203:
Pick '''Edit''', click that, and an editing dialog pops up (MX uses '''altEditor ''' software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.
Pick '''Edit''', click that, and an editing dialog pops up (MX uses '''altEditor ''' software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.


There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions:
There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions, so you must manually ensure you enter correct data:
*some fields can only accept integers, other expect decimals,
* some fields can only accept integers, other expect decimals,
* and some fields can accept negatives, others don't accept signed numbers
* and some fields can accept negatives, others don't accept signed numbers
*some fields have a minimum and/or maximum acceptable value
* some fields have a minimum and/or maximum acceptable value
* time-stamps must use a colon between the 24 hour time sections for hours and minutes


As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error.  There was a third-party proposal to add simple validation into a replacement editor, that would hav
As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error.  There was a third-party proposal to add simple validation, as listed above into a replacement editor, as well as adding in-line editing, but it could not be made compatible with the particular third-party modules used by the admin interface at release 3.6.0.


Pick '''Delete''', click that, and a simple dialog pops up (MX uses '''altEditor ''' software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted.  Despite the MX engine getting a copy of the fields that are to be deleted, it does not validate the fields, just the line number. The button labelled 'Close' or 'Cancel' does the same effect as clicking the "x" in the top right corner, it prevents the api sending any deletion message back to the MX engine.
Pick '''Delete''', click that, and a simple dialog pops up (MX uses '''altEditor ''' software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted.  Despite the MX engine getting a copy of the fields that are to be deleted, it only checks the line number. The button labelled 'Close' or 'Cancel' does the same effect as clicking the "x" in the top right corner, it prevents the api sending any deletion message back to the MX engine.




==Cautions if using an obsolete MX release==
==Cautions if using an obsolete MX release==


* '''Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor'''
* '''Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor'''
Line 198: Line 232:
= Using CreateMissing.exe editing functionality=
= Using CreateMissing.exe editing functionality=


'''You must be running MX in order to use this utility''', as some information is shared between CumulusMX.exe and CreateMissing.exe.
Basically, this utility will create a new '''dayfile.txt''', if there is already a file with that name it gets renamed, and existing fields are copied across. If there are some dates missing in an existing file it will create new lines ''if data for those missing lines can be derived'' from data already in [[Standard_log_files]]. If an individual line in an existing '''dayfile.txt''' is shorter than the number of fields expected in the current MX release, then '''CreateMissing.exe''' will calculate daily values for the missing fields based on data already in [[Standard_log_files]].


Please see [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] and/or https://github.com/cumulusmx/CreateMissing/blob/master/README.md.
'''You must have installed MX in order to use this utility''', as some information in [[Cumulus.ini]] and some [[Software#By_Mark_Crossley|.dll files]] are shared between CumulusMX.exe and CreateMissing.exe .


Information there is not repeated here.
''It is important to understand'' that when you start MX interactively, or as a service, it reads the contents of '''dayfile.txt''' into an internal array (held in random access memory - RAM).  Therefore, if you run '''CreateMissing.exe''' while ''CumulusMX.exe'' is running, any updates to the file are ''not'' seen by MX until that software is restarted.  Release 3.20.0 (beta build 3199 onwards) adds a new "Utils" menu with a new option to refresh the internal values without restarting MX.


Basically, this utility will not replace any data that is already in the daily summary log file. It will create new lines if data for those missing lines can be derived from data already in [[Standard_log_files]]. If an individual line in '''dayfile.txt''' is shorter than the number of fields expected in the current MX release, then '''CreateMissing.exe''' will calculate daily values for the missing fields based on data already in [[Standard_log_files]].
Instructions for downloading and running '''CreateMissing.exe''' can be found on [[Calculate_Missing_Values#CreateMissing.exe|another Wiki page]] and most of the information there is not repeated here.  


The developer includes brief installation and running instructions at https://github.com/cumulusmx/CreateMissing/blob/master/README.md.


= Using the Cumulus 1 editing feature =
= Using the Cumulus 1 editing feature =
Line 229: Line 264:
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
#*Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
#*Depending on your weather station, Cumulus is able to read values at least every minute (maybe every 10 seconds), and consequently update [[today.ini]] frequently (each minute in Legacy Cumulus/each logging interval in MX) if an extreme happens,
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,  
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th (maybe 1/120th depending on weather station reading frequency) of the data,  
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)


Line 266: Line 301:
==Legacy Workaround==
==Legacy Workaround==


The legacy functionality is not designed to insert (or correct) individual missing extreme figures for daily summary lines that already have some fields in them, only to insert complete missing lines. However, we can workaround that constraint:
The legacy Cumulus 1 functionality is not designed to insert (or correct) individual missing extreme figures for daily summary lines that already have some fields in them, only to insert complete missing lines. However, we can workaround that constraint:


'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
Line 281: Line 316:
# Click the ''Create Missing'' button
# Click the ''Create Missing'' button
# Wait while Cumulus scans all the standard log files, and for each source or derived field in those files, tracks the highest and lowest for each meteorological day that is now missing in dayfile.txt.
# Wait while Cumulus scans all the standard log files, and for each source or derived field in those files, tracks the highest and lowest for each meteorological day that is now missing in dayfile.txt.
#* As an aside, be aware that when the contents of [[today.ini]] was used to create a new line in dayfile.txt originally, the high and low that were copied across were derived from all source values that Cumulus obtained from your weather station (depending on your weather station type this might be around every minute; some weather stations provide new readings less frequently, some provide them more frequently).
#* As an aside, be aware that when the contents of [[today.ini]] was used to create a new line in dayfile.txt originally, the high and low that were copied across were derived from all source values that Cumulus obtained from your weather station (this will usually be at least once a minute, depending on your weather station type it might be around every ten seconds; some weather stations provide new readings less frequently, some provide them more frequently).
#*Continuing this aside, the high and low that Cumulus is now deriving are based purely on those values stored in the standard data log, by default that is just every 10 minutes.
#*Continuing this aside, the high and low that Cumulus is now deriving are based purely on those values stored in the [[Standard log files|standard data log]], by default that is just every 10 minutes, but it might be only every 30 minutes.
#The new lines that Cumulus 1 adds to your dayfile.txt are also added to another log file '''dayfileeditlog.txt''', so we can access that to find the new values.
#The new lines that Cumulus 1 adds to your dayfile.txt while you use the editor, are also added to another log file '''dayfileeditlog.txt''', so we can access that to track the new values.
#For simplicity, click '''OK''' to save the edited '''dayfile.txt''' file, and exit the editor.
#For simplicity, click '''OK''' to save the edited '''dayfile.txt''' file, and exit the editor.
#Rename the amended dayfile.txt as "dayfile(generated).txt".
#Rename the amended dayfile.txt as "dayfile(generated).txt".
Line 291: Line 326:
#*You can use an editor designed for computer programmers or developers that can edit text file
#*You can use an editor designed for computer programmers or developers that can edit text file
#*You can use a spreadsheet application, such as "Libre Office Calc" or "Microsoft Excel", providing you obey the editing rules defined later
#*You can use a spreadsheet application, such as "Libre Office Calc" or "Microsoft Excel", providing you obey the editing rules defined later
#Now merge in (as read-only text) the previous contents of the original dayfile.txt from the copy taken in step 2
#You should first merge in (as read-only text) the previous contents of the original dayfile.txt from the copy taken in step 2, although you could skip the fields you have updated, that makes it more complicated!
#Now merge in (where there are fields missing) just those fields from the respective lines in "dayfile(generated).txt".
#You should next merge in (where there are fields to be updated or previously missing) ''just those fields from the respective lines'' in either "dayfile(generated).txt", or '''dayfileeditlog.txt''' (whichever you find easier to use, the latter has fewer lines so it may be easier to use).
#Finally, copy your temporary "dayfile.txt" into the "Cumulus\data" folder, so from now on Cumulus accesses the file with the maximum number of fields present.
#Finally, copy your temporary "dayfile.txt" into the "Cumulus\data" folder, so from now on Cumulus accesses the file with the maximum number of fields present.


One note of caution:
One note of caution:
*If a required source or derived value is not present in the standard data log lines, Cumulus will write into related dayfile.txt fields zero for values, and "00:00" for time stamps, because it does not understand the concept of "Null".
*If a required source or derived value is not present in the standard data log lines, Cumulus will write into related dayfile.txt fields zero for values, and in the editor "00:00" (in normal running it would insert your rollover time) for time stamps, because it does not understand the concept of "Null".
*Cumulus 1 does not have the functionality to retrospectively calculate derived values when it is reading source values in the standard data log line.
*Cumulus 1 does not have the functionality to retrospectively calculate derived values (e.g. cannot work out apparent temperature) when it is reading source values (e.g. temperature, humidity, wind speed) in the standard data log line.




Line 303: Line 338:


If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it.  
If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it.  
== Warning if editing outside MX ==
You need to understand the following difference between the legacy Cumulus 1 and MX. In the legacy software (and early MX releases), "dayfile.txt" is read by the software if any output reports past days.  In later MX releases, the only time the file is read is when the software is restarted, it is not read again.
From MX release 3.9.2 - b3097, Historic Charts were added to both the interface and the default web pages.  As a consequence, when you start MX in this release (or any subsequent release), the entire content of dayfile.txt is held internally (i.e. in RAM), and it is these internally held values that are used by the .json api/files that feed these [[Highcharts - Historic]] plots.  MX will append an additional array when it stores a new line in the file.  The MX in-built editor actually displays/edits the internally held data, and then writes back (including any deletions or edits) that internal array into the file.
'''Prior''' to release 3.20.0 - b3200, if you were to edit the "dayfile.txt" using an external editor without stopping MX, any change you make is not seen by internal array, and such external edits would be lost should you (in the same MX session) use the internal editor.  To propagate your external edits, you need to first stop, and then restart MX, and then the internal array will match the file.
''From release  3.20.0 - b3200,'' there is a utility provided that can copy what is held in the daily summary log file and update the internal array.  Access this utility from the '''"Utils" menu''' added to the interface at this release.


== General Editing Rules ==
== General Editing Rules ==
These rules should be followed whatever Cumulus file you edit externally.


# Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
# Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
Line 317: Line 364:
#*Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
#*Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
#Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
#Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
#All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
#All figures must be within the range of sensible figures for that field (e.g. no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in any number over 100 for a relative humidity)
# Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing.
# Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing. Also ensure all lines have same end of line characters, this is one way accidental blank lines can be inserted!
# Don't add a header line to the file, Cumulus expects all lines to be data lines.
# Don't include any header line in the file, Cumulus expects all lines to be data lines.


=== File specific Editing Rules ===
=== File specific Editing Rules ===
These additional (above rules still apply) rules are specific to any external editing of "dayfile.txt".
[[File:Open office (editing cumulus log files).png| right]]
[[File:Open office (editing cumulus log files).png| right]]
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM. (If you are using old software, both Microsoft and Google editors, used to insert BOM automatically).
# All rows must ''start with date'' and include at least 14 further fields ''in correct sequence''.
# All rows must ''start with date'' as a text field (not a date formatted field) and include at least 14 further fields ''in correct sequence''.  
# The (meteorological) date format uses ''two digits for the year''.
# Remember the month must be the middle figure in the text field representing date, USA date convention cannot apply within this logfile.
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types). 
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or  Microsoft's '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.  
#Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile.  
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.  
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.  
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.  
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.  
#* If you move your software (any flavour) to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
#* If you move your software (any flavour) to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
# The (meteorological) date format, in this text field, uses ''two digits for the year'':
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types). 
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or  Microsoft's '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, in either case Cumulus will no longer be able to use the log file.
# Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
# Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
#* If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
#* If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
# Rows can vary in length but only by missing off ''fields at the end''. The minimum number of fields after the date is 14, the maximum varies between different versions.
# Rows can vary in length but only by missing off ''fields at the end''. The minimum number of fields after the date is 14, the maximum varies between different Cumulus releases.
#Each field has a pre-defined format, and the same format must always be used in that field position.
#* (The variation between maximum number of fields may cause a problem, if you regress to an earlier release!)
# Each field has a pre-defined format, and the same format must always be used in that field position.
#No fields will accept letters.
#No fields will accept letters.
#*Some fields (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
#*Some fields (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
#* Most value fields are in ''real number format'' using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
#* Most value fields are in ''real number format'' using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
#Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
# Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
#when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a ''lowest or highest value'' and that must be paired with a time-stamp in the next field.
#when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a ''lowest or highest value'' and that must be paired with a time-stamp in the next field.
#*Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
#* Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
#Time stamp fields must always be in ''format HH:mm'' i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
# Time stamp fields must always be in ''format HH:mm'' i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
#**Be aware you will have problems if you, or your editing software, add seconds.
#**Be aware you will have problems if you, or your editing software, add seconds.
#*If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
#* If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
#* Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
#* Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
# Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
# Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
5,838

edits

Navigation menu