Today.ini: Difference between revisions

m (add categories)
 
(41 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Category:Ini Files]] [[Category:Cumulus 1]] [[Category:Cumulus MX]] This Wiki page previously only covered Cumulus 1; it still covers the legacy software, but has been updated so it focuses on MX.
== Introduction ==
The files '''today.ini''' and '''yesterday.ini''' are used by Cumulus to store the Highs and Lows (and their timestamps) for key parameters as reached on each of the two days. These files are normally located in the Data sub-folder of the path where your Cumulus.exe is installed. See [[FAQ#I_can.E2.80.99t_find_my_data_files.21| FAQ on location of data log files]] for where else to look.


= Introduction to tracking of Daily Extremes/Totals/Averages =
The highs and lows are held in memory while Cumulus is running. At the end of a day [either midnight, or 9am (or 10am) depending on your configuration and season], the final values and timestamps written to today.ini file are rolled-over to become values and timestamps in yesterday.ini; and a new today.ini is created. From the examples below you can see that the two files have a few differences in content, so rolling-over does involve a little editing work.


Steve Loft first wrote Cumulus software in 2003, although it was not shared with the public until 27th January 2004. His (no longer available) web site claimed (amongst other reasons) that Cumulus was invented to cope with [[Meteorological_day|tracking high and low for days starting at 9 a.m.]] as (at that time) no other software could do that. Therefore, it can be assumed that even in 2003, Cumulus software was tracking the highest/lowest for all time as well as for each day . (For daily period, there is also tracking of some totals, and in some cases by dividing by number of samples averages can be calculated). This Wiki page covers all aspects of the daily tracking.
If you restart Cumulus during the day it will read both these files at startup, so it can resume tracking extremes of the key parameters starting from latest stored values in today.ini. Equally, if you close Cumulus during the day it will write the final values for highs and lows and their timestamps to today.ini as part of the close down process.

The legacy Cumulus 1 software has a '''Recent extremes''' frame on its [[Cumulus_Screenshots#Main_Screen|main screen]]. In release 1.0 (27th January 2004) this only showed extremes for today, but from 1.1 (17th February 2004) it also shows extremes for yesterday. The interface provided with Cumulus MX is also able to show the extremes for today and yesterday (on a single web page).

=Essential reading=

Cumulus 1 has a [[Downloads#Resources|Help file]] that is part of the Cumulus 1 installation package; that provides a simple explanation for each file.

David Jamieson created this Wiki page on 27 August 2009, to cover "today.ini" and "yesterday.ini" files. His introductory text simply stated these files were for tracking extremes in the two days and new files were created at 9am or midnight rollover. David also included a listing of a typical "today.ini" file.

Since the initial creation of the Wiki, it has attempted to become reference documentation that answers questions that arise frequently in the support forum. This does mean its pages have become harder to read, and thus there is now an attempt to split "essential" facts that give a basic understanding (as in following subsections) from "technical" facts (later on this page with more complex instructions on how to cope with problems).

Therefore for a quick read, stop after this essential section or use the page index above to skip quickly to whatever is of interest to you.


== How Cumulus tracks extremes and why two files were introduced ==

Cumulus software handles daily tracking by storing values internally, i.e. within its use of random access memory. As the software processes (the time interval for doing this varies between weather station type and also on exactly which release you are running) data from the weather station, incoming values are compared against internally held values, and when appropriate the values being processed update the internal values.

Cumulus initially made the assumption that it would be left running continuously, so holding values internally enabled them to be shown on that screen. However, subsequently it was realised that Cumulus had to be stopped and restarted to install a new version (and Microsoft's updates system restarts computers as part of its installation process). Information held in RAM is lost when the software is closed.

Apparently when version 1.3 was released (18 January 2005) it did not include "today.ini" nor "yesterday.ini". An attempt to track the history of the "today.ini" file appears [[Changes to this file at particular releases|below]]. Unfortunately, Steve Loft lost some of his notes, and the Cumulus 1 version history in his [[Downloads#Earlier_Versions|changes.txt]] is incomplete. So it is pure guesswork that version 1.4 resolved the problem of losing daily information on closing Cumulus by introducing a '''today.ini'' file to hold the daily extremes and total records.

Steve Loft did document that at rollover, the daily extreme/average records were "transferred" (there are some differences) from "today.ini" to [[yesterday.ini]]. That second file was not vital, but made it easier to display yesterday's extremes on the main screen of the legacy software.

The sub-section below covers end of day actions in more detail, subsequently explaining actions might happen twice a day!

When you close Cumulus, it will write the final values for highs and lows and their timestamps to ''today.ini'' as part of the close down process.

==End of day actions==

End of day actions happen when Cumulus detects that it is processing weather data for the rollover time [either midnight, or 9am (or 10am) depending on your configuration and season], this might be during the processing of archive data while Cumulus is catching up after it has been restarted, or in normal running when those clock times are reached.

Oversimplifying the process a bit, the contents of today.ini, goes to two places:
# yesterday.ini
# a new line appended onto [[dayfile.txt]]

After that a new today.ini is created populated with initial values for each extreme/total entry.

The three files have a few differences in content, so rolling-over does involve a little editing work:
# The multiple lines in the [General] section of today.ini shrink to just one item in yesterday.ini (Date) and one item in dayfile.txt (abbreviated date)
# The 'Total' and 'Samples' values in "today.ini" become the single 'AvgTemp' in "yesterday.ini" and "dayfile.txt"


=== Complexity if you are recording sunshine hours ===

Sunshine hours are recorded starting at midnight, regardless of what rollover time is used.

For MX only, the sunshine hours are partly recorded in [[yesterday.ini]] and partly in today.ini. So even if your rollover is 9 am, there will be an update to both files at midnight which is when the current sunshine hours figure in today.ini is copied to yesterday.ini before the figure in today.ini is reset to zero.

==Where are the files stored?==

The files '''today.ini''' and '''[[yesterday.ini]]''' are located in the [[Data folder|"data" sub-folder]] of the path where your cumulus.exe (or CumulusMX.exe) is installed.

(NOTE: Microsoft Windows Operating Systems, may relocate some files, as explained at [[FAQ#I_can.E2.80.99t_find_my_data_files.21| FAQ on location of data log files]]).


When Cumulus is left running, a daily backup of all the files is created as part of this rollover in a subfolder 'daily' of the [[Backup_folder|backup folder]]. Depending on the release you are running, the files included in the backup may be a snapshot of their content just before rollover, just after rollover, or at the first standard interval after rollover.

== How Cumulus updates the file ==

This depends on which flavour of Cumulus you run. The [[#End of day actions|end of day process]] is same for both flavours, but updates during the day work differently for the legacy software and for MX.

For '''today.ini''', it is important to be aware that MX can read a file created by Cumulus 1 (MX can read the time format in the value for '''Timestamp=''' parameter whether it is in Cumulus 1 or MX format), but Cumulus 1 cannot read a file that has been updated by MX.

Cumulus MX and the legacy software handle the updating of "today.ini" differently, this difference is critical should you want to [[Correcting_Extremes#Correction_of_extremes_for_today|edit out rogue data]].

=== How Cumulus 1 updates the file ===

Steve Loft never shared his souce code, so what follows is just a guess at how the update might work. The frequency at which data is read from a weather station varies depending on the type, but is at least every 30 seconds.

The indications are that Cumulus 1 has an internal one minute timer that triggers the logging of readings (after conversion to units selected) to an internally held [[Recent history|recent history database]] and the comparison of those readings against existing internally held extremes/totals. The file is updated immediently afterwards, each update only changes those lines (within sections) where the extreme/total/count/time has changed, and other lines retain the same content as before the update.

Whilst you are strongly advised not to manually edit the file with Cumulus running, because access to the file cannot be shared, if you are able to complete the edit between one real-time interval and the next, any change you make is retained.

(NOTE: A full set of latest spot readings are logged to a file at a configurable interval that might be every 10 or 30 minutes).

===How MX replaces the file===

Note the subtle difference in this sub-section header. MX does all updates to the today extremes/totals/counts only to internally held values. Although the [[Software#Latest_MX_Source_Code|source code]] is available, you need to be more technical than the person typing this to understand at what frequency these internal updates occur. It is probably every time data is read from the weather station, but might be just when the externally stored [[Recent history|recent history database]] is updated.

Critically, MX only updates the "today.ini" file at the configurable [[Standard log files|Standard interval]] used for logging the spot values (might be every 10 or 30 minutes). Each update is a ''rewrite of the entire file contents'', from the internally (RAM) held values.

You would be wasting your time should you try a manual edit of the file while MX is running, as the next MX update will overwrite any manual changes! Don't be misled by the fact that a sharing violation is less likely to be an issue (because of the longer interval between MX updates).

== Format of the file ==

The files are text files, consisting of many lines. Some lines consist of a single piece of text surrounded by square brackets, these are the section names. The sections (for "today.ini" after the first [General]) can be in any order, Cumulus will maintain whatever order the sections are currently in.

Under each section name, there is a list of parameters <code>attribute=value</code>, with one parameter per line. The attribute names are defined by Cumulus, but can appear in any order:
* Where a time-stamp is stored, note that in "today.ini" only the hour and minute parts of a time are stored.
* Where an extreme/total value is stored, note that it is always post conversion to the units selected by the Cumulus user.
** It was mentioned earlier that the file exists to store values that Cumulus holds in RAM. Internally those numbers are in binary (base 2), but in the file the numbers are expressed to base 10. The value of any integer part of numbers is unchanged between the two bases, but decimal parts in base 2 and base 10 do not convert exactly, therefore in the file you may see some strange looking numbers with lots of decimal places.

The key difference between all [[:Category:Ini_Files]] for the different flavours is the formatting of any time-stamps that include a date:
* [[File:Badge v1.png]] Cumulus 1 used the format specified in Control Panel for your region settings, for UK that would typically be <code>day/month/year (space) hour:minute</code> in today.ini
* Cumulus 2 (withdrawn) worked differently, all date/time stamps were converted to UTC, and stored in ISO 8601 format of <code>yyyy-MM-ddTHH:mm:ss</code>(using the net specifiers that MX uses).
* [[File:Badge vMx.png]] Cumulus MX uses the same format as Cumulus 2, but all date/times are expressed according to date/time read from the computer running MX, so use whatever time-zone you have selected on that device.

Please look at the category page (link above) to read more on formatting differences between the legacy Cumulus and MX.

===Typical Sections within file===

As Cumulus is developed it is adding further sections, but the legacy Cumulus (by 1.9.4) used the following:
* The [General] section stores current date and time.
* The [Wind} section stores the highest wind speed and highest gust, it holds the sum of wind speeds as wind run, and details for the dominant wind.
* The [Temp], or temperature, section stores the highest and lowest temperature, the sum of all temperatures from every sample (Total), and the number of Samples in that total, the Cumulative Chill Hours total (for the season), and the cumulative Heating and Cooling Degree Days for the current day-so-far.
* The [Pressure], [Humidity], [AppTemp] (for apparent temperature), and [Dewpoint] sections just hold Highs and Lows.
* The [WindChill] section only holds lowest, the [HeatIndex] section only has highest.
* The [Rain] section holds a lot of different parameters, including the Start count that derives most rain outputs, and the LastTip date-time.
* Other sections present in 1.9.4 are [ET], [Solar], [NOAA], and [FineOffset]; whether these are updated depends on what sensors you have, and whether you have set up NOAA reports.


MX development has been far more rapid than the legacy software, and there have been a lot of changes to the content of the two files. Unfortunately, the MX release announcements rarely go into enough detail to permit good documentation, so all mentions relating to MX on this page are guesses from examination of the file, and it cannot be guanteed that the information is correct for whatever MX release you may be running!

A typical MX release will include all the 1.9.4 sections, plus:
* The '''[Records]''' section contains one line denoting when extreme records were last revised. For example "Alltime=2020-03-06T06:42:13", indicates when the all-time extreme records was last updated. In the example file, there were various extreme records broken at that time on 6 March (lowest temperature, lowest apparent temperature, and greatest wind chill), but none have been broken since.
* Other sections (depending on MX release) may include [FeelsLike], [Humidex], [Lightning], and [TempMidnight]; whether these are updated depends on what sensors you have.


== Dealing with rogue values ==


===Composite Example of 'today.ini' file===
The today file is written to throughout the current day as any new High or Low for the day at reached. If your weather station reports a rogue value, an incorrect update to the High or Low may result, but details of that update should be in the [[Diags]], so look there to see what correction will be required to eliminate the rogue value. Normally you would not need to edit today.ini, as if you wait until the next day, Cumulus provides (with its Edit menu) an easy way to amend the stored extremes for this month, this year, monthly records, and all-time. The wrong figures may be displayed on the yesterday screen for this next day, but yesterday.ini can be edited more easily!


'''Notes in round brackets () are not part of the file, they simply explain elements of the composite.'''
'''Note:''' If you do decide to edit today.ini outside Cumulus, you must close Cumulus down first!
=== Editing today.ini within Cumulus ===
You may wish to [[FAQ#My_station_invented_some_rain_that_didn.27t_really_occur.2C_and_I_want_to_set_it_to_zero_.28or_some_other_figure.29 | edit the rain data]] if there has been a quirk in your weather station reporting during the day and the figure is abnormally inflated (or indeed under-reported). The stored rainfall today value in today.ini may be edited within Cumulus via ''Today's rain'' screen on the '''Edit''' menu. This edit does not affect any false derived rainfall values for the month or year, unless you subsequently close and restart Cumulus the same day to force recalculation of the longer period totals.
=== Editing today.ini or yesterday.ini outside Cumulus ===
If the rogue value is not rainfall, then you might need to edit today.ini or yesterday.ini using an external editor. '''Note:''' If you do edit either of these files, you must close Cumulus down first!


* This is an example made up of composites that may not all be present in an actual file.
The files are text files, so edit using any plain text editor. The format consists of a number of sections (section title is delimited by []), and a number of properties (consisting of attribute name, an equals sign and a value). Blank lines can be included for readability, they are ignored in these .ini type log files. All values have to be specified in the correct format, see example below.
* This composite contains time-stamp formats used by Cumulus 1.x.y and Cumulus MX in the few places where they differ.
* Because this example is made up of composites, the times shown are not all consistent, in a real file no time anywhere will be later than the time at the top!
* The order the sections appear in this composite may not match your file; as mentioned above the section order can be edited.


==Example today.ini file==
<pre>
<pre>
[General]
[General]
Date=29/04/2011
Date=29/09/2009
Timestamp= (format for Cumulus 1 e.g. "29/09/2009 09:50:00"; format for Cumulus MX e.g. "2019-09-29T09:50:00")
Timestamp=29/04/2011 11:50:00
CurrentYear=2009
CurrentMonth=9
CurrentDay=13


[Wind]
[Wind]
Line 36: Line 152:
DominantWindBearingX=-3914.11743164063
DominantWindBearingX=-3914.11743164063
DominantWindBearingY=4215.82763671875
DominantWindBearingY=4215.82763671875

[Temp]
Low=8.30000019073486
LTime=01:16
High=16.8999996185303
HTime=11:41
Total=7500.697265625
Samples=714
ChillHours=1.64999914169312
HeatingDegreeDays=5.34738397598267
CoolingDegreeDays=0


[Pressure]
[Pressure]
Line 59: Line 164:
Start=1923.59997558594
Start=1923.59997558594
Yesterday=0
Yesterday=0
LastTip=2011-04-22 22:08
LastTip=2009-09-14 10:48
HourlyHigh=0
HourlyHigh=0
HHourlyTime=00:00
HHourlyTime=00:00
ConsecutiveRainDays=2
ConsecutiveRainDays=2
ConsecutiveDryDays=0
ConsecutiveDryDays=0
RG11Today=20


[ET]
[ET]
Annual=0
Annual=1148.2578125
Startofday=0
Startofday=1147.24182128906


[Temp]
[Humidity]
Low=8.30000019073486
Low=65
LTime=01:16
High=88
High=16.8999996185303
LTime=11:45
HTime=06:06
HTime=11:41
Total=7500.697265625

Samples=714
[Solar]
ChillHours=3147.15673828125
SunshineHours=0
HeatingDegreeDays=5.34738397598267
SunshineHoursToMidnight=0
CoolingDegreeDays=0.502222061157227


[HeatIndex]
[HeatIndex]
Line 92: Line 199:
Low=6.39816427230835
Low=6.39816427230835
LTime=01:30
LTime=01:30



[Dewpoint]
[Dewpoint]
Line 98: Line 206:
High=10.7219848632813
High=10.7219848632813
HTime=11:38
HTime=11:38

[Humidity]
Low=65
High=88
LTime=11:45
HTime=06:06

[Solar]
SunshineHours=1.08333301544189 (This is Cumulus 1 example)
SunshineHoursToMidnight=5.80002069473267 (This is Cumulus 1 approach)
HighSolarRad=1048
HighSolarRadTime=09:41
HighUV=7.40000009536743
HighUVTime=09:41
SunStart=0

[NOAA]
LatestMonthlyReport=NOAAMOSep2012.txt
LatestYearlyReport=NOAAYR2012.txt

(The above section will look like the one below if NOAA reporting has not been set up)


[NOAA]
[NOAA]
LatestMonthlyReport=
LatestMonthlyReport=
LatestYearlyReport=
LatestYearlyReport=


[FineOffset] (if not using a Fine Offset station, the MX defaults are as shown)
FOSensorClockTime=(the format here is different for Cumulus 1 e.g. "29/09/2009 09:50:00" and Cumulus MX e.g. "0001-01-01T00:00:00")
FOStationClockTime=(the format here is different for Cumulus 1 e.g. "29/09/2009 09:50:00" and Cumulus MX e.g. "0001-01-01T00:00:00")
FOSolarClockTime=0001-01-01T00:00:00 (this parameter only appears in later MX releases, the time shown is the default if no solar sensor)
</pre>
</pre>
==Example yesterday.ini==
See [[Average_temperature]] for one key transformation between today.ini and yesterday.ini, no ET parameters are retained, plus there are some wind and some rain parameters not copied from today.ini to yesterday.ini.


NOTE: Yesterday's total rainfall from weather station is stored in today.ini, but yesterday's total rainfall from [[Rain_measurement#Optical_Rain_Gauges|RG-11]] sensor is stored in yesterday.ini.
<pre>
[General]
Date=28/04/2011


<br>
[Wind]
<hr>
Speed=7.2
<br>
SpTime=09:00
Gust=15.9
Time=12:15
Bearing=315
Direction=NW
Windrun=35.3391151428223
DominantWindBearing=230


=Non-essential more technical information=
[Temp]
Low=5.6
LTime=05:34
High=21.0
HTime=14:46
HeatingDegreeDays=3.76118206977844
CoolingDegreeDays=0.702986478805542
AvgTemp=12.4447975158691


You can skip the following subsections unless you have a particular need to read them.
[Pressure]
Low=29.72
LTime=09:00
High=29.85
HTime=22:12


== First use of Cumulus ==
[Rain]
High=1.2
HTime=00:34
HourlyHigh=0.899999976158142
HHourlyTime=02:04
RG11Yesterday=0


When you use Cumulus software for the very first time, it records a start date, and assumes you have no data anywhere for earlier than that start time.
[Humidity]
Low=49
High=99
LTime=14:34
HTime=00:34


Some weather station types have an internal logger that can be accessed for historic data. It is theoretically possible, but not recommended, to read in that historic data, so it is included in your Cumulus extreme tracking. The minimum content for "today.ini" is the "[General]" section. If you stop Cumulus, manually edit "today.ini" to remove all other sections, and then restart Cumulus, then the software will attempt to read that historic data. Please read [[FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station | FAQ here]] for full guidance. Remember this only applies when you are first starting use of Cumulus with a weather station.
[Solar]
SunshineHours=0
HighSolarRad=0
HighSolarRadTime=09:00
HighUV=0
HighUVTime=09:00


[HeatIndex]
High=21
HTime=14:46


== Restart and Catch-up ==
[AppTemp]
Low=3.98110413551331
LTime=05:34
High=21.0052318572998
HTime=15:50


If you restart Cumulus during the day it will read the ''today.ini'' file at startup, so it can resume tracking extremes of the key parameters starting from latest stored values in today.ini. If your weather station type permits, a restart of Cumulus can go into a catch-up mode during which it reads any historic data from the weather station, for the period while Cumulus was not running, before starting the reading of the current data.
[WindChill]
Low=5.59999990463257
LTime=05:34


During that catch-up of historic data, internally held daily extremes, and the "today.ini" file, will be updated with revised highs and lows as Cumulus processes the historic data from the station's memory; and if necessary Cumulus will do a roll-over (see [[#End of day actions]]) as it processes the readings for the relevant time.
[Dewpoint]

Low=5.4555492401123
On restart Cumulus writes a backup of today.ini (and some of the other Cumulus files) into the [[Backup_folder|backup folder]] found below the folder with the cumulus.exe (or CumulusMX.exe). With Cumulus stopped, you can copy the today.ini file in either a restart backup, or a daily backup (see [[#Where are the files stored?]]), also copying the other files in that backup folder into their original folders (mostly [[data_folder|data sub-folder]]) overwriting the files in those destinations. When you restart Cumulus, the tracking will begin again as if the time has been rewound back to the date those copied files were last updated. This rewinding works best if historic data can be read from your weather station. One example of when this might be useful is if you spot a rogue value very soon after it has been recorded, the rewinding often brings in correct data for the recent period. Another example is if your computer on rebooting initially shows the wrong time and so Cumulus records some data against that wrong time; a rewind can eliminate the wrongly timed records and replace them with records timed correctly.
LTime=05:34

High=11.6547174453735

HTime=09:03
You are strongly advised not to stop/restart Cumulus close to either midnight or your rollover time. Steve Loft defines "close" in this context as within whatever time you have set as interval between logging of spot values (e.g. 10 or 30 minutes). The potential problems were significantly worse for earlier versions of Cumulus 1, but restart problems have been reduced in newer builds of C1. In general, MX is more tolerant over restart timings, but the way its code works you will encounter more problems with accuracy of output if MX is stopped for more than the few minutes needed for an upgrade (or computer reboot).
</pre>

== Editing rainfall in today.ini within Cumulus ==

This sub-section applies to Cumulus 1 and Cumulus MX.

At the time of writing, no Cumulus release contains an editor for today's daily extreme records as stored in "today.ini". There is just one exception, both flavours do contain an editor for 'Today's rain'.

Cumulus internally stores a "start of day rain counter". This is copied into "today.ini" into the ''Start='' line within the '[rain]' section. Subtracting that from the current rain counter value allows Cumulus to calculate (in your chosen units) the toatl rainfall for today. The provided editor changes the internal "start of day rain counter" so that the current rainfall total becomes what you enter into the editor.

It is important to note:
* The rainfall counter was invented in Cumulus 1 as a way that could calculate rain when the software was made to serve multiple weather station types
* The counter is based on a piece of data that can be read from a particular weather station type, it might be based on an all-time total rainfall, or based on a rainfall this year figure
* The counter is not intended to be of any interest to the Cumulus user
* Any increase in the rainfall counter is treated as representing valid rain (spike removal functionality exists only for rain rate and rain in last hour, not the total rain)
* Cumulus has special code to detect if the counter value decreases, without going into the complexity, Cumulus will normally reset its internal start of day counter based on the decrease in the number received.
* The "Today's rain" editor does not affect any derived rainfall output;
** The following in "today.ini" are not affected:
*** last tip time-stamp ("LastTip=")
*** highest rainfall rate so far today and time-stamp ("High=" and "HTime=")
*** hourly high amount and time-stamp ("HourlyHigh=" and "HHourlyTime=")
*** Highest 24 hour amount
** Outside "today.ini" it does not affect any recent records nor monthly, yearly, monthly-all-time, or all-time extremes:
*** The recent history entries whether held internally (Cumulus 1), or for MX in [[Cumulusmx.db]], are not amended so any tags used for web page data, local API, custom SQL, MQTT, HTTP (see [[Webtags/Parameters#Cumulus_MX_software|this Wiki section]]) will report incorrect values
*** Total Rainfall this month, rainfall this year/season will only be recalculated if Cumulus is stopped and restarted after the edit
*** Highest rainfall rate, highest hourly rainfall, highest 24-hour rainfall, highest daily rainfall; applying to these longer periods are not changed unles each extreme record is individually manually edited (Cumulus provides an editor for most of these, but cannot generally suggest what new value to use).
*** The consecutive wet/dry days ("ConsecutiveRainDays=" and "ConsecutiveDryDays=") in this month/year - remember these counts are to last roll-over, they exclude today
** The editor does not alter any lines already logged in [[Standard_log_files |log file]] for the current month. This means you will see distorted graphs attempting to portray rainfall within a day
** The editor does not update any database records that may be affected by the change of the rainfall total



== Dealing with rogue values ==

The "today.ini" file is written to throughout the current day, as described [[#How Cumulus updates the file|earlier]]. As mentioned above, Cumulus only provides extremely restricted ability to edit "Today's rain", not any other extreme/total that is stored in the "today.ini" file.

If your weather station reports a rogue value, an incorrect update to a High, Low, or Total, may get stored in this file. At end of day, it will then be stored in [[dayfile.txt]]. Cumulus developer advice is that instead of manually editing "today.ini", you should wait until the day has been stored in "dayfile.txt" and then use the editor for that log file to make the desired changes.

The rogue value may also affect extreme records held for this month, this year, monthly-all-time, and/or all-time. Cumulus does provide editing functionality for most (not all) entries in the files holding those extreme records and you can read instructions on [[Correcting Extremes]] Wiki page. Here it is sufficient to say it is worth looking in the diagnostics, to see if you can spot when the problem occurred, because that helps you work out what may be affected:
* [[File:Badge vMx.png]] [[MXdiags folder]]
* [[File:Badge v1.png]] [[Diags_folder]]


=== Manual editing of "today.ini" ===

The developer advice to wait until next day has a flaw; every process that happens in the meantime sees wrong data, and that might include sending data to several external sites, and a number of extra custom processes you might have in your system.

Therefore this Wiki page will now give some advice on how to manually edit the file:
# You must stop Cumulus (please see earlier in this page for details as reason depends on flavour you are running)
# Always take a back-up of existing file, or (if you decide it is easiest to create a new file rather than edit the existing one) rename it; your corrections may cause problems, so you must be able to revert
# Any plain text file editor can be used (that includes coding editors like Geany, Notepad++, NoteTab, and many others)
# Be careful to ensure any change maintains existing format (integer, decimal,time, date/time, text) paying attention to any punctuation (including decimal commas or decimal points, direction type of any slashes)
# For readability you can insert blank lines into files '''today.ini''' and '''yesterday.ini''', Cumulus will not mind.
# It is up to you to work out what new value/time to type, Cumulus won't accept nulls, but there are some "initial values/dates" that it will accept (I won't tell you these here, because they depend on release you are running and you should not be editing this file unless you have enough technical understanding to work out what Cumulus will accept)
# Remember changing extremes for [[Calculate_Missing_Values#Some_definitions|source value extremes and derived extremes]] is complicated:
#* Please see specific advice in sub-sections below

==== Edits related to pressure ====

Should be straight forward, the maximum/minimum can be edited without this affecting anything else. You should be able to look in either [[Standard_log_files |log file]] or for MX in [[Cumulusmx.db]] for pressure readings earlier or later in the day to find the new extreme value/time to replace the rogue pressure value and time.

==== Edits relating to Temperature ====

#If you change a temperature source extreme (i.e. highest/lowest temperature) you should also be taking the old value out of 'Total' line and decreasing by one the'Samples' count
# You cannot work out from any change in a temperature source extreme (i.e.. highest/lowest temperature) how to change the related derived extremes (e.g. wind chill, apparent temperature, feels like, dew point). Each derived value is worked out by combining spot values at a particular time, so you have to recalculate as many spot derived values as possible in order to work out the new derived extreme.

==== Editing Chill Hours====

Prior to release 3.12.0, the web template "thisyearT.htm" included a single cumulative figure for [[Heat/cold_degree_days_and_Chill_hours#Calculation_of_Chill_Hours|seasonal chill hours]], and that figure was taken from the chill hours figure stored in "today.ini". A lot of processing was needed to calculate this figure so, in these earlier releases, it was unusual for anyone to correct any rogue figure.

From release 3.12.0 onwards, yesterday's cumulative total is stored in "yesterday.ini". Once you have corrected any rogue maximum or minimum temperature, you can look at the outdoor temperature reported in every row in [[Cumulusmx.db]] since last rollover. For each figure that is below your chosen threshold temperature, add the fraction 1/60 (assuming interval between rows is one minute). The cumulative totals of those fractional increments are added to obtain the figure that should be in today.ini. Unless today is the first day of the month set as first month of season, add on the figure stored in yesterday.ini, to get value to put in today.ini.


==== Edits related to humidity or wind speed ====

Obviously changes to humidity, wind speed, also have an effect on derived temperature extremes mentioned above, but again you have to recalculate as many spot derived values as possible in order to work out the new derived extreme.


====Edits related to rainfall ====

If you want to edit ''rainfall since midnight'' in today.ini, it should be fairly easy to calculate a new value. Remember you also need to manually edit entries in [[Standard_log_files |log file]] and for MX in [[Cumulusmx.db]].

To edit the "rainfall counter" in [[Standard_log_files |log file]] remember that for any decrease in the actual rain, you need to increase the rain counter, and vice versa. It is probably easiest to work backwards through time working out the alteration to the rain counter for the actual rainfall increment, in the period since the preceding entry.


Other derivatives like "maximum hourly rain", "highest rainfall rate", or "highest 24 hour rainfall", all will have to be guessed or recalculated manually, there is no easy way to work these out as past values for the day are not logged anywhere.


==Cross-references for explanations of key parameters==
==Cross-references for explanations of key parameters==

[[Wind_measurement | Bearing and Direction]],
[[Wind_measurement | Bearing and Direction]],
[[Windrun]];
[[Windrun]];
Line 192: Line 367:
[[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | Dewpoint]];
[[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | Dewpoint]];


[[Webtags | Rain]],
[[Webtags|Rain]],
[[FAQ#The_longest_dry.2Fwet_spell_for_this_month.2Fyear_includes_days_from_last_month.2Fyear | ConsecutiveRainDays]], [[FAQ#The_longest_dry.2Fwet_spell_for_this_month.2Fyear_includes_days_from_last_month.2Fyear | ConsecutiveDryDays]];
[[FAQ#The_longest_dry.2Fwet_spell_for_this_month.2Fyear_includes_days_from_last_month.2Fyear | ConsecutiveRainDays]], [[FAQ#The_longest_dry.2Fwet_spell_for_this_month.2Fyear_includes_days_from_last_month.2Fyear | ConsecutiveDryDays]];


[[Cumulus.ini#Section:_NOAA | NOAA]].
[[Reports folder|NOAA]]


== Changes to this file at particular releases ==

''This section may not be a complete history, and may not be kept up to date; so don't take it as authoritative''


* 3.18.0 build 3189 (pre release 14 June 2022)
[[Category:Log Files]]
** Added '''TempMidnight''' section for tracking of min/max temperature based on day starting at midnight, to complement tracking starting at rollover time
* 3.17.0 build 3184 (released 23 May 2022)
** Some changes to handling of lighning (regarding when no strikes)
* 3.12.0 beta build 3134 (released 29 July 2021)
** Added '''Lightning''' section (lightning distance and last strike time)
* 3.11.0 build 3129 (released on 7 May 2021)
**Fix: End of day backup now always runs at rollover, so like most releases of the legacy software, the stored file represents the true end of day position.
*** For example in the '''[General]''' section, the ''Date'' (calendar date) and ''Timestamp'' attributes will relate to when it was last updated, but the three '''Current'' attributes will relate to the meteorological date just ended.
*** For the today.ini in the backup/daily folder, last update time-stamp will indicate time as at one update cycle before end of day, but file last modified will show that was at end-of-day
** New: Added (to [Temp]) GrowingDegreeDaysThisYear1=, and GrowingDegreeDaysThisYear2=.
* 3.7.0 - build 3089 (released on 28 July 2020)
** Canadian Humidity Index (Humidex) added for highest extreme monitoring in today.ini (and other longer period extreme log files)
* Release 3.6.6 build 3082 (released on 1 June 2020)
** Values in the file had been stored (in all previous builds) using 15 decimal places (because of difficulty in representing decimal parts of numbers in binary), from this release revised so all values stored using 17 significant figures
* Release 3.6.3 - build 3079 (released on 21 May 2020)
** Fix long standing problem with today.ini becoming corrupted when Microsoft Windows is shutdown
* Release 3.6.0 build 3076 (released on 4 May 2020)
** Added '''Feels Like''' section to daily high and low extremes monitored in today.ini (and other longer period extreme log files)
* Release 3.1.0
** Added '''Records''' section, this holds one attribute "Alltime" that holds a datetime stamp recording the most recent update to the [[alltime.ini]] file
*From version 1.9.4 build 1089 to release 3.10.5 build 3122
** The '''today.ini''' stored in the daily backup contains the position as at '''start of day''' (or in specified older MX releases, up to one UPDATE interval later), so it does not contain any information (except Cumulative Chill Hours) that relates to day that has just ended.
*Up to version 1.9.4 build 1088 (released 28 Jan 2014)
**The '''today.ini''' stored in the ''cumulus\backup\daily\FOLDER_NAME'' ((where FOLDER_NAME is based on date and time of creation) represented the end of day (including on last day of month) extremes
*version 1.8.9 (released on 31st March 2010)
**Fix problem with month names (timestamp in today.ini) using ''short date format''
* version 1.8.6 (released on 14th April 2009)
**Don't write today.ini unless station contacted
*version 1.8.5 (released on 12th February 2009)
**Fix wrong times for some of today's extremes
*Not documented
** It is believed that today.ini may have been introduced at version 1.4
*Version 1.0 27th January 2004 First release
**Included internal tracking of daily extremes, and display on main screen, but not storing in a file.

Latest revision as of 17:27, 6 August 2022

This Wiki page previously only covered Cumulus 1; it still covers the legacy software, but has been updated so it focuses on MX.

Introduction to tracking of Daily Extremes/Totals/Averages

Steve Loft first wrote Cumulus software in 2003, although it was not shared with the public until 27th January 2004. His (no longer available) web site claimed (amongst other reasons) that Cumulus was invented to cope with tracking high and low for days starting at 9 a.m. as (at that time) no other software could do that. Therefore, it can be assumed that even in 2003, Cumulus software was tracking the highest/lowest for all time as well as for each day . (For daily period, there is also tracking of some totals, and in some cases by dividing by number of samples averages can be calculated). This Wiki page covers all aspects of the daily tracking.

The legacy Cumulus 1 software has a Recent extremes frame on its main screen. In release 1.0 (27th January 2004) this only showed extremes for today, but from 1.1 (17th February 2004) it also shows extremes for yesterday. The interface provided with Cumulus MX is also able to show the extremes for today and yesterday (on a single web page).

Essential reading

Cumulus 1 has a Help file that is part of the Cumulus 1 installation package; that provides a simple explanation for each file.

David Jamieson created this Wiki page on 27 August 2009, to cover "today.ini" and "yesterday.ini" files. His introductory text simply stated these files were for tracking extremes in the two days and new files were created at 9am or midnight rollover. David also included a listing of a typical "today.ini" file.

Since the initial creation of the Wiki, it has attempted to become reference documentation that answers questions that arise frequently in the support forum. This does mean its pages have become harder to read, and thus there is now an attempt to split "essential" facts that give a basic understanding (as in following subsections) from "technical" facts (later on this page with more complex instructions on how to cope with problems).

Therefore for a quick read, stop after this essential section or use the page index above to skip quickly to whatever is of interest to you.


How Cumulus tracks extremes and why two files were introduced

Cumulus software handles daily tracking by storing values internally, i.e. within its use of random access memory. As the software processes (the time interval for doing this varies between weather station type and also on exactly which release you are running) data from the weather station, incoming values are compared against internally held values, and when appropriate the values being processed update the internal values.

Cumulus initially made the assumption that it would be left running continuously, so holding values internally enabled them to be shown on that screen. However, subsequently it was realised that Cumulus had to be stopped and restarted to install a new version (and Microsoft's updates system restarts computers as part of its installation process). Information held in RAM is lost when the software is closed.

Apparently when version 1.3 was released (18 January 2005) it did not include "today.ini" nor "yesterday.ini". An attempt to track the history of the "today.ini" file appears below. Unfortunately, Steve Loft lost some of his notes, and the Cumulus 1 version history in his changes.txt is incomplete. So it is pure guesswork that version 1.4 resolved the problem of losing daily information on closing Cumulus by introducing a 'today.ini file to hold the daily extremes and total records.

Steve Loft did document that at rollover, the daily extreme/average records were "transferred" (there are some differences) from "today.ini" to yesterday.ini. That second file was not vital, but made it easier to display yesterday's extremes on the main screen of the legacy software.

The sub-section below covers end of day actions in more detail, subsequently explaining actions might happen twice a day!

When you close Cumulus, it will write the final values for highs and lows and their timestamps to today.ini as part of the close down process.

End of day actions

End of day actions happen when Cumulus detects that it is processing weather data for the rollover time [either midnight, or 9am (or 10am) depending on your configuration and season], this might be during the processing of archive data while Cumulus is catching up after it has been restarted, or in normal running when those clock times are reached.

Oversimplifying the process a bit, the contents of today.ini, goes to two places:

  1. yesterday.ini
  2. a new line appended onto dayfile.txt

After that a new today.ini is created populated with initial values for each extreme/total entry.

The three files have a few differences in content, so rolling-over does involve a little editing work:

  1. The multiple lines in the [General] section of today.ini shrink to just one item in yesterday.ini (Date) and one item in dayfile.txt (abbreviated date)
  2. The 'Total' and 'Samples' values in "today.ini" become the single 'AvgTemp' in "yesterday.ini" and "dayfile.txt"


Complexity if you are recording sunshine hours

Sunshine hours are recorded starting at midnight, regardless of what rollover time is used.

For MX only, the sunshine hours are partly recorded in yesterday.ini and partly in today.ini. So even if your rollover is 9 am, there will be an update to both files at midnight which is when the current sunshine hours figure in today.ini is copied to yesterday.ini before the figure in today.ini is reset to zero.

Where are the files stored?

The files today.ini and yesterday.ini are located in the "data" sub-folder of the path where your cumulus.exe (or CumulusMX.exe) is installed.

(NOTE: Microsoft Windows Operating Systems, may relocate some files, as explained at FAQ on location of data log files).


When Cumulus is left running, a daily backup of all the files is created as part of this rollover in a subfolder 'daily' of the backup folder. Depending on the release you are running, the files included in the backup may be a snapshot of their content just before rollover, just after rollover, or at the first standard interval after rollover.

How Cumulus updates the file

This depends on which flavour of Cumulus you run. The end of day process is same for both flavours, but updates during the day work differently for the legacy software and for MX.

For today.ini, it is important to be aware that MX can read a file created by Cumulus 1 (MX can read the time format in the value for Timestamp= parameter whether it is in Cumulus 1 or MX format), but Cumulus 1 cannot read a file that has been updated by MX.

Cumulus MX and the legacy software handle the updating of "today.ini" differently, this difference is critical should you want to edit out rogue data.

How Cumulus 1 updates the file

Steve Loft never shared his souce code, so what follows is just a guess at how the update might work. The frequency at which data is read from a weather station varies depending on the type, but is at least every 30 seconds.

The indications are that Cumulus 1 has an internal one minute timer that triggers the logging of readings (after conversion to units selected) to an internally held recent history database and the comparison of those readings against existing internally held extremes/totals. The file is updated immediently afterwards, each update only changes those lines (within sections) where the extreme/total/count/time has changed, and other lines retain the same content as before the update.

Whilst you are strongly advised not to manually edit the file with Cumulus running, because access to the file cannot be shared, if you are able to complete the edit between one real-time interval and the next, any change you make is retained.

(NOTE: A full set of latest spot readings are logged to a file at a configurable interval that might be every 10 or 30 minutes).

How MX replaces the file

Note the subtle difference in this sub-section header. MX does all updates to the today extremes/totals/counts only to internally held values. Although the source code is available, you need to be more technical than the person typing this to understand at what frequency these internal updates occur. It is probably every time data is read from the weather station, but might be just when the externally stored recent history database is updated.

Critically, MX only updates the "today.ini" file at the configurable Standard interval used for logging the spot values (might be every 10 or 30 minutes). Each update is a rewrite of the entire file contents, from the internally (RAM) held values.

You would be wasting your time should you try a manual edit of the file while MX is running, as the next MX update will overwrite any manual changes! Don't be misled by the fact that a sharing violation is less likely to be an issue (because of the longer interval between MX updates).

Format of the file

The files are text files, consisting of many lines. Some lines consist of a single piece of text surrounded by square brackets, these are the section names. The sections (for "today.ini" after the first [General]) can be in any order, Cumulus will maintain whatever order the sections are currently in.

Under each section name, there is a list of parameters attribute=value, with one parameter per line. The attribute names are defined by Cumulus, but can appear in any order:

  • Where a time-stamp is stored, note that in "today.ini" only the hour and minute parts of a time are stored.
  • Where an extreme/total value is stored, note that it is always post conversion to the units selected by the Cumulus user.
    • It was mentioned earlier that the file exists to store values that Cumulus holds in RAM. Internally those numbers are in binary (base 2), but in the file the numbers are expressed to base 10. The value of any integer part of numbers is unchanged between the two bases, but decimal parts in base 2 and base 10 do not convert exactly, therefore in the file you may see some strange looking numbers with lots of decimal places.

The key difference between all Category:Ini_Files for the different flavours is the formatting of any time-stamps that include a date:

  •   Cumulus 1 used the format specified in Control Panel for your region settings, for UK that would typically be day/month/year (space) hour:minute in today.ini
  • Cumulus 2 (withdrawn) worked differently, all date/time stamps were converted to UTC, and stored in ISO 8601 format of yyyy-MM-ddTHH:mm:ss(using the net specifiers that MX uses).
  •   Cumulus MX uses the same format as Cumulus 2, but all date/times are expressed according to date/time read from the computer running MX, so use whatever time-zone you have selected on that device.

Please look at the category page (link above) to read more on formatting differences between the legacy Cumulus and MX.

Typical Sections within file

As Cumulus is developed it is adding further sections, but the legacy Cumulus (by 1.9.4) used the following:

  • The [General] section stores current date and time.
  • The [Wind} section stores the highest wind speed and highest gust, it holds the sum of wind speeds as wind run, and details for the dominant wind.
  • The [Temp], or temperature, section stores the highest and lowest temperature, the sum of all temperatures from every sample (Total), and the number of Samples in that total, the Cumulative Chill Hours total (for the season), and the cumulative Heating and Cooling Degree Days for the current day-so-far.
  • The [Pressure], [Humidity], [AppTemp] (for apparent temperature), and [Dewpoint] sections just hold Highs and Lows.
  • The [WindChill] section only holds lowest, the [HeatIndex] section only has highest.
  • The [Rain] section holds a lot of different parameters, including the Start count that derives most rain outputs, and the LastTip date-time.
  • Other sections present in 1.9.4 are [ET], [Solar], [NOAA], and [FineOffset]; whether these are updated depends on what sensors you have, and whether you have set up NOAA reports.


MX development has been far more rapid than the legacy software, and there have been a lot of changes to the content of the two files. Unfortunately, the MX release announcements rarely go into enough detail to permit good documentation, so all mentions relating to MX on this page are guesses from examination of the file, and it cannot be guanteed that the information is correct for whatever MX release you may be running!

A typical MX release will include all the 1.9.4 sections, plus:

  • The [Records] section contains one line denoting when extreme records were last revised. For example "Alltime=2020-03-06T06:42:13", indicates when the all-time extreme records was last updated. In the example file, there were various extreme records broken at that time on 6 March (lowest temperature, lowest apparent temperature, and greatest wind chill), but none have been broken since.
  • Other sections (depending on MX release) may include [FeelsLike], [Humidex], [Lightning], and [TempMidnight]; whether these are updated depends on what sensors you have.


Composite Example of 'today.ini' file

Notes in round brackets () are not part of the file, they simply explain elements of the composite.

  • This is an example made up of composites that may not all be present in an actual file.
  • This composite contains time-stamp formats used by Cumulus 1.x.y and Cumulus MX in the few places where they differ.
  • Because this example is made up of composites, the times shown are not all consistent, in a real file no time anywhere will be later than the time at the top!
  • The order the sections appear in this composite may not match your file; as mentioned above the section order can be edited.
[General]
Date=29/09/2009
Timestamp= (format for Cumulus 1 e.g. "29/09/2009 09:50:00"; format for Cumulus MX e.g. "2019-09-29T09:50:00")
CurrentYear=2009
CurrentMonth=9
CurrentDay=13

[Wind]
Speed=10.7008972167969
SpTime=10:09
Gust=22.0114517211914
Time=08:42
Bearing=90
Direction=E
Windrun=63.1526298522949
DominantWindBearing=317
DominantWindBearingMinutes=1041
DominantWindBearingX=-3914.11743164063
DominantWindBearingY=4215.82763671875

[Pressure]
Low=1014.89996337891
LTime=11:13
High=1018.79998779297
HTime=00:06

[Rain]
High=0
HTime=00:00
Start=1923.59997558594         
Yesterday=0                    
LastTip=2009-09-14 10:48
HourlyHigh=0
HHourlyTime=00:00
ConsecutiveRainDays=2
ConsecutiveDryDays=0
RG11Today=20

[ET]
Annual=1148.2578125
Startofday=1147.24182128906

[Temp]
Low=8.30000019073486
LTime=01:16
High=16.8999996185303
HTime=11:41
Total=7500.697265625
Samples=714
ChillHours=3147.15673828125
HeatingDegreeDays=5.34738397598267
CoolingDegreeDays=0.502222061157227

[HeatIndex]
High=16.8999996185303
HTime=11:41

[AppTemp]
Low=5.0417857170105
LTime=01:30
High=15.0359125137329
HTime=11:52

[WindChill]
Low=6.39816427230835
LTime=01:30


[Dewpoint]
Low=5.30104923248291
LTime=00:52
High=10.7219848632813
HTime=11:38

[Humidity]
Low=65
High=88
LTime=11:45
HTime=06:06

[Solar]
SunshineHours=1.08333301544189    (This is Cumulus 1 example)
SunshineHoursToMidnight=5.80002069473267  (This is Cumulus 1 approach)
HighSolarRad=1048
HighSolarRadTime=09:41
HighUV=7.40000009536743
HighUVTime=09:41
SunStart=0

[NOAA]
LatestMonthlyReport=NOAAMOSep2012.txt
LatestYearlyReport=NOAAYR2012.txt

(The above section will look like the one below if NOAA reporting has not been set up)

[NOAA]
LatestMonthlyReport=
LatestYearlyReport=


[FineOffset] (if not using a Fine Offset station, the MX defaults are as shown)
FOSensorClockTime=(the format here is different for Cumulus 1 e.g. "29/09/2009 09:50:00" and Cumulus MX e.g. "0001-01-01T00:00:00")
FOStationClockTime=(the format here is different for Cumulus 1 e.g. "29/09/2009 09:50:00" and Cumulus MX e.g. "0001-01-01T00:00:00")
FOSolarClockTime=0001-01-01T00:00:00 (this parameter only appears in later MX releases, the time shown is the default if no solar sensor)





Non-essential more technical information

You can skip the following subsections unless you have a particular need to read them.

First use of Cumulus

When you use Cumulus software for the very first time, it records a start date, and assumes you have no data anywhere for earlier than that start time.

Some weather station types have an internal logger that can be accessed for historic data. It is theoretically possible, but not recommended, to read in that historic data, so it is included in your Cumulus extreme tracking. The minimum content for "today.ini" is the "[General]" section. If you stop Cumulus, manually edit "today.ini" to remove all other sections, and then restart Cumulus, then the software will attempt to read that historic data. Please read FAQ here for full guidance. Remember this only applies when you are first starting use of Cumulus with a weather station.


Restart and Catch-up

If you restart Cumulus during the day it will read the today.ini file at startup, so it can resume tracking extremes of the key parameters starting from latest stored values in today.ini. If your weather station type permits, a restart of Cumulus can go into a catch-up mode during which it reads any historic data from the weather station, for the period while Cumulus was not running, before starting the reading of the current data.

During that catch-up of historic data, internally held daily extremes, and the "today.ini" file, will be updated with revised highs and lows as Cumulus processes the historic data from the station's memory; and if necessary Cumulus will do a roll-over (see #End of day actions) as it processes the readings for the relevant time.

On restart Cumulus writes a backup of today.ini (and some of the other Cumulus files) into the backup folder found below the folder with the cumulus.exe (or CumulusMX.exe). With Cumulus stopped, you can copy the today.ini file in either a restart backup, or a daily backup (see #Where are the files stored?), also copying the other files in that backup folder into their original folders (mostly data sub-folder) overwriting the files in those destinations. When you restart Cumulus, the tracking will begin again as if the time has been rewound back to the date those copied files were last updated. This rewinding works best if historic data can be read from your weather station. One example of when this might be useful is if you spot a rogue value very soon after it has been recorded, the rewinding often brings in correct data for the recent period. Another example is if your computer on rebooting initially shows the wrong time and so Cumulus records some data against that wrong time; a rewind can eliminate the wrongly timed records and replace them with records timed correctly.


You are strongly advised not to stop/restart Cumulus close to either midnight or your rollover time. Steve Loft defines "close" in this context as within whatever time you have set as interval between logging of spot values (e.g. 10 or 30 minutes). The potential problems were significantly worse for earlier versions of Cumulus 1, but restart problems have been reduced in newer builds of C1. In general, MX is more tolerant over restart timings, but the way its code works you will encounter more problems with accuracy of output if MX is stopped for more than the few minutes needed for an upgrade (or computer reboot).

Editing rainfall in today.ini within Cumulus

This sub-section applies to Cumulus 1 and Cumulus MX.

At the time of writing, no Cumulus release contains an editor for today's daily extreme records as stored in "today.ini". There is just one exception, both flavours do contain an editor for 'Today's rain'.

Cumulus internally stores a "start of day rain counter". This is copied into "today.ini" into the Start= line within the '[rain]' section. Subtracting that from the current rain counter value allows Cumulus to calculate (in your chosen units) the toatl rainfall for today. The provided editor changes the internal "start of day rain counter" so that the current rainfall total becomes what you enter into the editor.

It is important to note:

  • The rainfall counter was invented in Cumulus 1 as a way that could calculate rain when the software was made to serve multiple weather station types
  • The counter is based on a piece of data that can be read from a particular weather station type, it might be based on an all-time total rainfall, or based on a rainfall this year figure
  • The counter is not intended to be of any interest to the Cumulus user
  • Any increase in the rainfall counter is treated as representing valid rain (spike removal functionality exists only for rain rate and rain in last hour, not the total rain)
  • Cumulus has special code to detect if the counter value decreases, without going into the complexity, Cumulus will normally reset its internal start of day counter based on the decrease in the number received.
  • The "Today's rain" editor does not affect any derived rainfall output;
    • The following in "today.ini" are not affected:
      • last tip time-stamp ("LastTip=")
      • highest rainfall rate so far today and time-stamp ("High=" and "HTime=")
      • hourly high amount and time-stamp ("HourlyHigh=" and "HHourlyTime=")
      • Highest 24 hour amount
    • Outside "today.ini" it does not affect any recent records nor monthly, yearly, monthly-all-time, or all-time extremes:
      • The recent history entries whether held internally (Cumulus 1), or for MX in Cumulusmx.db, are not amended so any tags used for web page data, local API, custom SQL, MQTT, HTTP (see this Wiki section) will report incorrect values
      • Total Rainfall this month, rainfall this year/season will only be recalculated if Cumulus is stopped and restarted after the edit
      • Highest rainfall rate, highest hourly rainfall, highest 24-hour rainfall, highest daily rainfall; applying to these longer periods are not changed unles each extreme record is individually manually edited (Cumulus provides an editor for most of these, but cannot generally suggest what new value to use).
      • The consecutive wet/dry days ("ConsecutiveRainDays=" and "ConsecutiveDryDays=") in this month/year - remember these counts are to last roll-over, they exclude today
    • The editor does not alter any lines already logged in log file for the current month. This means you will see distorted graphs attempting to portray rainfall within a day
    • The editor does not update any database records that may be affected by the change of the rainfall total


Dealing with rogue values

The "today.ini" file is written to throughout the current day, as described earlier. As mentioned above, Cumulus only provides extremely restricted ability to edit "Today's rain", not any other extreme/total that is stored in the "today.ini" file.

If your weather station reports a rogue value, an incorrect update to a High, Low, or Total, may get stored in this file. At end of day, it will then be stored in dayfile.txt. Cumulus developer advice is that instead of manually editing "today.ini", you should wait until the day has been stored in "dayfile.txt" and then use the editor for that log file to make the desired changes.

The rogue value may also affect extreme records held for this month, this year, monthly-all-time, and/or all-time. Cumulus does provide editing functionality for most (not all) entries in the files holding those extreme records and you can read instructions on Correcting Extremes Wiki page. Here it is sufficient to say it is worth looking in the diagnostics, to see if you can spot when the problem occurred, because that helps you work out what may be affected:


Manual editing of "today.ini"

The developer advice to wait until next day has a flaw; every process that happens in the meantime sees wrong data, and that might include sending data to several external sites, and a number of extra custom processes you might have in your system.

Therefore this Wiki page will now give some advice on how to manually edit the file:

  1. You must stop Cumulus (please see earlier in this page for details as reason depends on flavour you are running)
  2. Always take a back-up of existing file, or (if you decide it is easiest to create a new file rather than edit the existing one) rename it; your corrections may cause problems, so you must be able to revert
  3. Any plain text file editor can be used (that includes coding editors like Geany, Notepad++, NoteTab, and many others)
  4. Be careful to ensure any change maintains existing format (integer, decimal,time, date/time, text) paying attention to any punctuation (including decimal commas or decimal points, direction type of any slashes)
  5. For readability you can insert blank lines into files today.ini and yesterday.ini, Cumulus will not mind.
  6. It is up to you to work out what new value/time to type, Cumulus won't accept nulls, but there are some "initial values/dates" that it will accept (I won't tell you these here, because they depend on release you are running and you should not be editing this file unless you have enough technical understanding to work out what Cumulus will accept)
  7. Remember changing extremes for source value extremes and derived extremes is complicated:
    • Please see specific advice in sub-sections below

Edits related to pressure

Should be straight forward, the maximum/minimum can be edited without this affecting anything else. You should be able to look in either log file or for MX in Cumulusmx.db for pressure readings earlier or later in the day to find the new extreme value/time to replace the rogue pressure value and time.

Edits relating to Temperature

  1. If you change a temperature source extreme (i.e. highest/lowest temperature) you should also be taking the old value out of 'Total' line and decreasing by one the'Samples' count
  2. You cannot work out from any change in a temperature source extreme (i.e.. highest/lowest temperature) how to change the related derived extremes (e.g. wind chill, apparent temperature, feels like, dew point). Each derived value is worked out by combining spot values at a particular time, so you have to recalculate as many spot derived values as possible in order to work out the new derived extreme.

Editing Chill Hours

Prior to release 3.12.0, the web template "thisyearT.htm" included a single cumulative figure for seasonal chill hours, and that figure was taken from the chill hours figure stored in "today.ini". A lot of processing was needed to calculate this figure so, in these earlier releases, it was unusual for anyone to correct any rogue figure.

From release 3.12.0 onwards, yesterday's cumulative total is stored in "yesterday.ini". Once you have corrected any rogue maximum or minimum temperature, you can look at the outdoor temperature reported in every row in Cumulusmx.db since last rollover. For each figure that is below your chosen threshold temperature, add the fraction 1/60 (assuming interval between rows is one minute). The cumulative totals of those fractional increments are added to obtain the figure that should be in today.ini. Unless today is the first day of the month set as first month of season, add on the figure stored in yesterday.ini, to get value to put in today.ini.


Edits related to humidity or wind speed

Obviously changes to humidity, wind speed, also have an effect on derived temperature extremes mentioned above, but again you have to recalculate as many spot derived values as possible in order to work out the new derived extreme.


Edits related to rainfall

If you want to edit rainfall since midnight in today.ini, it should be fairly easy to calculate a new value. Remember you also need to manually edit entries in log file and for MX in Cumulusmx.db.

To edit the "rainfall counter" in log file remember that for any decrease in the actual rain, you need to increase the rain counter, and vice versa. It is probably easiest to work backwards through time working out the alteration to the rain counter for the actual rainfall increment, in the period since the preceding entry.


Other derivatives like "maximum hourly rain", "highest rainfall rate", or "highest 24 hour rainfall", all will have to be guessed or recalculated manually, there is no easy way to work these out as past values for the day are not logged anywhere.

Cross-references for explanations of key parameters

Bearing and Direction, Windrun;

Temp, Total and Samples, Humidity, Heating degree days, Cooling degree days, AppTemp, Heat index, wind chill, Chill hours, Dewpoint;

Rain, ConsecutiveRainDays, ConsecutiveDryDays;

NOAA


Changes to this file at particular releases

This section may not be a complete history, and may not be kept up to date; so don't take it as authoritative

  • 3.18.0 build 3189 (pre release 14 June 2022)
    • Added TempMidnight section for tracking of min/max temperature based on day starting at midnight, to complement tracking starting at rollover time
  • 3.17.0 build 3184 (released 23 May 2022)
    • Some changes to handling of lighning (regarding when no strikes)
  • 3.12.0 beta build 3134 (released 29 July 2021)
    • Added Lightning section (lightning distance and last strike time)
  • 3.11.0 build 3129 (released on 7 May 2021)
    • Fix: End of day backup now always runs at rollover, so like most releases of the legacy software, the stored file represents the true end of day position.
      • For example in the [General]' section, the Date (calendar date) and Timestamp attributes will relate to when it was last updated, but the three Current attributes will relate to the meteorological date just ended.
      • For the today.ini in the backup/daily folder, last update time-stamp will indicate time as at one update cycle before end of day, but file last modified will show that was at end-of-day
    • New: Added (to [Temp]) GrowingDegreeDaysThisYear1=, and GrowingDegreeDaysThisYear2=.
  • 3.7.0 - build 3089 (released on 28 July 2020)
    • Canadian Humidity Index (Humidex) added for highest extreme monitoring in today.ini (and other longer period extreme log files)
  • Release 3.6.6 build 3082 (released on 1 June 2020)
    • Values in the file had been stored (in all previous builds) using 15 decimal places (because of difficulty in representing decimal parts of numbers in binary), from this release revised so all values stored using 17 significant figures
  • Release 3.6.3 - build 3079 (released on 21 May 2020)
    • Fix long standing problem with today.ini becoming corrupted when Microsoft Windows is shutdown
  • Release 3.6.0 build 3076 (released on 4 May 2020)
    • Added Feels Like section to daily high and low extremes monitored in today.ini (and other longer period extreme log files)
  • Release 3.1.0
    • Added Records section, this holds one attribute "Alltime" that holds a datetime stamp recording the most recent update to the alltime.ini file
  • From version 1.9.4 build 1089 to release 3.10.5 build 3122
    • The today.ini stored in the daily backup contains the position as at start of day (or in specified older MX releases, up to one UPDATE interval later), so it does not contain any information (except Cumulative Chill Hours) that relates to day that has just ended.
  • Up to version 1.9.4 build 1088 (released 28 Jan 2014)
    • The today.ini stored in the cumulus\backup\daily\FOLDER_NAME ((where FOLDER_NAME is based on date and time of creation) represented the end of day (including on last day of month) extremes
  • version 1.8.9 (released on 31st March 2010)
    • Fix problem with month names (timestamp in today.ini) using short date format
  • version 1.8.6 (released on 14th April 2009)
    • Don't write today.ini unless station contacted
  • version 1.8.5 (released on 12th February 2009)
    • Fix wrong times for some of today's extremes
  • Not documented
    • It is believed that today.ini may have been introduced at version 1.4
  • Version 1.0 27th January 2004 First release
    • Included internal tracking of daily extremes, and display on main screen, but not storing in a file.