Daily Summary: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search

Daily Summary (view source)

Revision as of 09:29, 22 July 2021

5,193 bytes added ,  09:29, 22 July 2021
m
→‎DAILY SUMMARY: paste from another page
m (Text replacement - "http://sandaysoft.com/forum/" to "https://cumulus.hosiene.co.uk/")
m (→‎DAILY SUMMARY: paste from another page)
(4 intermediate revisions by 2 users not shown)
Line 2: Line 2:
= DAILY SUMMARY =
= DAILY SUMMARY =


[[Category:WebTools]]
[[Category:WebTools]][[Category:User Contributions]]
This page describes routines that read [[Dayfile.txt]] log file, and display a selected part of its content in a friendly way.


== Introduction ==
==Options for reading dayfile.txt for other uses==
# This article updates [[AnnualDataSummary|Annual Data Weather article]] contributed by David Jamieson ([http://kippfordweather.uk current web site]) in March 2011.
 
# A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is [http://www.fordingbridgeweather.co.uk/datasummary.php here]; note that example uses a different carrier page (see below) from the Weather Blue set of Alternative Templates available from the French only site [http://meteoduquebec.com/  Météo du Québec] and the international site [http://weatherbyyou.com/demo.php <strong>Weather</strong> <small>by</small><sub>You</sub>].
*Some people take a copy of the local dayfile.txt, and use it locally for other purposes.
# This article is called '''Daily Summary''' as it summarises scripts that various authors have contributed that take as their source daily statistics output from Cumulus
* Various third-party routines that make use of the [[dayfile.txt]] file (and in some cases, others) are listed at [[:Category:WebTools]].
#* Typically these scripts allow the maximimum, minimum, average, or total of a statistic calculated for an individual day to be seen in a table that shows such values for many days often spanning at least one year.
** Some people use [[Upload_Dayfile|the method described here]] or [[Toolbox|Cumulus Toolbox]] to get a copy of the local file to use on their web server. (If your web server is local, you can simply copy the file to that server, once a day).
** Search in the Cumulus support forum for examples of third-party JavaScript projects that read the web copy, for example to insert data for one year ago, or to enable extremes for each day in a week to be included in a web page.
**Dayfile.txt (and other files!) are used as described at the [[:Category:CumulusUtils]] link.
*Other people load the contents of their dayfile.txt into a database table (see sub-section directly below), allowing SQL enquiries to efficiently display, or summarise, the contents of the file.
**Cumulus MX provides optional functionality to create a database table, and to automatically insert a new row each time a new line is added to the local '''dayfile.txt''' file.
 
=== Populating a database table ===
 
*[[File:Badge v1.png]] The [[ImportCumulusFile]] page describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have been edited for MX, so you will need to use an older version of that script that matches the version of Cumulus 1 you are using.
*[[File:Badge vMx.png]] Please see [[MX_Administrative_Interface#Standard_Daily_Summary_Table]] section for details of how '''CumulusMX.exe''' has an optional feature with a standard option to insert a new row into a database table holding columns relating to the dayfile.txt fields. 
** Either (for all early MX releases) '''ExportMySql.exe''', or (for all later MX releases) '''ExportToMySQL.exe''' (see [[Software#ExportToMySQL]]), can update the database table with past rows.
 
=== Using a database table ===
 
Your web site can use the database table, created as described in last sub-section, for a range of queries (from simple to complicated) written in SQL, and sometimes queries are discussed in the support forum.
 
For examples of some of the third party tools (Cumulus [[Category:User_Contributions|user contributions]]) using the database daily summary table described above, see [[#Some_example_Scripts|here]].
 
====Just another example of how much you can get from just one (daily summary) log file====
 
I have a number of web pages, based on the "this month" web page supplied with Cumulus.  Thus I can look at this month, see several consecutive months, compare any month with any other month, and so on.  I also have display just key values, all the values supported by monthly web tags, or even all the extra derivatives I can derive from examining daily summary data for any month.
 
To give me that versatility, with the minimum data transfers between Cumulus and my database/web servers, all the fields I might want to display are calculated after a month ends (and on a number of days during the month) by calling a script that stores these calculated values as columns in another database table that has one row for each month.
 
==== Alternative schemas ====
 
Of course, the schema in your database table does not need to exactly mimic the fields in the dayfile.txt file.
* Perhaps you do not use all of the fields; your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table
* Perhaps you want columns representing values from external sensors or other log files.
 
Alternatively, you might feel that you want to validate all the fields read from the daily summary file, before writing them to your database table to ensure your database only contains good quality data. Equally, to simplify querying your database table, you might want missing data to be stored as real nulls in the database columns, rather than the "zero" or "space" or empty character string, that Cumulus uses in its inconsistent way.  That way you don't get any longer period averages skewed by the invalid zeroes.
 
To implement those changes, you need a little knowledge about Structured Query Language, and you may need to write your own script (possibly based on the script in [[ImportCumulusFile]].
 
*[[File:Badge vMx.png]] MX makes this implementation slightly easier for you. If you are able to create the database table with the columns you want, then you can input the SQL for updating the table directly into MX quoting web tags for the necessary data, and Cumulus can automatically trigger this as part of its end of day routine
*[[File:Badge v1.png]] In the legacy Cumulus 1 software, there is no built in way to create or update database tables.
 
== Third-party contributions ==
 
# This page was inspired by the [[AnnualDataSummary|Annual Data Summary page]] contributed by David Jamieson ([http://kippfordweather.uk current web site]) in March 2011.
# A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is [http://www.fordingbridgeweather.co.uk/datasummary.php here]
#* A different carrier page (see below) is the Weather Blue set of Alternative Templates available from the French only site [http://meteoduquebec.com/  Météo du Québec] and the international site [http://weatherbyyou.com/demo.php <strong>Weather</strong> <small>by</small><sub>You</sub>].
# This page is called '''Daily Summary''' as it summarises scripts that various authors have contributed that take as their source daily statistics output from Cumulus
#* Typically these scripts allow the maximum, minimum, average, or total of a statistic calculated for an individual day to be seen in a table that shows such values for many days often spanning at least one year.
# The scripts mentioned below range quite widely:
# The scripts mentioned below range quite widely:
* Some are direct descendants of David Jamieson's original idea, in that they read the ''[[dayfile.txt|daily summary]]'' log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another
* Some are direct descendants of David Jamieson's original idea, in that they read the ''[[dayfile.txt|daily summary]]'' log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another
Line 26: Line 70:


== Update of David Jamieson PHP script by BCJKiwi ==
== Update of David Jamieson PHP script by BCJKiwi ==
Here is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines.  They also include a couple of updates from DAJ version as advised by posts in the Forum.
Here is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines.  They also include a couple of updates from DAJ version as advised by posts in the Forum.


=== Errors rectified by this version ===
=== Errors rectified by this version ===
*Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row').
*Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row').
*In version 1.1 of this tool, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight; from jQuery version 1.6 that was deprecated and the '.prop' object is now specified for the same purpose.
*In version 1.1 of this tool, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight; from jQuery version 1.6 that was deprecated and the '.prop' object is now specified for the same purpose.
Line 38: Line 84:


=== Configuration Variables  ===
=== Configuration Variables  ===
By default in 'readDayfile.php':  
By default in 'readDayfile.php':  
*  Assume the dayfile.txt is read from a subfolder called 'data'.
*  Assume the dayfile.txt is read from a subfolder called 'data'.
Line 82: Line 129:
The instructions in [[AnnualDataSummary| David's original]] article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant.
The instructions in [[AnnualDataSummary| David's original]] article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant.


*Download the following zip file...[https://cumuluswiki.wxforum.net/files/AnnualDataSummary_12.zip AnnualDataSummary_12.zip], the zip contains 3 files:
*Download the following zip file...[https://cumuluswiki.org/files/AnnualDataSummary_12.zip AnnualDataSummary_12.zip], the zip contains 3 files:
*#'''readDayfile.php''' - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style
*#'''readDayfile.php''' - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style
*#'''datasummary.php''' - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) ''Settings.php'' and ''common.php''. You can use your own page instead of this file - see [[#Creating_your_own_Carrier_Page|carrier page customisation]] below.
*#'''datasummary.php''' - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) ''Settings.php'' and ''common.php''. You can use your own page instead of this file - see [[#Creating_your_own_Carrier_Page|carrier page customisation]] below.
Line 241: Line 288:
**[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=10755#p87081 Top 10 Records Page]- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. ).  The latest source php can be seen using a [https://weather.wilmslowastro.com/test/top10s/top10sContent.php?view=sce query-string on Mark Crossley's web site] and there have been several updates since the forum thread was created.
**[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=10755#p87081 Top 10 Records Page]- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. ).  The latest source php can be seen using a [https://weather.wilmslowastro.com/test/top10s/top10sContent.php?view=sce query-string on Mark Crossley's web site] and there have been several updates since the forum thread was created.
**[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16534#p126964 What to do with data from MySQL] - This posting lists a number of web pages on PaulMy's web site where he presents statistics from the daily summary table of his database in different formats.
**[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16534#p126964 What to do with data from MySQL] - This posting lists a number of web pages on PaulMy's web site where he presents statistics from the daily summary table of his database in different formats.
**[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=7834#p66013 Comparing with one year ago] - This post has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, to display alongside for example statistics for yesterday. An equivalent script using jQuery to read dayfile.txt directly is available for those who don't have a database. If you have some programming skills, you can vary either of these scripts to for example display statistics for each day in the last week (or month). I have written scripts that display the statistics for last week using a combination of this daily table in database approach and Cumulus recent tags. I have another script that uses the daily table in the databse to calculate the equivalent of what Cumulus 1 shows in View menu, this month selection, for any past month and adds the equivalent of what Cumulus this month web tags show and the resulting statistics are shown for various selectable mixtures of past months.
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=8631 Comparing with one year ago] - This post [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=7834#p66013 as requested here] has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, to display alongside for example statistics for yesterday. An equivalent script using jQuery to read dayfile.txt directly is available for those who don't have a database. If you have some programming skills, you can vary either of these scripts to for example display statistics for each day in the last week (or month). I have written scripts that display the statistics for last week using a combination of this daily table in database approach and Cumulus recent tags. I have another script that uses the daily table in the databse to calculate the equivalent of what Cumulus 1 shows in View menu, this month selection, for any past month and adds the equivalent of what Cumulus this month web tags show and the resulting statistics are shown for various selectable mixtures of past months.


There are '''various scripts out there that output graphs''' from the values in recent history web tags, or values read off either a daily table in a database or dayfile.txt directly. There are two to highlight here:
There are '''various scripts out there that output graphs''' from the values in recent history web tags, or values read off either a daily table in a database or dayfile.txt directly. There are two to highlight here:
Line 292: Line 339:
</pre>
</pre>
The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them.  The column names used above are from the schema by David Jamieson, as used by Cumulus MX.
The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them.  The column names used above are from the schema by David Jamieson, as used by Cumulus MX.
[[Category:User Contributions]]
Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/4398...9512"

Navigation menu