5,838
edits
Daily Summary: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
→DAILY SUMMARY
SaratogaWX (talk | contribs) m (Text replacement - "http://wiki.sandaysoft.com/" to "https://cumuluswiki.wxforum.net/") |
m (→DAILY SUMMARY) |
||
| (3 intermediate revisions by 2 users not shown) | |||
| Line 2: | Line 2: | ||
= DAILY SUMMARY = | = DAILY SUMMARY = | ||
[[Category:WebTools]] | [[Category:WebTools]][[Category:User Contributions]] | ||
This page describes routines that read [[Dayfile.txt]] log file, and display a selected part of its content in a friendly way. | |||
== Introduction == | == Introduction == | ||
| Line 8: | Line 10: | ||
# A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is [http://www.fordingbridgeweather.co.uk/datasummary.php here]; note that example uses a different carrier page (see below) from the Weather Blue set of Alternative Templates available from the French only site [http://meteoduquebec.com/ Météo du Québec] and the international site [http://weatherbyyou.com/demo.php <strong>Weather</strong> <small>by</small><sub>You</sub>]. | # A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is [http://www.fordingbridgeweather.co.uk/datasummary.php here]; note that example uses a different carrier page (see below) from the Weather Blue set of Alternative Templates available from the French only site [http://meteoduquebec.com/ Météo du Québec] and the international site [http://weatherbyyou.com/demo.php <strong>Weather</strong> <small>by</small><sub>You</sub>]. | ||
# This article is called '''Daily Summary''' as it summarises scripts that various authors have contributed that take as their source daily statistics output from Cumulus | # This article is called '''Daily Summary''' as it summarises scripts that various authors have contributed that take as their source daily statistics output from Cumulus | ||
#* Typically these scripts allow the | #* Typically these scripts allow the maximum, minimum, average, or total of a statistic calculated for an individual day to be seen in a table that shows such values for many days often spanning at least one year. | ||
# The scripts mentioned below range quite widely: | # The scripts mentioned below range quite widely: | ||
* Some are direct descendants of David Jamieson's original idea, in that they read the ''[[dayfile.txt|daily summary]]'' log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another | * Some are direct descendants of David Jamieson's original idea, in that they read the ''[[dayfile.txt|daily summary]]'' log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another | ||
| Line 35: | Line 37: | ||
*The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote '''fdata.split("\r\n");''' i.e. ''defines that carriage return then line feed (newline) used to split daily observations in dayfile''. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code. | *The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote '''fdata.split("\r\n");''' i.e. ''defines that carriage return then line feed (newline) used to split daily observations in dayfile''. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code. | ||
* See more background in forum thread [ | * See more background in forum thread [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=8600 AnnualDataSummary PHP]. | ||
=== Configuration Variables === | === Configuration Variables === | ||
| Line 82: | Line 84: | ||
The instructions in [[AnnualDataSummary| David's original]] article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant. | The instructions in [[AnnualDataSummary| David's original]] article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant. | ||
*Download the following zip file...[https://cumuluswiki. | *Download the following zip file...[https://cumuluswiki.org/files/AnnualDataSummary_12.zip AnnualDataSummary_12.zip], the zip contains 3 files: | ||
*#'''readDayfile.php''' - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style | *#'''readDayfile.php''' - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style | ||
*#'''datasummary.php''' - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) ''Settings.php'' and ''common.php''. You can use your own page instead of this file - see [[#Creating_your_own_Carrier_Page|carrier page customisation]] below. | *#'''datasummary.php''' - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) ''Settings.php'' and ''common.php''. You can use your own page instead of this file - see [[#Creating_your_own_Carrier_Page|carrier page customisation]] below. | ||
| Line 139: | Line 141: | ||
=== Styling === | === Styling === | ||
The table styling is completely configurable using the included [[Stylesheet|Cascading Style Sheet]] datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see [ | The table styling is completely configurable using the included [[Stylesheet|Cascading Style Sheet]] datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=5319 support forum thread about colouring with php] | ||
In the 'datasummary.css' file: | In the 'datasummary.css' file: | ||
| Line 177: | Line 179: | ||
Finalised version of script in forum 7:08 am 15 Jan 2014, minor update 8:51 pm 15 Nov 2014 | Finalised version of script in forum 7:08 am 15 Jan 2014, minor update 8:51 pm 15 Nov 2014 | ||
Find my package in [ | Find my package in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=8690 alternative Javascript/jQuery solution] within the '''3rd party tools''' sub forum of the Cumulus forum. | ||
I originally created the alternative Javascript version in December 2011 by editing the DAJ original (with similar aims to BCJKiwi with his PHP version 1.2) to offer easier customisation plus standards compliance, i.e. addressing all the issues [[AnnualDataSummary#Possible_problems|with David's version]]. If you don't use PHP, this is the script you should install, and you will find various threads on the forum about it in the Web site - General sub-forum as well as posts in the thread containing the source. | I originally created the alternative Javascript version in December 2011 by editing the DAJ original (with similar aims to BCJKiwi with his PHP version 1.2) to offer easier customisation plus standards compliance, i.e. addressing all the issues [[AnnualDataSummary#Possible_problems|with David's version]]. If you don't use PHP, this is the script you should install, and you will find various threads on the forum about it in the Web site - General sub-forum as well as posts in the thread containing the source. | ||
| Line 195: | Line 197: | ||
=== Using JavaScript === | === Using JavaScript === | ||
As well as the alternative JavaScript version above, SFWS produced a few other scripts that were developed from David's original script including [ | As well as the alternative JavaScript version above, SFWS produced a few other scripts that were developed from David's original script including [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=8631&hilit=one+year+ago#p71709 Comparing with one year ago], a script that allowed you to compare statistics from yesterday (in Cumulus' yesterdayT.htm style) with those from a year earlier (or any other past date). The forum thread includes feedback and consequent changes. | ||
=== PHP scripts === | === PHP scripts === | ||
| Line 201: | Line 203: | ||
Most of these are found in the Website Development section of the Cumulus support forum - '''Web general''' sub forum: | Most of these are found in the Website Development section of the Cumulus support forum - '''Web general''' sub forum: | ||
* [http://saratoga-weather.org/legacy-scripts/dayfile.txt Source: Parse dayfile.txt PHP no database] - This script by the late Kevin Reed who ran the web site TNETweather.com, it is a no frills reader written in PHP (change '.txt' into '.php' when you download it) that parses dayfile.txt and produces an output table with headings. The script can be downloaded on Ken True's site from the link quoted, but see the forum reference [ | * [http://saratoga-weather.org/legacy-scripts/dayfile.txt Source: Parse dayfile.txt PHP no database] - This script by the late Kevin Reed who ran the web site TNETweather.com, it is a no frills reader written in PHP (change '.txt' into '.php' when you download it) that parses dayfile.txt and produces an output table with headings. The script can be downloaded on Ken True's site from the link quoted, but see the forum reference [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=2419#p20155 Forum: Parse dayfile.txt PHP no database] for the only further information still available. | ||
*[ | *[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=5319 'Annual Data Summary - Coloured Values'] - An extensively coloured version of David's original script modified by Mark Crossley with some useful discussion on the trade-off between testing values and efficient code here. The source link in that thread no longer works, but (at time of typing this) the source may be found [http://www.komokaweather.com/weather/datasummary-colour2.php?view=sce here on Paul's site]. | ||
*[ | *[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=11133 'Annual data Summary (DAJ script) - show day-of-week'] - This is just one of several places on the forum where further ideas are explored by Mark Crossley and beteljuice. | ||
*[ | *[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=11397&start=405#p91488 'Yet Another Dayfile Reader (PHP)'] - Download from here the ultimate PHP parsing dayfile.txt version that has progressed a long way from David's original, it incorporates all the ideas discussed in earlier forum topics, some of the features are: the highlighting makes day of week appear in row headers, you can swap between daily and seasonal views and there is a table summary including averages. See that forum url for links to some of the many sites that have adopted this version, if you do not have a database, but do have PHP, then this is version for you. Mark Crossley has his own offshoot of this script, [https://weather.wilmslowastro.com/datasummary.php?view=getorfmiland data summary], that reads a database table, but uses much of the same script so produces a similar summary. | ||
* [ | * [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16891#p130117 'Re: New Script that displays a selected daily statistic for every day'] - In [[#Enhancement of JavaScript script]] above, [[User:Sfws|Sfws]] refers to a pure PHP version that reads data from a database table. That version is now available in the linked paragraph, but using a site specific schema. | ||
* [ | * [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16891#p129796 'New Script that displays a selected daily statistic for every day'] - Download from here, in the above topic, the newest (at time of writing) way to display daily statistics. This is the "daily pick" script where you select one criterion and all the daily values available are output, for ease of display there is a separate table for each month with HTML <detail> tags to show/hide. Both this and previous scripts were written for a schema that is different to the David Jamieson/Mark Crossley/Cumulus MX standard, but the daily pick script has both schemas available, comment out the one you don't want and remove comments round the one you do want. See further information in posts in the topic and the topic that it follows on from, at [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16880 MySQL Code Help]. | ||
*[ | *[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=15983 'SQL dayfile data Viewer'] - This is another (widely adopted) PHP script for reading daily values from a database. It displays a number of key statistics, and you can order the results by the contents of any column. Read the forum topic for more information. You can scroll through the days or set a filter to only show certain days. There is a summary line, but because some columns include units, the calculation of the summary for those columns fails. It is quite simple to edit out the units, so all columns show a summary, but most users adopt the script as published and just accept that some summary figures are replaced by text. | ||
== Displaying daily statistics from a database table == | == Displaying daily statistics from a database table == | ||
| Line 222: | Line 224: | ||
#* Database designers talk about the ''schema'' meaning the specification that describes the design of the table. | #* Database designers talk about the ''schema'' meaning the specification that describes the design of the table. | ||
#* A schema lists the column names together with what type and size of variables they hold. | #* A schema lists the column names together with what type and size of variables they hold. | ||
#* For the daily summary database table, the columns can copy the fields in the [[dayfile.txt|daily summary log file]], and this is what David Jamieson does in his approach [ | #* For the daily summary database table, the columns can copy the fields in the [[dayfile.txt|daily summary log file]], and this is what David Jamieson does in his approach [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=2367 a PHP script to read the contents of the dayfile.txt] where the columns are generated in the same order as the corresponding fields in the log file, so David's script uses the field numbers to identify the columns by Position (rather than by Column Name). | ||
#* Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39). | #* Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39). | ||
# If you have the skills to write your own script, then you might want to consider a different schema, daily information is in three log files for Cumulus 1, "dayfile.txt" should have the statistics for all completed days (that excludes today), "today.ini" has values for today-so-far, and "log.xml" is the Weather Diary where snow records (and general weather text) can be manually recorded. A schema that combines information extracted from those three files to show everything Cumulus has for a day is available in [ | # If you have the skills to write your own script, then you might want to consider a different schema, daily information is in three log files for Cumulus 1, "dayfile.txt" should have the statistics for all completed days (that excludes today), "today.ini" has values for today-so-far, and "log.xml" is the Weather Diary where snow records (and general weather text) can be manually recorded. A schema that combines information extracted from those three files to show everything Cumulus has for a day is available in [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16880#p129719 this forum post]. | ||
=== Loading and Updating the database table === | === Loading and Updating the database table === | ||
Obviously if you are using a script that reads statistics from a table, you also need to be running a script that ensures the table is updated at the end of each day, and therefore always up to date. | Obviously if you are using a script that reads statistics from a table, you also need to be running a script that ensures the table is updated at the end of each day, and therefore always up to date. | ||
* Functionality is included in Cumulus MX to do this update, it uses the schema devised by Mark Crossley by default, but Cumulus MX can also handle your own schema; for full details of the functionality see [ | * Functionality is included in Cumulus MX to do this update, it uses the schema devised by Mark Crossley by default, but Cumulus MX can also handle your own schema; for full details of the functionality see [https://cumulus.hosiene.co.uk/viewtopic.php?f=27&t=12908&start=15#p107639 Cumulus MX Announcements - PLEASE READ THESE!]. | ||
* Cumulus 1 users need to use a script to create a database table, populate and update it from the daily summary log file. The most widely used script is an update by Mark Crossley of the script mentioned above by David Jamieson; and it is described in [[ImportCumulusFile|this Wiki article]]. | * Cumulus 1 users need to use a script to create a database table, populate and update it from the daily summary log file. The most widely used script is an update by Mark Crossley of the script mentioned above by David Jamieson; and it is described in [[ImportCumulusFile|this Wiki article]]. | ||
** It is however worth also reading the forum post [ | ** It is however worth also reading the forum post [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=2367 a PHP script to read the contents of the dayfile.txt] as that explains the date delimiter, how a Comma Separated Variable file may use other punctuation to separate the fields, how to specify the location of your dayfile.txt log to the script and the reason the key variable is used. The thread contains several insights into how Cumulus was developed and it is a definite must to read Steve's posts in that thread if you are developing your own scripts. | ||
=== Some example Scripts === | === Some example Scripts === | ||
| Line 239: | Line 241: | ||
Look in the Website Development section of the Cumulus support forum - '''Web general''' sub forum: | Look in the Website Development section of the Cumulus support forum - '''Web general''' sub forum: | ||
* Look at the Javascript and PHP scripts already listed for showing raw values, but some others present a summary view: | * Look at the Javascript and PHP scripts already listed for showing raw values, but some others present a summary view: | ||
**[ | **[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=10755#p87081 Top 10 Records Page]- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. ). The latest source php can be seen using a [https://weather.wilmslowastro.com/test/top10s/top10sContent.php?view=sce query-string on Mark Crossley's web site] and there have been several updates since the forum thread was created. | ||
**[ | **[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16534#p126964 What to do with data from MySQL] - This posting lists a number of web pages on PaulMy's web site where he presents statistics from the daily summary table of his database in different formats. | ||
**[ | **[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=7834#p66013 Comparing with one year ago] - This post has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, to display alongside for example statistics for yesterday. An equivalent script using jQuery to read dayfile.txt directly is available for those who don't have a database. If you have some programming skills, you can vary either of these scripts to for example display statistics for each day in the last week (or month). I have written scripts that display the statistics for last week using a combination of this daily table in database approach and Cumulus recent tags. I have another script that uses the daily table in the databse to calculate the equivalent of what Cumulus 1 shows in View menu, this month selection, for any past month and adds the equivalent of what Cumulus this month web tags show and the resulting statistics are shown for various selectable mixtures of past months. | ||
There are '''various scripts out there that output graphs''' from the values in recent history web tags, or values read off either a daily table in a database or dayfile.txt directly. There are two to highlight here: | There are '''various scripts out there that output graphs''' from the values in recent history web tags, or values read off either a daily table in a database or dayfile.txt directly. There are two to highlight here: | ||
* An alternative to the Cumulus 1 or Cumulus MX trend web page is described in [ | * An alternative to the Cumulus 1 or Cumulus MX trend web page is described in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=12321 this topic] this can use all the sources mentioned. | ||
* Highcharts (that cannot be used on a business web site) are exploited by Mark Crossley in [ | * Highcharts (that cannot be used on a business web site) are exploited by Mark Crossley in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=8672 this topic]; it is a very long topic, and discusses various issues to consider with using such data, as well as including the script. | ||
If you find a site displaying a graph or table you would like to copy on your own site, you might be able to see the PHP source that is responsible, but as I have commented [ | If you find a site displaying a graph or table you would like to copy on your own site, you might be able to see the PHP source that is responsible, but as I have commented [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16425&p=126065#p126065 elsewhere] most authors do not make it easy to see their code. Of course one of the reasons for using PHP script is that processing is done quickly on the web server and only the resulting HTML is passed onto the client browser and this does protect interlectual property and adds security by not showing the database access code. | ||
=== Other jQuery approaches === | === Other jQuery approaches === | ||
| Line 292: | Line 294: | ||
</pre> | </pre> | ||
The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them. The column names used above are from the schema by David Jamieson, as used by Cumulus MX. | The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them. The column names used above are from the schema by David Jamieson, as used by Cumulus MX. | ||
[[Category:User Contributions]] | |||
