Dayfile.txt and Category:Cumulus MX: Difference between pages
m (→List of Fields) |
|||
| Line 1: | Line 1: | ||
<big>'''This article is about the Daily Summary logging file'''</big> |
|||
<big>'''If you have any suggestions for improving this article'''</big>, either edit this page yourself, or put your suggestion in the correct [https://cumulus.hosiene.co.uk/viewforum.php?f=38 Support Sub-Forum]. Thank you. |
|||
{{TOCright}} |
|||
= Introduction = |
= Introduction = |
||
If you have been using Cumulus 1, and now wonder whether to try using MX, be aware that: |
|||
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. In all flavours of Cumulus, this file is only updated (with exclusive lock) during the end of meteorological day process. In that process it is also read if the generation of NOAA reports has been requested. |
|||
*Currently, more people are still using Cumulus 1 than are using MX; but during 2020, many have experimented with MX, then successfully moved to MX, just a few have gone back! |
|||
*Cumulus 1 is a stable release, it has functionality that is not available in MX: |
|||
**Select a graph |
|||
**View period |
|||
**provided you are happy to keep running Cumulus 1 on a Windows pc, and use a weather station that is compatible with Cumulus 1, then you don't have to change over |
|||
*Cumulus MX is still under development, a change in a particular release may introduce bugs |
|||
*MX does however have additional features not available in Cumulus 1 |
|||
**it can run on Unix-derived (Mac, Linux, Raspberry Pi operating systems) devices as well as Windows devices |
|||
**it can work with newer weather stations and newer sensors |
|||
**it reports feels like, Canadian Humidity Index |
|||
**it has built in updating of database tables |
|||
**it can output to more external sites |
|||
**it has MQTT, HTTP, and other extras, built in |
|||
Please see [[Moving from Cumulus 1 to MX]] for more information. |
|||
In Cumulus 1 only, the figures contained in the file are used for the 'This period' display accessed from the '''View''' menu and to build any graphs based on daily values. |
|||
== What does Cumulus MX do? == |
|||
The format of this file is the same for both Cumulus 1 and Cumulus MX, although the number of fields in the file increases in various versions as shown at the end of this article. The file can be ported between flavours only if both are run with exactly the same locale settings, as using a different locale may change the field separator or the symbol used for decimal points. |
|||
{{TOCright}} |
|||
If you are new to Cumulus, then you will be wondering what benefits Cumulus MX has over other weather recording software. In that case start by reading [[About Cumulus|the article that introduces Cumulus]]. |
|||
You may want to read that article first, that that will explain what Cumulus software can do and perhaps help you to: |
|||
== Changes in different releases of Cumulus == |
|||
*Learn what it can offer you if you are unsure whether Cumulus is right for you |
|||
*If you already use Cumulus, this might remind you of everything it offers. |
|||
If you do go off to read that article now, you will be linked back to this page. |
|||
{{Version badge 1}} |
|||
== This article == |
|||
'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round. |
|||
This Wiki article was originally created by Mark Crossley and contained exactly what Steve Loft said in the [https://cumulus.hosiene.co.uk/viewforum.php?f=39 MX early builds support forum] when he first started experimenting with Cumulus MX and access was restricted to those willing to experiment with his tests. |
|||
'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order) |
|||
In this rewrite, somebody who used to use Cumulus 1, and experimented with MX, before moving over to MX first on a Windows PC and then on a Raspeberry Pi, has attempted to ensure this article contains everything I learnt from my experience as I explore more of the functionality of MX; and as I learn more from posts in the forum. |
|||
There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard final release of Cumulus 1 as this section was updated. |
|||
If you can correct any mistakes I make, have a suggestion as to how to make it clearer, can add anything I have not yet covered, or know something that I might not know, then please remember, anyone can update this article, I don't have any special access in the Wiki and any page I edit can be edited/corrected by anyone else. |
|||
[[File:Badge vMx.png]] |
|||
During a period of my time in employment I was responsible for approving documentation on a large computerisation project, and later for supplying updated information for a public faced web site, and in both cases there were house style, and I probably continue to use that style. You might be afraid to add your contribution because my style is not the same as your natural one. Don't worry; as long as you use short paragraphs or bullet points, with lots of headings, then your contribution can blend in. |
|||
'''Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor''' |
|||
As this article has grown, I have been able to shorten it by moving material to new articles, hence you will see a lot of cross-references below. You may have suggestions for what else can be moved out of this article into separate articles? |
|||
'''Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor''' |
|||
If you have some ideas, but are unsure whether to apply changes, please use the discussion page first. If you have ideas, but want someone else to add them here, please write your suggestion in the support forum for wiki suggestions at https://cumulus.hosiene.co.uk/viewforum.php?f=38 |
|||
= When Cumulus is left running = |
|||
= Cumulus flavours = |
|||
* Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day. |
|||
* There are no updates to dayfile.txt at any other times, but (for Cumulus 1 only) the contents of the file are read and processed for many of the display and edit menu options that can be selected from the main Cumulus 1 screen. |
|||
* Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in [[Today.ini]], updating that file as required. |
|||
**What is stored in today.ini is processed into what get stored as next line in dayfile.txt as described in next section. |
|||
*(It also updates [[Alltime.ini]], [[Monthlyalltime.ini]], [[Year.ini]], and [[Month.ini]] when appropriate. |
|||
*(Periodically, the weather readings and derived values are stored in [[Standard log files]] which are set up for each month). |
|||
* Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day. |
|||
* If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original. |
|||
== |
==Cumulus 1 == |
||
*Cumulus 1 was created in 2003 when Steve Loft moved to Wanlockhead and bought himself a weather station. Initial releases were very much tailored to his needs, changing what weather station it supported to match the make he was using. |
|||
*Subsequently, he made his software available to anybody, and enhanced what it did to suit not only his requirements buy also what other users asked for. |
|||
*The functionality of Cumulus grew and grew, there were a few bugs, and a few mistakes, but generally Cumulus software had a high reliability, and grew in popularity, especially when the internet made it easier for people to praise Cumulus on many unrelated sites. |
|||
*Increased popularity meant an increase in demand for more functionality, increased functionality made it more popular, and the spiral of development continued. |
|||
*Cumulus 1 had extensive help screens built into the package, it had an installation package, and produced a main screen when it was running that summarised the weather and gave access to all the settings and editors. This made Cumulus 1 very user friendly. |
|||
*Demand for enhancements soon exceeded the amount of spare time, Steve could devote to Cumulus outside his full-time job. A register of enhancement requests was created, so that Steve could track which he had implemented, but that register was lost when Steve moved to a different host. |
|||
*One widely supported request could not be implemented for Cumulus 1, to offer a version that would run on multiple operating systems, so Cumulus 2 was born (see below). |
|||
*Cumulus 1 development was halted for a while by the focus on Cumulus 2. |
|||
*Cumulus 1 development restarted when Cumulus 2 was aborted. |
|||
*The development environment for Cumulus 1 became obsolete, and Windows operating systems changed, making the development of Cumulus 1 more difficult, so Steve decided to have another go at a replacement (Cumulus 3). |
|||
*Development of Cumulus 1 ceased once Cumulus 3 (aka MX, see below) was able to work better than its C# predecessor Cumulus 2. |
|||
*Steve Loft decided not to release source code for Cumulus 1, so nobody else can develop Cumulus 1 any further. |
|||
*Consequently, Cumulus 1 functionality can not be changed, and without knowledge of how it was written, there is no ongoing support, just the experience of those who have used it, or are still using it. |
|||
*Fortunately, Steve Loft documented Cumulus 1 very well in the forum, and in the new Wiki that was started in August 2009, so Cumulus 1 continued to attract new users even when Cumulus 3 was made available as MX. |
|||
*When this article was first created in 2017, Cumulus 1 was still recommended for most users, and it would remain so for another couple of years. |
|||
*Even now in 2020, '''there are many, many more people using Cumulus 1 than are using MX'''. |
|||
== Cumulus 2 == |
|||
*Steve Loft produced a Cumulus 2 where he tried to start again in September 2009. It was written in C# (which is the language used for MX), and it is fair to say that Steve did not find that new programming language easy, and in March 2010 he was really struggling to make Cumulus 2 work how he desired. |
|||
* Cumulus 2 did prove that a number of concepts (like separating "engine" from "admin interface") could work and it was a useful learning curve for when Steve decided to write Cumulus 3 (see below). |
|||
*One change that had been requested by several Cumulus 1 users was for better international viewing of web pages, with less dependence on time zones. To achieve this, one suggestion was that Cumulus should work in GMT (more widely known now as UTC). Cumulus 2 therefore read and logged all readings by UTC. Unfortunately, converting from local time used by weather stations, and most computer devices, never worked as smoothly as Steve Loft hoped, so this is one idea that was not adopted for Cumulus 3. |
|||
* Furthermore, Cumulus 2 never succeeded in getting some of the basic functionality like driving web pages to work, so it never offered much of the more useful functionality of Cumulus 1. |
|||
*But it was a good testing ground for new functionality and enhancements and regardless of whether they could be made to work fully in Cumulus 2, some were highlighting what Cumulus 1 lacked. |
|||
*In August 2010, the new features being tested in Cumulus 2 were added to Cumulus 1, and Cumulus 2 was discontinued. |
|||
== Cumulus 3 == |
|||
*In 2015, Cumulus 3 also known as MX once it was made available to users, was experimental and it had limited functionality, much less than was available in Cumulus 1. This made MX innovative, but unfriendly. |
|||
*Consequently, at that time, most Cumulus users were using Cumulus 1, and just those wishing to take part in beta testing used MX. |
|||
*Steve Loft started development of MX while he was still in full-time employment, but as retirement approached he worked fewer days per week and was faced with the question as to whether to spend more time on MX or more time with his wife, Beth, exploring places. |
|||
*When he fully retired, a life on the road beckoned, and they started travelling. Work on MX decreased, and work on Cumulus 1 was no longer possible, as he was limited to what his laptop and internet connection at stops could cope with. |
|||
*Various people offered to help him with MX if he was willing to make his source code available. Initially, Steve did not want anyone else to interfere with his creation, but when he and his wife found a new home the priorities changed in favour of a focus on his new life, and he wanted to cease involvement with Cumulus software, its wiki and forum. |
|||
*Steve Loft who wrote and developed Cumulus 1 and MX while he was in Scotland, decided to cease to offer any support from his new home in France. After quite a while considering it, he decided to make the source code available. |
|||
*The various people who had offered to help develop MX now were able to see the source code and decide whether they really did want to get involved. |
|||
*One programmer launched Cumulus 4, a new approach. Work continued on this for a while, but as far as I know it never made it into a working system, and I believe like Cumulus 2, it is abandoned. |
|||
*Other programmers looked at the source and thought we can make MX useful by adding the missing functionality, both what Steve added in the source but never got into a public release; and the functionality that makes Cumulus 1 so popular but is missing in MX and makes it difficult for those who are using MX. |
|||
== The MX future == |
|||
* Mark Crossley was one of those who tried updating the MX source and producing a new release. |
|||
* In 2019, he made a successful first new release, and then focussed on adding some of the missing functionality. By 2020, he was not just adding in his own version of features that had been in Cumulus 1, he was also making MX talk to new weather station designs and deal with new sensors. |
|||
* '''During 2020 much extra functionality has been added to MX''', and MX is now able to persuade [[Moving from Cumulus 1 to MX|Cumulus 1 users]] that it might be the right time to make the swap to MX. |
|||
*Cumulus 1 was designed to work with weather stations that were available when it was written, the technology used by stations, and the models available, have both been changing since then. |
|||
*The ongoing development is adding lots more functionality into MX, it can do a lot more with the the numbers it reads from weather stations, and it can be updated when weather station features change. |
|||
*Therefore, the advice to newcomers is to use Cumulus MX, sometimes called Cumulus 3, because there was a Cumulus 2 (that was abandoned) and sometime ago there was a start on a Cumulus 4. |
|||
*Similarly, the advice to established Cumulus 1 users is you should now consider a move to MX as you are now missing out on many features available only in MX. |
|||
*However, there are no instructions built into the MX package, so it is hoped that the update of this article will help people to understand MX sufficiently to use it both more easily and to maximum capability. |
|||
*Currently, Mark Crossley who has been responsible for all recent MX releases is able to answer questions in the support forum [https://cumulus.hosiene.co.uk/viewforum.php?f=40 for recent MX releases], but this article will hopefully allow him to spend less time answering questions and more time improving MX (and more time for everything else in his life) |
|||
It would be wrong not to repeat what Mark has said here - '''MX is still not bug free, there is a lot more to correct as well as all the enhancements to cope with new weather station hardware'''. |
|||
There is a page (created in October 2018) listing [[MX Issues|MX Issues to be resolved]], but I suspect it is out of date. If you look through the release announcements for 2020, yes there are a lot of new features being added, but there is even more work on resolving bugs. |
|||
== Restrictions on who can use MX == |
|||
MX makes extensive use of library packages: |
|||
* like bootstrap (cascade styling), |
|||
*datatables (display and manipulation of tables), |
|||
*JQuery (JavaScript package that provides code supports for multiple browsers and other libraries to work together), |
|||
*high stock (for drawing charts), |
|||
*datepicker (a JavaScript based routine for making date selection possible using a calendar type interface as not all browsers directly support that), |
|||
*handlebars (templates for generating HTML), |
|||
* alapaca (JavaScript from Gitana Software that generates interactive HTML5 forms), |
|||
*Steelseries (provides the gauges used), |
|||
*altEditor (for editing the log files) and a few more. |
|||
Fuller information on these is in [[#Library_software|Library software]] section. |
|||
Most of these are open software and free for personal use, but some have restrictions on commercial use requiring a licence. Consequently, MX does have to declare it is not for use on a commercial web site. |
|||
=== Message from Steve Loft about who can use MX === |
|||
Note: The graphs used in Cumulus MX are drawn using Highcharts and they are free for non-commercial use only, i.e. you may not use them on a company web site, see http://shop.highsoft.com/faq/non-commercial for clarification. '''For this reason, and others, use of Cumulus MX in a commercial environment is expressly forbidden.''' |
|||
''Please include a link to the Highstock web site (as the supplied web page does) if you use the charts under the terms of the non-commercial licence.'' |
|||
== Documentation for MX == |
|||
=== Cross reference to article on [[Moving from Cumulus 1 to MX]] (includes general issues regarding running on Windows) === |
|||
Whether you have been using Cumulus 1, or are new to Cumulus, you may be tempted to install MX on a PC, or other device, running Windows Operating System; the article linked in the heading will help you. |
|||
You may be reading this article on MX as a Cumulus 1 user considering moving to MX, if so then it explains all issues when moving from Cumulus 1 to MX, including what functionality you will gain and what you will lose by moving to MX. The article will help you to decide whether to stick with Cumulus 1's stable release as many MX releases have bugs as the developer tries revising the code, to simplify it, as well as to introduce new features. |
|||
As a new Cumulus user, using MX on Windows (but with no experience of Cumulus 1), the separate article linked from the heading to this section will still interest you as it is a definitive guide to installing Cumulus MX on Windows. |
|||
=== Cross reference to article on [[Setting up Raspberry Pi]] for Cumulus === |
|||
Whether you have been using Cumulus 1, or are new to Cumulus, you may be tempted to install MX on a Raspberry Pi. A Raspberry Pi (and similar devices from other manufacturers) is much simpler than a normal computer, but it can still run various operating systems that allow you to use it perhaps for both running MX and a web server. Find out more in the article linked in the heading. |
|||
*A new row is appended to dayfile.txt, the values are prepared from reading "today.ini" file, not all values available in "today.ini" are stored in dayfile.txt. |
|||
*Some of this information is also stored in [[yesterday.ini]]. |
|||
=== Cross reference to article on [[Updating MX to new version]] === |
|||
*Back ups of both today.ini, and dayfile.txt, log files ''in their state after the end of day update'' are copied to the 'cumulus\backup\daily' folder, a maximum of only 9 daily sub-folders are retained. |
|||
Whether you have not updated for a long time, or simply wonder whether you are updating the easiest way, follow the link in the heading for this section, to an article that focusses on all to do with updating from one MX version to another. |
|||
Critically, this linked article contains advice '''both''' for those updating each time there is a new release available '''and''' for those using an old version (perhaps when MX was still in beta) wishing to update to a new version skipping some in-between versions and need advice on whether to do it in multiple steps. |
|||
=== Cross reference to [[Cumulus MX FAQ]] === |
|||
A new FAQ for MX has been started at [[Cumulus_MX_FAQ|another page]]. As I add the link here, the Cumulus MX FAQ is in a mess, but hopefully someone will have time to sort it out. |
|||
Meanwhile most MX specific questions are now covered by the updating of text on this "Cumulus MX" page. |
|||
=== Cross reference to [[MX Issues]] === |
|||
The [[MX Issues|Cumulus MX Known Issues]] article was based on [https://cumulus.hosiene.co.uk/viewtopic.php?f=39&t=12943 Steve Loft's forum post] so based on an early beta version of MX, and there has been minimal attempt to update the contents as MX has been developed further and come out of the original beta. <big>Anyone willing to check the linked article and do a further update?</big> |
|||
=== Cross-reference to [[What to do when I have a problem with MX]] === |
|||
The text that was here has been moved to a separate article, that makes it more accessible, please see [[What to do when I have a problem with MX|What to do when I have a problem with MX article]] |
|||
=== Cross reference to [[MX Administrative Interface|Administrative Interface]] === |
|||
This requires a whole topic to itself, and indeed it has an article to itself, reached from link in heading for this section. |
|||
=== Message from Steve Loft about documentation === |
|||
''The following was written by Steve Loft when he first made his MX beta available to Cumulus users. Although somewhat outdated it is preserved here.'' |
|||
There's quite a lot to read before you start - please do read all of this page and all the references it mentions, most of it is very important. |
|||
Note that most of the Cumulus 1 documentation also applies to Cumulus MX. MX specific documentation is currently in very early stages and some settings may not be obvious. Looking at the [[FAQ|Frequently Asked Questions for Cumulus 1]], [[Cumulus MX FAQ|Frequently asked questions for MX]], and articles elsewhere in this wiki will help, as will looking at the Cumulus 1 help file, it is available on the [[Software#Resources|Software Resources page]]. If you already use Cumulus 1, the help is part of the standard installation. |
|||
'''Optional''' |
|||
* Some people require a copy of the local file to use on their web server. Consequently, after it has been updated they file transfer it to (or if their web server is local, copy it to) their web server. One way of doing this is [[Upload_Dayfile| described here]]. |
|||
*Some people take a copy of the local file, and use it locally for other purposes. See [[#How you can use the daily log|How you can use the daily log section]] and also the [[[[User Contributions|Cumulusutils]]]] link. |
|||
*For some people it is easier to follow an option of converting the file into a database table, and that table having a new row added each time the file gains a new line as described below. |
|||
If MX is your first encounter with Cumulus, you will be at a disadvantage regarding documentation of many of the features, while those who have previously been familiar with Cumulus 1 will find most aspects of MX easier to pick up. |
|||
=== Populating a database table === |
|||
=== Message about this update to documentation === |
|||
* The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have bee edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using. |
|||
*There is a section at [[Cumulus_MX#Optional_Sections|Cumulus MX]] to explain how MX includes the ability to generate SQL for creating the database table, for updating it with past data, and to add a new row at the end of the day for the standard database table version of this daily summary log. |
|||
Although the message above from Steve Loft has been retained, it is no longer really true. When that was written on 2 January 2015, MX had been worked on for a year or so and had just opened up for beta testing. |
|||
==== Using that table ==== |
|||
In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log. |
|||
Since then, of course MX has come out of beta and added a lot more functionality. More importantly, it has gained a large user base (although Cumulus 1 is still used by considerably more people than MX, there has been a recent surge of converts), and that means it is much better known, and consequently it is possible now to document it much better. The update made to this page draws on what has been said spread over lots of posts on the support forum and attempts to make it more accessible by repeating it on this page. Consequently, you don't now need to search in the way that Steve Loft's original text above implies. |
|||
For examples of some of the third party tools (Cumulus [[Category:User_Contributions|user contributions]]) using the database daily summary table see [[Daily Summary#Some_example_Scripts|here]]. Of course there are also a lot of tools written to use a copy of the dayfile.txt log file, and some of these could be adapted to use the database table instead, if you are a programmer. |
|||
In writing this update, I have drawn on my own experience of moving from Cumulus 1 to MX, and thus my knowledge of Cumulus is from over a decade of experience with this software and what it can do. |
|||
In my case I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table, i.e. I have one database table column for each of the weather derivatives I show on my web page that show this month's values; it is many more derivatives than are shown on the standard web page, but some are initially hidden. Consequentially, when my daily update script detects from the date that it is processing the last day of a month, it then starts another script that reads all the rows in the daily summary table for that month, and stores the highest/lowest/total (as relevant) in my monthly_summary table (nothing to do with the "monthly" table that MX can generate from its standard log file). This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Just another example of how much you can get from just one log file! |
|||
Before I swapped, I made a detailed study to check MX could do all I used to do with Cumulus 1 and much more. Before I add items to this article I play around with MX experimenting with what works and what does not work, but I have saved you the pain of where I went wrong, just telling you what is correct. I do need to add, that I don't have a separate testing environment, and therefore I am not willing to attempt anything that might muck up my collecting of weather information, plus my knowledge of modern technology is poor as I belong to that generation who did not have desktop computers, nor mobile devices, until some time into my working life. This all places restrictions on what I can test out, and therefore on the coverage of these notes. |
|||
==== Alternative schemas ==== |
|||
'''If anyone else, can improve these notes, wants to split off more parts, or in any other way make the documentation better, then please do. This article already contains improvements that were suggested by others.''' |
|||
Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files. |
|||
{{Version badge 1}} |
|||
With Cumulus 1, you would need to be a programmer and write your own script to update the database table with your own schema. You might use the importCumulusFile article to start you off. |
|||
If this page, and those other Wiki pages it links to (e.g. [[Cumulus MX FAQ]]), do not answer all your questions then see [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the support forum for current Cumulus MX] as that will let you see what other people have asked about, any posts I have not yet incorporated into this page, and there you get the opportunity to post your own query. |
|||
You might also, as I did, want your script to validate what it reads from the daily summary log to ensure only valid numbers and times are stored in your database table, while any invalid inputs are stored as nulls by your script. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the contents of the Weather Diary, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). When I used Cumulus 1 I wrote a PHP script to find all these additional values, for example it reads the [[today.ini]] and [[month.ini]] log files as stored in the end of day backup (not the ones being updated for new day in data folder), and the [[log.xml|weather diary in log.xml in data folder]]. |
|||
= Installing and Running Cumulus MX = |
|||
[[File:Badge vMx.png]] |
|||
MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned. I continue to use my Cumulus 1 script (with some changes as for example the weather diary works differently, I am querying SQLite from \CumulusMX\data\diary.db) now I use MX. In the MX standard functionality, you are limited to using web tags for your inputs, and some of those are affected by the end of day process. I tried various content in the custom EOD query, but it did not give me what I need for the scripts that produce my web pages. |
|||
There is no automatic installer (this may change). Cumulus MX is supplied as a zipped package on a link from [[software|Software page]]. |
|||
MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do. |
|||
== |
== Replacing Cumulus 1 == |
||
See [[Moving from Cumulus 1 to MX]] article. If you wish to run MX on Windows, then you can unzip the contents of the download package over your cumulus 1 installation, i.e. so the same data and Reports folders continue to be used. But it would be best if you take a back-up copy of the Cumulus 1 installation first! |
|||
* It reads the daily log and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|this Cumulus 1 FAQ]]) |
|||
* Thus you must not have another process attempting access to the daily log when Cumulus is re-starting. |
|||
* For Cumulus 1, back ups of 8 selected log files including dayfile.txt that are copied to start-up folders in the 'cumulus\backup' folder, the last 8 start-up folders only are retained. |
|||
* For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else. |
|||
The package contains several extra .dll files, and everything else you need, to continue to read from your weather station, to load up the admin interface (there are some settings you will need to change), and some simple web templates (that replace the standard Cumulus 1 ones). You might want to read topics on the MX support forum to discover about other people's experiences. |
|||
== How you can use the daily summary log file == |
|||
== Completely new MX installation == |
|||
* If you want to run scripts that use the daily summary log file, it is best if you take a copy first, you can ask Cumulus 1 to take a copy after each update by using the '''Daily''' box in the bottom left of the ''Sites/Options'' frame within the ''Internet'' options screen from the '''Configuration''' menu; that will safely take a copy of 'dayfile.txt' after it is updated. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up and so not at usual rollover time according to the computer clock. See Cumulus 1 '''Help''' for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from running the command file I specify in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure. |
|||
* Cumulus MX has option to list files to be transferred once a day as part of rollover, so you can use that to generate your extra copy. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up. |
|||
* A third party tool "Cumulus Toolbox" can also be used to copy/transfer files at a particular time. Note this cannot tell whether Cumulus has done its rollover at the normal time, or during catch-up. |
|||
* There are other ways to specify that when a file changes it is copied somewhere. |
|||
*The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file when Cumulus restarts, when Cumulus processes end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, either your external process or the Cumulus process may fail. |
|||
*Normally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines. |
|||
*Some third-party tools have to be told what separator you use for dates, before they can read your dayfile.txt. |
|||
See earlier for links to other articles about installing on a Windows PC or a Raspberry Pi. Here only a brief indication of installation is covered. |
|||
= Safely editing the daily summary log = |
|||
*Create a new directory (recommended name CumulusMX) and unzip the contents of the [[Software|download package]] into it. |
|||
Both Cumulus 1 and 3 (2 did not) now provide editors where you can see what is in your dayfile.txt log, and if you click on a particular line you can edit the fields in that line, or indeed delete that line. |
|||
*See notes below for extras required in various operating systems. |
|||
*The package contains everything else you need to read from your weather station (if it is a supported model), to load up the admin interface (for settings and some simple templates used to create web pages to see on a device connected to your home network). You might want to read topics on the MX support forum to discover about other people's experiences. |
|||
== Running Cumulus MX == |
|||
If you use these provided editors, then you don't need to worry about the editing rules in the next section. The in-built editors always insert a full line including all fields available in the version of Cumulus you are currently using, so you cannot accidentally insert blank lines. The Cumulus 1 editor also does some validation to ensure the right entry format is used for each field. There is no validation in the MX editor (it was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions), but all lines are passed back via an application programme interface to the MX engine which might raise an error when it can't replace the line in the logging file. |
|||
# Make sure your weather station (and any extra sensors) is connected to the device on which you have installed Cumulus MX, before you try to run Cumulus MX. |
|||
== Important Rules when editing dayfile.txt == |
|||
# Start '''Cumulus MX engine''' (command to do this varies between operating systems, so see sub-heading for your device below |
|||
# Start '''Admin Interface''', it runs in a browser, by default on port 8998, see [[#User_Interface|section]] below. |
|||
If you have been running Cumulus 1 before, then [[Moving from Cumulus 1 to MX|as instructed here]] your MX installation will require various files from your Cumulus 1 installation including all files in the '''data''' and '''Reports''' folder and all [[:Category:Configuration Files|Configuration Files]] including [[Cumulus.ini#Swapping_from_Cumulus_1_to_MX|Cumulus.ini]] and follow that link for details of a few of the parameters that you may need to change. |
|||
If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it. |
|||
If you are running MX for the first time, without a configuration file (none is included in download package), see [[Cumulus.ini#Cumulus_MX|here]] for screen shots showing what you see as the engine starts running, and what you see in the admin interface where you set your weather station type. In that link there are more instructions. |
|||
=== General Editing Rules === |
|||
=== .NET and Mono === |
|||
# Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use. |
|||
# Take another copy and use that for your editing, don't edit the actual file being used by the software. |
|||
#*This prevents any conflicts between access by the software and access by your script or tool being used to modify the file. |
|||
#*It also means that you can go back to the last working copy, you can't upset your "revert" copy. |
|||
#The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values. |
|||
#* It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used. |
|||
#** These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line. |
|||
#*You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus. |
|||
#**If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields. |
|||
#*Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format. |
|||
#Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted. |
|||
#All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity) |
|||
# Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing. |
|||
# Don't add a header line to the file, Cumulus expects all lines to be data lines. |
|||
The software currently called .NET was originally for all operating systems, but Microsoft then decided to restrict it to just Windows (and that was the position while MX was being developed and is still true in early 2020 when this article was rewritten), mostly to encourage greater dominance by Microsoft software and hardware. |
|||
=== File specific Editing Rules === |
|||
Mono was then born based on .NET to work with all operating systems, Mono subsequently changed independently from .NET (although Microsoft still has a leading role). |
|||
[[File:Open office (editing cumulus log files).png| right]] |
|||
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM. |
|||
# All rows must ''start with date'' and include at least 14 further fields ''in correct sequence''. |
|||
# The (meteorological) date format uses ''two digits for the year''. |
|||
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types). |
|||
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or on Microsoft Windows '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file. |
|||
#Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile. |
|||
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space. |
|||
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus. |
|||
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits. |
|||
#* If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent. |
|||
# Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses. |
|||
#* If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option). |
|||
# Rows can vary in length but only by missing off ''fields at the end''. The minimum number of fields after the date is 14, the maximum varies between different versions. |
|||
#Each field has a pre-defined format, and the same format must always be used in that field position. |
|||
#No fields will accept letters. |
|||
#*Some fields (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields. |
|||
#* Most value fields are in ''real number format'' using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point. |
|||
#Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields. |
|||
#when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a ''lowest or highest value'' and that must be paired with a time-stamp in the next field. |
|||
#*Cumulus will only accept highest/lowest figures if each value has any related time-stamp. |
|||
#Time stamp fields must always be in ''format HH:mm'' i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes |
|||
#**Be aware you will have problems if you, or your editing software, add seconds. |
|||
#*If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time. |
|||
#* Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter. |
|||
# Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool. |
|||
#* Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps. |
|||
#*If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator). |
|||
#*Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder. |
|||
#Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day. |
|||
#* Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases. |
|||
# The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. Similar rules apply for single character terminators used by other operating systems, don't let windows change a single character terminator into its double character terminator. |
|||
#*Problems with terminating characters are intercepted by operating system, before it reaches the software, but may still stop the software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus. |
|||
More recently, Microsoft launched an alternative called .NET Core that took out of .NET the parts that were Windows specific, and it ceased work on further development of .NET beyond version 4.x.x. |
|||
'''Perhaps confusingly, in November 2020, there will be change around of names, and the multi-operating system .NET Core product will take over the .NET name as version 5. I don't pretend to understand the technical details, but the impression I get is that the new .NET in November will be similar to Mono, so apps designed for that will still work, but apps using .NET to make code designed for windows will stop working'''. Since the Cumulus code is currently coded to behave slightly differently using .NET and using MONO, I guess it is possible old versions of MX might stop working when the new .NET is installed via Windows Update. |
|||
=== Requirements for running on Windows === |
|||
=== Dealing with errors identified by the software === |
|||
To run MX on Windows, you need .Net version of at least 4.5.2 installed. This is only available for Vista SP2, Windows 7 SP1, Windows 8, Windows 8.1. |
|||
If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus 1 will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be. |
|||
For Windows 10 you need version 4.8 or later, this should already be installed by your windows update feature. The .Net download for version 4.8 should be here https://dotnet.microsoft.com/download/dotnet-framework/net48. |
|||
If you do have a 'duplicate' error, you need to decide which row to ''delete'', and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained. |
|||
For those who downloaded the first MX Beta in January 2015, the code was only experimental and that version had to be run by a Windows Administrative User, but Steve Loft soon improved the code and now none of the code requires any elevated rights and it can be run by a normal user (or a user with administrative rights) without needing to be started by '''Run as administrator'''. |
|||
=== Editing externally === |
|||
However, Cumulus MX initiates a web server, which is what runs the [[MX_Administrative_Interface|Admin Interface]]. To access that, all users need to be given elevated rights to the port on which the web server runs. By default this is port 8998, so that is used in the example below of the one-off command needed to give all users access to the port. You can use a '''-port=nnnn''' parameter when starting MX to make it use another port, if you use that then the command below needs revising accordingly. |
|||
See earlier section on this page for the rules and later section for advice on dealing with rogue values. |
|||
To enter the command, first open a command window as administrator. One way to do this is to right click the windows symbol at the start of the windows task bar. The option to choose there is dependent on some settings which determine what appears when you right click: |
|||
=== Using the Cumulus 1 editing feature === |
|||
*the normal default on Windows 10 is '''Windows PowerShell (admin)''', |
|||
*the normal default on earlier versions of Windows is '''Command Prompt (Administrator)''' |
|||
*an alternative is '''Windows Terminal''' |
|||
Whichever of these you can use, the result is it opens a new window on your monitor with a prompt for typing. In that window type the command: |
|||
<pre> |
|||
netsh http add urlacl url=http://*:8998/ user=\users |
|||
</pre> |
|||
You only need to do that once. If you do not issue this command, administrative rights are needed every time to access the port. |
|||
{{Version badge 1}} |
|||
'''This section applies to Cumulus 1.x.y only'''. The last command in '''Edit''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu |menu]] is ''dayfile.txt''. ''This is how you view'' the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. Click the ''Help'' button for detailed instructions. Cumulus Help is comprehensive. |
|||
Talking about command windows, if you want to check that the port is open for listening (i.e. able to access the admin interface) type <tt>netstat -an | findstr 8998</tt> into the command window. |
|||
You can use this Cumulus 1 editor as follows: |
|||
*use '''insert''' key to add one or more missing rows (complete days) manually, |
|||
*correct individual values by over-typing, |
|||
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message), |
|||
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Standard_log_files]] and automatically calculating the best approximations for each field for those missing days. |
|||
The admin interface URL '''http://*:8998/''' needs to have that wildcard "*" replaced by a precise location if we are to access the admin interface. The missing part of the URL depends on how your local network is set up. If you are accessing the admin interface on the same device as that running MX (and you don't have another web server on that device) the "*" can be replaced by "localhost", i.e. '''http://localhost:8998/''' will be used to load the admin interface into your browser. In the more general case when you want to access the admin interface from anywhere on your local wired and wireless interface, then the "*" needs to be replaced by a string of 4 numbers representing what is called a IPv4 address (w.x.y.z) of the device you have installed MX on. |
|||
==== Create Missing ==== |
|||
Look at your hub or router (this should have come with instructions on how to access its settings in your browser) and on one screen it should show what devices are connected to your LAN and wifi. Look for the IPv4 address, for example 192.168.1.64, it has assigned to the device where MX is running. That is what should replace the "*". However, there is one more complication, either the Windows networking settings may change, or else your hub or router may reconfigure, both can happen at any time and both can assign a different IPv4 address to the device running MX. |
|||
The Cumulus 1 editor provides a "Create Missing" option where it will, for any dates for which a line does not exist, create a line if it can from reading the [[Monthly log files|detailed log file]] to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation for each dayfile.txt field, storing the time from the relevant other log file in any time-stamp field in dayfile.txt. If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged. For ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]]. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | work around]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Standard_log_files]] if the whole day (a line starting with that date) is missing in ''dayfile.txt''. Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps. |
|||
To give your Computer a fixed address for the MX admin interface, |
|||
==== Using Standard logs to deal with shorter (or incomplete) dayfile.txt records for particular dates ==== |
|||
#first find the network card via Network and Sharing Centre (Control Panel), |
|||
#then click on Change Adapter Settings, |
|||
#then Right click on Ethernet or WiFi Adapter, |
|||
#next select Properties |
|||
# In the window that opens, right click on Internet Protocol Version 4 (TCP/IP 4), |
|||
#Next select properties |
|||
# Onn that pop up screen tell the computer to "use the following IP address" |
|||
#Fill out the form with |
|||
#* a subnet mask of 255.255.255.0 and |
|||
#*gateway address between 192.168.1.1 and 192.168.1.254 (depending on the address of your hub/router). |
|||
==== Setting up for either manual or automatic running ==== |
|||
'''WORKAROUND if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt''' |
|||
Example: '''add extra fields to records created by an earlier version of Cumulus''' [to help you, the versions (not builds) at which fields were added are indicated below]. |
|||
Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files. |
|||
To run Cumulus MX, Windows needs to know |
|||
''One method '''for Cumulus 1 only''' is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use ''create missing'' in the [[dayfile.txt]] selection in the '''Edit''' menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).'' |
|||
#which '''.exe''' you want to run |
|||
#the path where all the required '''.dll''' files are located |
|||
Therefore it is best to always start MX using what Windows calls a '''shortcut''', because when creating the shortcut you can enter all the required information into the properties. If you want MX to automatically start whenever you log into your PC, then the place to store your shortcut is <tt>C:\Users\...\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\Run_CumulusMX</tt>. Don't forget to put your Microsoft username where I have put ... |
|||
'''Cumulus MX''' |
|||
Although there is a tentative plan for an executable to be added to the MX package that can read the standard logs and calculate all the fields needed for a line in dayfile.txt, this is only a gleam in the developer's eye! |
|||
With this you might want to right click on that shortcut, select properties, then you can set the starting position for the command window, the colours and font it will use, and even choose to start minimised, amongst many other selections. |
|||
The only method available at the time of writing involves multiple steps: |
|||
#*Either use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table |
|||
#*Or if you already have the table and have MX updating the standard logs into that database table |
|||
#Then, you can fairly easily update to add any missing values into existing columns in this monthly table, and from it update columns in a daily summary table. See [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] where I describe two PHP scripts designed for this type of task. |
|||
There are 3 ways on Windows to create a shortcut to run MX, as mentioned above you can't just click on the executable in file manager, because Windows needs to be told the path for loading all the related .dll files. |
|||
=== Importing data not recorded by Cumulus === |
|||
As an alternative to manual line ''insert'', in the Cumulus 1 editor, you may wish to use a procedure for importing, and processing, pre-Cumulus observations into [[Standard_log_files]]. Once there is data for required days in monthly logs, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt. |
|||
#Create a shortcut on your desktop (and/or the taskbar) for the '''CumulusMX.exe''' executable <tt>cmd.exe /C start CumulusMX C:\CumulusMX\CumulusMX.exe -debug</tt>, the "-debug" is ''optional'', it starts the logging in debugging mode so the log created in MXDiags folder has more information. There are other optional parameters all listed later. |
|||
== Cumulus 2 == |
|||
#*In that shortcut define the path where the executable is located as the path to start in. |
|||
#* '''Remember, if you have not done the <tt>''netsh http add urlacl url=http://*:8998/ user=\users''</tt> command mentioned above, you must run as administrator.''' |
|||
#* Add any of the parameters that can be used with the executable, as listed later, such as specifying a different port for the admin interface , or starting with debugging. |
|||
#*Choose whether to start as minimised or as a command window |
|||
#*You can also choose text colour (foreground) and background colour, you can choose where the window appears (so if you have two monitors you can choose which one it appears on), and how big that window is. There is a forum post by water01 about this. |
|||
#*Now you can click the shortcut to start the engine |
|||
#OR place the shortcut as defined above in the start up folder for the user account so MX automatically starts when you connect/log in (see a later section for how to find that start up folder). |
|||
# OR declare a task in the scheduler to start MX; in the '''Actions''' tab fill in fields as follows (the other tabs should be obvious): |
|||
#**'''Action''' <tt>Start a program</tt> from drop-down |
|||
#**'''Program/script''' <tt>cmd.exe</tt> (this is standard Windows environment to run something) |
|||
#**'''Add arguments''' <tt>/C start Start_MX \CumulusMX\CumulusMX.exe -debug -port=nnnn</tt> (the "/C" means this task will close once it has started the task, the "Start_MX" is how the task will be labelled as it is running, the next argument "\CumulusMX\CumulusMX.exe" actually starts the executable and it does not need a drive prefix as that is in next box. |
|||
#**Note in this example I have included next two optional parameters that can be used after the .exe call in that same box, here '''-debug''' (only include if you want full debugging logging) and '''-port=nnnn''' where nnnn is the port to be used for admin interface (only include if want to change from default 8998), |
|||
#** all optional parameters are listed later |
|||
#**'''Start in''' <tt>\CumulusMX</tt> (include a drive specifier if necessary) |
|||
==== Each time you want to run Cumulus MX on Windows: ==== |
|||
Does not provide any viewing functionality. |
|||
#First '''start the engine''' in one of the 3 ways from last sub-section |
|||
== Cumulus MX == |
|||
# Next '''start the admin interface''', it does not need to run all the time, but only when you need it (when you first use MX you will need it to access the settings where you tell MX what type of station you have and what units you want to use, and set various timing options), it normally runs on port 8998 (to vary that there is a '''-port''' parameter that is followed by required port and that port parameter has to be entered every time you start MX if you are not using the default port). More information on admin interface [[MX Administrative Interface|in separate article]]. |
|||
Try '''start /min C:\Cumulus\CumulusMX.exe''' to run MX as a minimised package (although in Windows you can change the properties of the shortcut you use to start minimised). |
|||
There is a dayfile editor within the admin interface to edit this log file. |
|||
==== Stopping Cumulus MX on Windows pc ==== |
|||
Only from MX version 3.6.0 has this been able to read the log file if it has some lines that were created using Cumulus 1 (with less than 46 fields). In the same version of MX, the number of fields in this log file was increased by 4 when Feels Like temperature was added. From that version all lines viewed in this editor will have 50 fields. The content of any field that was not in the line when it was created will be an empty string as far as this editor is concerned and any line edited and saved, therefore gains all these empty fields and will be stored as 50 fields until version 3.6.10. |
|||
The recommended way is to click into the command window in which MX is running, ''hold down Control key'' and press '''C'''. It is normal for there to be a short wait, then a message "Cumulus Terminating" and then after another short wait, it will say "Cumulus Stopped" and immediately after that the command window will close. |
|||
From Emergency Version 3.6.12 (formally released in 3.7.0), all lines have 54 fields. The extras are Canadian Humidity Index (Humidex). |
|||
Some people, click in the task bar and select close, or click the '''X''' button on top right of command window. Although these are not official advice, they do seem to work. |
|||
[[File:Badge vMx.png]]For Cumulus MX, when you select a line, both '''Edit''' and '''Delete''' buttons are enabled. There is no way of inserting new lines into dayfile.txt from within MX, nor of changing the dates in the file. |
|||
There are packages that can be programmed to send a control C to a running task, and to not continue until the task window has closed. Remember to also program in a subsequent delay in that package, to make sure the package waits for MX to close, or do a check that MX has released all the files it might need to update. |
|||
''You should not'' issue a '''TASKKILL''' instruction, as that will prevent MX correctly writing out to all the files it should update on exit. Consequently, it will not restart correctly and may actually lose settings and data. |
|||
=== |
=== Requirements for running on Linux and OS X === |
||
You will need to install the '''Mono-complete''' runtime (the latest version of Mono should work with all functionality of latest MX in all locales). Mark Crossley says "There shouldn't be any outstanding issues with Mono, afaik they are all resolved - except for the Moon image rotation in the southern hemisphere which does not work with Mono 6.0 thru to the latest 6.8.0, only version 5.x works correctly atm for System Drawing." |
|||
Pick Edit, click that and an editing dialog pops up, that does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), and the right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using). |
|||
* For OS X, you can download this here - http://www.mono-project.com/download/. |
|||
* How you install on Linux depends on the flavour of Linux you are running. There are download links for Linux at the same URL, but it is often easier to use a package manager, which will download and install it automatically. |
|||
**For example, in 'Raspbian' on the Raspberry Pi, you can install mono with the following commands, but '''first you need to have set up various pre-requisites''' (see [[Setting_up_Raspberry_Pi]] article for details): |
|||
<pre>sudo apt update && sudo apt upgrade |
|||
=== Editing externally === |
|||
sudo apt install mono-complete</pre> |
|||
Note that you do need to have the '''mono-complete''' package installed, not just the Mono for developers. |
|||
The "sudo" prefix gives the command 'root' privileges, that allows administrative commands like update and install to run. |
|||
See earlier section on this page for the rules and later section for advice on dealing with rogue values. Remember if you edit this file externally, either do not add any missing fields, or if you do then these must remain empty for every extra field in every line that was originally shorter! |
|||
==== To actually run MX ==== |
|||
Open a terminal window, change to the Cumulus MX directory, and then type: |
|||
MX does not have the capability to recalculate, the values and time-stamp for any entry in dayfile.txt, by reading the [[Monthly log files|detailed log file]] to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation. It can be done in a script if you are a programmer. I have done this, but it was not as easy as I thought, the log file you read from may have duplicate times potentially making the calculation harder, and you cannot edit the file as you go along, you basically rename the old file and create a new file with original name as you work through all days changing individual fields whenever you need to. |
|||
<pre>sudo mono CumulusMX.exe</pre> |
|||
There are some optional parameters you might need to use, as they also apply to windows they are covered later. |
|||
== Editing the file or other Manipulation outside Cumulus == |
|||
Next start the administrative interface, basically same as described for Windows above. More information on admin interface [[MX Administrative Interface|in separate article]]. |
|||
{{Version badge 1}}Apart from bulk changes, Cumulus 1 made editing this log file quite easy. Although the create missing did not work where a line was present for the date, but an individual field needed to be corrected or populated, there was a [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates |work around]] for this. |
|||
==== Other issues ==== |
|||
[[File:Badge vMx.png]]The beta versions of MX did not include any feature to view or edit dayfile.txt, so any checking or editing had to take place outside the software. As mentioned above, MX does not provide any way to read, or calculate, values from any other file, nor will it do bulk edits. For any of these, you will need to take a copy of the Cumulus file and manipulate using a script or a spreadsheet processor. Do be careful not to change the encoding, the position of line feeds, the format of any field, the locale, or anything else that might stop MX being able to read the file. |
|||
There are lots of topics in the MX sub-forum about a multitude of issues about commands to use to install and check mono, how to stop MX and differences between different devices (including Mac) and different Linux versions. At the moment, there seems to be some uncertainty, and consequently, I have not attempted to include/summarise all the material I have found. |
|||
'''APPEAL''' - Please could any readers who have experience of running MX in a Linux or Mac environment please consider writing advice into this article. I want it to be a comprehensive accurate article. |
|||
The file is <tt>data\dayfile.txt</tt> within the directory holding the Cumulus executable, it can be viewed in a text editor, imported into various database systems, or imported into spreadsheets, to manipulate as you wish. Just remember that Cumulus reads it when it is restarted, and updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating (and possibly doing a rollover of logs), or around the midnight/9am/10am local rollover time when Cumulus is writing a new row. |
|||
=== Notes by Steve Loft === |
|||
'''Tip''': Take a copy of the file before you do any edit outside Cumulus, so you can revert to old file. |
|||
''please note these notes ARE now obsolete, library routines have changed a lot since this was written in 2014'' |
|||
'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end). (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page). |
|||
'''Any volunteers to replace this section with up to date information?''' |
|||
=== Dealing with rogue measurements === |
|||
Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes. |
|||
If you discover a rogue measurement (perhaps the wind affected your tipping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see [[today.ini]] or a [[FAQ]] for further advice. In general, you need to stop Cumulus, edit the monthly log row containing the dodgy values, edit 'today.ini' and possibly other '.ini' files, looking up the logs indicated below that show the updates with previous high or low. |
|||
**If you have a Raspberry Pi 2, there is a later version of Mono available, which you may find works better that the one in the standard distribution, particularly if you use decimal commas. Mono 3.2.8 (which is the default in some Linux distributions) will not work if you use commas for decimals, as in some countries. |
|||
If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time. As your first step you should update the appropriate field in the row for the affected date in dayfile.txt. Once ''dayfile.txt is correct'' the Cumulus editors will allow you to: |
|||
* update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. See [[Alltimelog.txt]] for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it. |
|||
* update the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu. |
|||
* update the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago. |
|||
* update the highs and lows month-by-month in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows. |
|||
* '' '''Note''' in each of above 4 editing screens you can: |
|||
*# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp. |
|||
*# load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and |
|||
*# optionally ''Copy'' (row by row) the logged values (and associated date/time information) into the relevant .ini file. |
|||
*# click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''. |
|||
*# store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel''). |
|||
(Each of these screens is a text editor, and works best when at full screen).'' |
|||
* create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report. |
|||
Ideally, you will subsequently try to edit the rogue data for the particular time it was logged; see [[Standard_log_files#Correcting_any_logged_data_problems]], but correcting the daily summary in dayfile.txt must always be the priority. |
|||
**On Linux you will need library '''libudev.so.0''' which may not be installed by default. Installing '''package libudev0''' may resolve this. There may be issues if you are using a 64-bit version of Linux. I'm not sure what the resolution is at the moment, if this is the case. |
|||
== Using the daily summary log on your web-site == |
|||
You need to specify something like '''/dev/ttyUSB0''' for the connection for your weather station. This is set in the "station settings" and stored in the [[Cumulus.ini#station|ComportName attribute]] in Cumulus.ini configuration file. |
|||
If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page. You could have a web page that shows a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file. |
|||
In some builds of MX you have to run as "root", there are ways of giving "root" like permissions when running MX as another user, see forum for details until this section has been updated. |
|||
Search the Cumulus support forum to see (for example) how others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago. |
|||
==Stopping MX== |
|||
If you use a script to read what is in the daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see [[Daily_Summary]] article for information about ALL of the ways to show values from this database table. |
|||
The best way to stop MX is by sending a '''control and C''' sequence to it. There is a start stop routine discussed in the support forum, where you will also find some topics about shutdown issues. |
|||
==Viewing summary figures for a month or period== |
|||
The MX source listing suggests it also accepts ''control and break'', '''Close main window''', ''user logoff'', and '''system shutdown''' and should still attempt to stop running tidily. There are various tasks like writing final values to log files and recreating the configuration file, [[Cumulus.ini]], that it needs to do as part of the MX closing routine, and obviously early closure of the device running MX (such as that caused by a power cut) will prevent a tidy end to MX. |
|||
'''These notes apply to Cumulus 1 only''' |
|||
To view a summary of dayfile.txt for a month, calendar year or selected period, use ''This month'' (choose any month, default is month from your computer system date), ''This year'' (choose any year, default is year from your computer system date), or ''This period'' (choose any start and end dates, default is yesterday calculated from your computer system date), within the '''View''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|menu]]. |
|||
*Remember the daily summary log has its records based on rollover to rollover days. |
|||
*In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover. |
|||
*If you use 9am or 10am rollover, and choose '''View''' ''This period'' between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank. |
|||
*If you use 9am or 10am rollover, and choose '''View''' ''This month'' before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank. |
|||
Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying). |
|||
*On the screen displayed after selecting ''This month'', you can change the month and year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated. |
|||
*On the screen displayed after selecting ''This year'', you can change the year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated. |
|||
*On the screen displayed after selecting ''This period'', you can change the start date and end date then click ''Update Display'' to get the equivalent calculations displayed for part of a month or any other period. |
|||
= Executables = |
|||
The MX package, at time of typing this, includes two executables: |
|||
Note differences between observation reports on View screens and those available as web tags. |
|||
*Date and time stamps: |
|||
**The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt; |
|||
**A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight). |
|||
*Reported statistics example: |
|||
**The screen shows total number of dry or wet days in the month; |
|||
**The web tags report longest dry or wet period in the month. |
|||
== CumulusMX.exe == |
|||
= List of fields in dayfile.txt = |
|||
Whilst effectively MX is run by a '''CumulusMX.exe''' or '''sudo mono CumulusMX.exe''' depending on device, you actually need to ensure all the other components are loaded, so you either have a package that runs it for you, or you click a shortcut that includes the necessary path setting. |
|||
== Variation by Cumulus version == |
|||
=== Optional parameters to add to the instruction to run the MX engine === |
|||
The dayfile.txt has grown as Cumulus's functionality has been extended, the table below shows fields grouped by the Cumulus version when those fields were added. |
|||
Beta builds in MX version 3.0.0 had an optional parameter <tt>-wsport nnnn</tt> that determined which port (represented by a 4 digit number ''nnnn'') was used for '''WebSockets'''. That parameter is now deprecated as WebSockets in all builds since 3045 uses the same port as the rest of the [[MX_Administrative_Interface#The_API_interface|Admin Interface]]. The remaining parameters that are still available are described in subsequent sub-sections. |
|||
If you have been using Cumulus for a while, then some lines will be shorter than others, the number of fields per line growing when a new release adds new fields, Cumulus does not in normal operation modify earlier lines, but both Cumulus 1 and MX provide editors where it is possible to modify any line. |
|||
==== Parameter for changing Port ==== |
|||
There is information earlier in this article about how you might be able to recalculate values to put in for fields that did not exist when any particular line was created. |
|||
When Cumulus starts, it will display the URL of the user interface. It runs on port 8998 by default; if this is not suitable for some reason you can over-ride it using the '-port' parameter on the command line, e.g. to use port 9999 instead: |
|||
'''For your installed build please see ''dayfileheader.txt'' (stored within the folder that contains your Cumulus executable), as that will list which fields are available for you.''' |
|||
<pre>sudo mono CumulusMX.exe -port 9999</pre> |
|||
== |
==== Parameter for adding debugging ==== |
||
MX has a default level of logging that stores in the [[MXDiags]] folder a log file that shows some of the interaction with the weather station and some of the output actions done as MX runs. A new log is started each time MX is restarted. |
|||
*The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, adds a line number in front of the date field. |
|||
** The Cumulus MX user may not be aware of this happening as it is within the internal workings. |
|||
*The original table below was for Cumulus 1 and then field number '''was''' starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages: |
|||
*#It stressed that the date field was different to the rest. The date '''must''' be a unique identifier, the same date should not be repeated in another line. |
|||
*#The remaining fields were all either numerical values or a time paired with preceding numerical value. |
|||
**Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a line was converted to an array, and you needed to address a single field. |
|||
*The alphabetic column identifiers used by many spreadsheets are shown, please see warnings about using spreadsheets for editing earlier on this page |
|||
*The type of field is shown, you must not put a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format |
|||
*The field description is shown, together with references to where that terminology is explained |
|||
If there is a problem, then there is a great benefit in actually increasing the level of detail in these logs; and that is done either within the settings ('''options''' section of ''station settings'') in admin interface while MX is running, or by adding 1 or 2 parameters when you start MX. Obviously this log file continues to grow, the longer MX is left running, and if debugging is switched on the file will grow in size must faster. Consequently, the default is not to add the extra debugging information and the settings can be used to switch it off again if you do have it switched on. Whether you start it with a parameter or enable it within settings, stopping MX will end the extra debugging, and on restart it will default back to no debugging unless turned on again with parameter or setting. |
|||
=== List of Fields === |
|||
{| class="wikitable" border="1" |
|||
|- |
|||
!style="width:50px" | Field number |
|||
!style="width:50px" | Spreadsheet column |
|||
!style="width:200px" | Field type |
|||
!style="width:600px" | Description |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|Included from when dayfile.txt first invented (assumed in first public release 1.0) |
|||
|- |
|||
|0 |
|||
|colspan="3"|For internal MX purposes, this field is the line number |
|||
|- |
|||
|1 |
|||
|A |
|||
|8 characters |
|||
|Date as ''2 figure day [separator] 2 figure month [separator] 2 figure year'' - the separator is that set in the windows system short date format (see [[setup]]) |
|||
|- |
|||
|2 |
|||
|B |
|||
|Unsigned number |
|||
|Highest wind [[Wind_measurement#Weather_Stations_and_Cumulus|gust]] speed |
|||
|- |
|||
|3 |
|||
|C |
|||
|unsigned integer |
|||
|[[Wind_measurement#Wind_Direction | Bearing]] of highest wind gust |
|||
|- |
|||
|4 |
|||
|D |
|||
|5 characters |
|||
|Time of highest wind gust |
|||
|- |
|||
|5 |
|||
|E |
|||
|signed decimal |
|||
|Minimum [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | temperature]] |
|||
|- |
|||
|6 |
|||
|F |
|||
|5 characters |
|||
|Time of minimum temperature |
|||
|- |
|||
|7 |
|||
|G |
|||
|signed decimal |
|||
|Maximum temperature |
|||
|- |
|||
|8 |
|||
|H |
|||
|5 characters |
|||
|Time of maximum temperature |
|||
|- |
|||
|9 |
|||
|I |
|||
|Unsigned number |
|||
|Minimum [[Pressure_Measurement | sea level pressure]] |
|||
|- |
|||
|10 |
|||
|J |
|||
|5 characters |
|||
|Time of minimum pressure |
|||
|- |
|||
|11 |
|||
|K |
|||
|Unsigned number |
|||
|Maximum sea level pressure |
|||
|- |
|||
|12 |
|||
|L |
|||
|5 characters |
|||
|Time of maximum pressure |
|||
|- |
|||
|13 |
|||
|M |
|||
|unsigned number |
|||
|Maximum [[Rain_measurement#Rain_Rate | rainfall rate]] |
|||
|- |
|||
|14 |
|||
|N |
|||
|5 characters |
|||
|Time of maximum rainfall rate |
|||
|- |
|||
|15 |
|||
|O |
|||
|unsigned number |
|||
|Total rainfall for the day |
|||
|- |
|||
|colspan="4" style="background:gray;"|Above here represents the minimum length for every line, a count of 15 items |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The release history between versions 1.2.5 and 1.5.1 was lost by Steve Loft, and for MX in "DataEditor.cs" this addition has " Extended for ???" as a comment) |
|||
|- |
|||
|16 |
|||
|P |
|||
|signed decimal |
|||
|[[Average temperature]] for the day |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(Wind run was added from version 1.8.4) |
|||
|- |
|||
|17 |
|||
|Q |
|||
|unsigned number |
|||
|Daily [[Windrun | wind run]] |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The next pair of entries were added from version 1.8.9 build 907 (June 2010) as part of a total redesign of how dayfile.txt was implemented in Cumulus 1) |
|||
|- |
|||
|18 |
|||
|R |
|||
|unsigned number |
|||
|Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]] |
|||
|- |
|||
|19 |
|||
|S |
|||
|5 characters |
|||
|Time of Highest Avg. Wind speed |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The two pairs of humidity entries were added in a v 1.9.0 beta, the exact build number is now lost) |
|||
|- |
|||
|20 |
|||
|T |
|||
|unsigned integer |
|||
|Lowest [[Temperature_(and_humidity)_measurement | humidity]] |
|||
|- |
|||
|21 |
|||
|U |
|||
|5 characters |
|||
|Time of lowest humidity |
|||
|- |
|||
|22 |
|||
|V |
|||
|unsigned integer |
|||
|Highest humidity |
|||
|- |
|||
|23 |
|||
|W |
|||
|5 characters |
|||
|Time of highest humidity |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The next two entries were added from version 1.9.0) |
|||
|- |
|||
|24 |
|||
|X |
|||
|(not documented) |
|||
|Total evapotranspiration (Only valid for Davis stations, shows zero otherwise) |
|||
|- |
|||
|25 |
|||
|Y |
|||
|unsigned |
|||
|Total hours of sunshine (only valid if sunshine sensor connected) |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The next 16 entries were added from version 1.9.1 May 2011) |
|||
|- |
|||
|26 |
|||
|Z |
|||
|signed decimal |
|||
|High [[Heat index]] (added to Cumulus in 1.7.11 only as spot value, not stored) |
|||
|- |
|||
|27 |
|||
|AA |
|||
|5 characters |
|||
|Time of high heat index |
|||
|- |
|||
|28 |
|||
|AB |
|||
|Signed decimal |
|||
|High [[Apparent temperature]] |
|||
|- |
|||
|29 |
|||
|AC |
|||
|5 characters |
|||
|Time of high apparent temperature |
|||
|- |
|||
|30 |
|||
|AD |
|||
|signed decimal |
|||
|Low apparent temperature |
|||
|- |
|||
|31 |
|||
|AE |
|||
|5 characters |
|||
|Time of low apparent temperature |
|||
|- |
|||
|32 |
|||
|AF |
|||
|unsigned number |
|||
|High hourly rain |
|||
|- |
|||
|33 |
|||
|AG |
|||
|5 characters |
|||
|Time of high hourly rain |
|||
|- |
|||
|34 |
|||
|AH) |
|||
|signed decimal |
|||
|Greatest [[wind chill]] (high wind speed, low temperature) (calculated since version 1.8.3 as spot value, not stored) |
|||
|- |
|||
|35 |
|||
|AI |
|||
|5 characters |
|||
|Time of greatest wind chill |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The next two pairs for dew point were added in version 1.9.2 beta build) |
|||
|- |
|||
|36 |
|||
|AJ |
|||
|signed decimal |
|||
|High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]] |
|||
|- |
|||
|37 |
|||
|AK |
|||
|5 characters |
|||
|Time of high dew point |
|||
|- |
|||
|38 |
|||
|AL |
|||
|signed decimal |
|||
|Low dew point |
|||
|- |
|||
|39 |
|||
|AM) |
|||
|5 characters |
|||
|Time of low dew point |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|(The next three entries were added in version 1.9.2 Build 1004) |
|||
|- |
|||
|40 |
|||
|AN |
|||
|unsigned integer |
|||
|Today's dominant/average wind direction |
|||
|- |
|||
|41 |
|||
|AO |
|||
|unsigned decimal |
|||
|[[Heat/cold degree days and Chill hours | Heating degree days]] |
|||
|- |
|||
|42 |
|||
|AP |
|||
|unsigned decimal |
|||
|[[Heat/cold degree days and Chill hours | Cooling degree days]] |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist) |
|||
|- |
|||
|43 |
|||
|AQ |
|||
|unsigned decimal |
|||
|High solar radiation |
|||
|- |
|||
|44 |
|||
|AR |
|||
|5 characters |
|||
|Time of high solar radiation |
|||
|- |
|||
|45 |
|||
|AS |
|||
|unsigned decimal |
|||
|High UV Index |
|||
|- |
|||
|46 |
|||
|AT |
|||
|5 characters |
|||
|Time of high UV Index |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 3.6.0, 2 more derived values and their times |
|||
|- |
|||
|47 |
|||
|AU |
|||
|signed decimal |
|||
|High [[Feels Like]] temperature |
|||
|- |
|||
|48 |
|||
|AV |
|||
|5 characters |
|||
|Time of high feels like temperature |
|||
|- |
|||
|49 |
|||
|AW |
|||
|signed decimal |
|||
|Low Feels Like temperature |
|||
|- |
|||
|50 |
|||
|AX |
|||
|5 characters |
|||
|Time of low feels like temperature |
|||
|- |
|||
|colspan="4" style="background:lightgray;"|The next two pairs were added in version 3.6.12 |
|||
You can also add '''CumulusMX.exe -debug''' (to have full debugging of actions by MX turned on as MX starts), and/or '''CumulusMX.exe -Logging=1''' (for the station to MX transfers to have increased debugging logging). |
|||
Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times), but until version 3.7.0 (build 3089) the extra fields are empty. |
|||
<pre>sudo mono CumulusMX.exe -debug -Logging=1</pre> |
|||
I believe that the last 2 fields will disappear when 3.7.0 is released, so have labelled them as error. |
|||
|- |
|||
|51 |
|||
|AY |
|||
|signed decimal |
|||
|High Canadian Humidity Index or [[Humidex]] |
|||
|- |
|||
|52 |
|||
|AZ |
|||
|5 characters |
|||
|Time of high Humidex |
|||
|- |
|||
|colspan="4" style="background:lightblue;"|Just confirming that the next 2 fields were included by mistake in an emergency release, and are not be in any other version, so have labelled them as error. |
|||
|- |
|||
|53 (error?) |
|||
|BA |
|||
|signed decimal |
|||
|Low Humidex (in 3.6.12, but no other version) |
|||
|- |
|||
|54 (error?) |
|||
|BB |
|||
|5 characters |
|||
|Time of low humidex (in 3.6.12, but no other version) |
|||
|} |
|||
Since this parameter is applied when you start MX, it applies while MX continues to run. Obviously, it must be applied every time you start MX if you want this increased level of logging to continue every time you restart MX. |
|||
==Example of the file== |
|||
The comments in the MX source suggests -debug turns on both debug and data logging (see [[MX_Administrative_Interface#Options|Station_Settings#Options]] in admin interface settings), but I believe that is wrong as per example above, there are 2 separate parameters. |
|||
An extract of a few lines of the dayfile.txt |
|||
==== Parameter for changing Locale ==== |
|||
On Linux and (in particular) OS X, Cumulus MX may not be given the correct locale to use, and you may get the default US locale even if that is not your locale. It will output the local it is using when it starts; if it is not correct, close it down and start it again, this time specifying your locale on the command line, using the -lang parameter . For example, in the UK, on a non-Windows device type: |
|||
<pre> |
<pre> |
||
sudo mono CumulusMX.exe -lang en-GB |
|||
01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5 |
|||
</pre> |
|||
Other local examples: '''CumulusMX.exe Current culture: English (United States)''', '''CumulusMX.exe -lang de-DE''', '''CumulusMX.exe -lang el-GR''' (this is one of the locales that reads numbers with '''integer,decimal''' format), '''CumulusMX.exe -lang nl-NL'''. |
|||
If you are not sure what value you need to supply for the -lang parameter, there is a list here - http://msdn.microsoft.com/en-gb/library/ee825488%28v=cs.20%29.aspx. You need to supply the code in the first column ("Language Culture Name") in that list. |
|||
02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7 |
|||
Note that this does not affect the language used by Cumulus MX (although it may in the future), it affects the decimal separator and the list separator. |
|||
03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9 |
|||
Note that you ''may'' need to supply your administrator password after typing the 'sudo ...' command line. The system will prompt you for this if it is needed. |
|||
04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9 |
|||
== ExportMySQL.exe == |
|||
05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3 |
|||
This second exe file has been available since the original MX package as Steve Loft developed this in April 2015, but sadly few people even notice it exists, and if they do, it is unlikely they know how to use it. Hopefully, some people will read this section and find out! |
|||
06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3 |
|||
Obviously it was updated when Mark Crossley added the Feels Like fields to log files. |
|||
07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7 |
|||
</pre> |
|||
Put simply, this executable will read log files and insert (insert ignore) rows into an existing database table. Since it only does inserts, despite the name of this function, it is not just for MySQL tables, the included SQL should work with whatever database table type you have. |
|||
The executable has a mandatory single parameter that tells it which log files to read, there are only 3 possible parameters ("dayfile", "monthly", or path to a file). It needs to know what locale (or culture settings) it is to use to work out what character separates each item in the log file list. It also needs to read your Cumulus.ini file, as it takes these "input parameters" from MySQL section in that: |
|||
*Host |
|||
*Port |
|||
*User |
|||
*Pass |
|||
*Database |
|||
*MonthlyTable |
|||
*DayfileTable |
|||
=== Daily summary log file === |
|||
# Use the feature in the admin interface: |
|||
#* Settings menu |
|||
#* MySQL settings page |
|||
#* In '''Dayfile.txt upload''' section, give your database table a name, or accept default ''Dayfile''. |
|||
#* Click '''Save''' to ensure this setting is updated |
|||
# Now scroll down to '''Create database table (save settings first)''' |
|||
#* Here click '''Create Dayfile''' |
|||
# Now you have a database table ready, you can use the executable to read all lines in your '''CumulusMX/data/dayfile.txt''' log file. |
|||
# Open a terminal display (if you are using Windows then, open a Command Window, a Windows Powershell window, or a Windows Terminal window) |
|||
# Run this executable in that terminal display (or command window) by using '''sudo mono ExportMySql.exe daily''' or <tt>ExportMySql.exe daily</tt> depending on device. |
|||
# In the terminal display (or command window) you will see '''Parameter = daily''' confirming what you entered and in the line below that a rapidly updating code that is the primary key displayed for each row it tries to insert into the table. If that primary key already exists in the table, it will still show the key, but no insert will take place. |
|||
#If you want MX to continue adding new rows to this database table, still in the admin interface, still in MySQL settings page: |
|||
#* Return to '''Dayfile.txt upload''' section, and select '''Enable'''. |
|||
=== Standard Log files === |
|||
# Use the feature in the admin interface: |
|||
#* Settings menu |
|||
#* MySQL settings page |
|||
#* In '''Monthly log file upload''' section, give your database table a name, or accept default ''Monthly''. |
|||
#* Click '''Save''' to ensure this setting is updated |
|||
# Now scroll down to '''Create database table (save settings first)''' |
|||
#* Here click '''Create Monthly''' |
|||
#If you want MX to continue adding new rows to this database table, still in the admin interface, still in MySQL settings page: |
|||
#* In the same '''Monthly log file upload''' section, now select '''Enable'''. |
|||
# Now you have a database table ready, you can use the executable to read all lines in either one (if path to that file is in parameter), or every (if parameter is monthly) standard log file. |
|||
#*If the parameter is "monthly" it will look in folder '''data''' for every file it can find with a file name of datestring + "log.txt" where datestring is a 3 letter code (in your locale) for each month (1 to 12) followed by a 2 digit year (from "00" to "99") so that is how it finds every standard log file in the folder. |
|||
# Open a terminal display (if you are using Windows then, open a Command Window, a Windows Powershell window, or a Windows Terminal window) |
|||
# Run this executable in that terminal display (or command window) by using '''sudo mono ExportMySql.exe monthly''' or <tt>ExportMySql.exe monthly</tt> depending on device. |
|||
#* Alternatively, replace '''monthly''' parameter by a full path to a single standard log file, and it will process just that log file. |
|||
# In the terminal display (or command window) you will see '''Parameter = monthly''' confirming what you entered and in the line below that a rapidly updating code that is the primary key (omitting the first two digits of the year) displayed for each row it tries to insert into the table. If that primary key already exists in the table, it will still show the key, but no insert will take place. So you can run this again to pick up any additions to the latest log file since the original run. Also notice that if you use the parameter "monthly" the order in which it will process different standard log files is not predicable, they probably will not be in any particular order, but as one feature of SQL databases is that the row order is not able to be determined, it does not matter if rows are not added in chronological order. |
|||
#* It is worth noting that it is safe to run this procedure while MX is also running, because this procedure only updates log entries that exist as this procedure reads the logs, and MX only adds new entries to the log and at the same time uploads that new entry (if enabled) to the database table. |
|||
Please be aware that the transfer to the database table adds two columns where bearings in the original log file given in degrees are output as compass directions, and these use up to 3 letters of how the compass directions are defined in the '''strings.ini''' file. Thus the number of columns in the database table will be at least 2 more than the number of fields in the log files. It is also important to stress that whilst the database table must contain one column defined for each field (plus the extra 2) being uploaded, you can add even more columns to your table if you want and populate those some other way. For example, I have added a Canadian Humidity Index (Humidex) column which is not in the standard logs, but is calculated by Cumulus, and can be calculated from columns that are uploaded from the standard log. Humidex is not uploaded by either ExportMySQL or the normal CumulusMX process, but neither objects to extra columns being there. |
|||
When testing this, I had some log files produced by various old versions of Cumulus 1 in my MX data folder as well as the log files in has generated since I swapped to MX. Plus I had used a PHP script to add feels like to those log files produced before version 3.6.0 and to correct feels like for those log entries made by versions 3.6.0 to 3.6.9 inclusive because they used a different formula to the one being used from version 3.6.10. This php script is a web page with a HTML form and can be obtained from the forum in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 Create Missing for MX] |
|||
I notice that the database rows produced by those short log file lines produced by say version 1.9.0 had nulls entered for all subsequent columns, except '''Feels Like''' and this column was initialised at 0.0! |
|||
For those log files produced by the final version 1.9.4, all columns are populated although feels like is set to 0.0. |
|||
= Library software = |
|||
For most Cumulus users, this whole section can be skipped, but I have included it for those few users who have a technical slant and might want to understand more. |
|||
Cumulus MX uses '''library''' software (i.e. software written by others and made available by the provider and often also by other content delivery nodes or 'cdn') for a lot of the standard functionality. The library software for the admin interface and the separate library software for the standard web pages are both mostly included in the distribution zip, although some is used via a link to a cdn. |
|||
Many of the libraries included by MX are very obsolete. However, Mark Crossley, the current developer, said the following on 30 Sept 2019: |
|||
"Chasing the latest versions of all the packages for the sake of it is a thankless task, and requires considerable effort to regression test each update. I am only updating packages when required to fix issues or for platform compatibility." |
|||
Just to mention the other side of this balance. It is difficult to code an addition to MX that works with these obsolete versions of libraries. All documentation provided by providers of the libraries relates to current versions of the packages (and what is documented to work now often will not work with obsolete versions). The documentation for the packages that are no longer supported is only available in archive sites if available at all. |
|||
==Library Software for the MX engine== |
|||
The distribution zip contains various '''.dll''' files and these are the libraries used by MX itself. The exact mix of libraries included has varied at various times, the list below is a snapshot of those included at the version that was investigated when this article was extended to include this section, and may not be right for the current MX version. |
|||
===Devart=== |
|||
The two files used are both related to the database functionality of MX. |
|||
===FluentFTP=== |
|||
As the name suggests, this is used by MX for controlling the file transfer functionality. This component was first introduced at version 3.0.0 build 3045 (adding more functionality than that available from System.Net.FtpClien, which was used at earlier MX builds), and FluentFTP.dll has been updated at some subsequent MX releases, see the announcements for details. |
|||
===Linq=== |
|||
Language INtegrated Query is used to work with sequences of items and pick the ones that are needed, putting them into output format required. MX uses two files in connection with preparing output for Twitter. There is a third Linq file for other processing. |
|||
===MQTT=== |
|||
When MX added capability to talk to other devices using the MQTT protocol, it added this component for that optional functionality. |
|||
===Newtonsoft=== |
|||
This component is used for processing JSON strings. It is a very popular choice for developers, and used therefore very widely. However, '''SystemText''' has superseded it, so MX is using an obsolete method. |
|||
===Renci SSH=== |
|||
This component is server connection software, it is what processes the host name, password, and so on. |
|||
===SQLite3=== |
|||
This is used for all interactions with the [[Weather Diary|weather diary]]. |
|||
===HTTP=== |
|||
These files handle the optional HTTP functionality. |
|||
===Unosquare=== |
|||
The EmbedIO file is open-source software that handles the web-sockets functionality of MX. The Swan file is open-source software that handles JSON formatting and threading of tasks in MX. |
|||
== Library software for admin interface == |
|||
As the following sections reveal, MX uses external libraries rather than writing its own code whenever possible. |
|||
*However, that does not mean MX is good at meeting development standards. |
|||
*MX only implements small parts of the functionality of most libraries, the minimum to make a feature work, not all the features available to make it work well |
|||
*MX does not use the latest versions of libraries |
|||
*MX does not attempt to obey guidance for good user interaction, and although validation is being added, many parts of MX do depend on user understanding what is valid |
|||
*MX does not make provision for screen readers and other accessibility aids. |
|||
===Alpaca=== |
|||
#Alpaca software is effectively a programming language extension to help people design forms like those MX uses for all its settings, and as a Cumulus user you really don't need to worry about it. |
|||
#It is used for most settings screens. See http://www.alpacajs.org/ for more information. The latest version there is 1.5.27 released on 14 May 2019. |
|||
# MX uses Alpaca Release 1.1.3 from https://github.com/gitana/alpaca which was first released on 15 May 2014. Although some individual components have been updated on that github URL, and elsewhere, MX has not incorporated these. |
|||
===Bootstrap=== |
|||
#Also known by some as ''Twitter Bootstrap'' which gives a clue as to its developer and to its origins as an internal tool for those building Twitter, that company still keep making updates as it is now the most popular styling library of all those available widely. |
|||
*The simplest way to think about this package is as a standard set of styling promoting easy responsive (means adapts to screen dimensions) web site design. |
|||
*To give just a few examples, it defines a standard way to represent buttons, form components, lists, navigation, and breadcrumbs. |
|||
*MX uses Bootstrap version 3.3.7, which is very restricted in what it offers |
|||
**Bootstrap version 5 is available (http://getbootstrap.com), so MX is using an obsolete library |
|||
** Bootstrap version 4.5.0 (Bootstrap 4 released as alpha in August 2015, beta in August 2017, and with fully working releases frequently from January 2018) was very widely praised for its improved functionality, and ability to work with latest jQuery and multiple modern devices/browsers. |
|||
*MX does not implement key features of Bootstrap like colouring text according to what it represents (primary, secondary, information, warning etc.) |
|||
===dataTables=== |
|||
#When MX sends out multiple lines of a log file to view or edit, the application programming interface (api) that transfers the information from the MX engine sends it in dataTables format for display on the web page in the admin interface. |
|||
#Thus dataTables does all the work of providing the ability to present the data in a HTML table, the functionality to move between multiple pages needed (as MX sends only up to 10 lines of a log file at a time to the admin interface). |
|||
#The free version of dataTables used by MX lacks the most useful functionality that needs a subscription licence. For example, its editing functionality requires a subscription. |
|||
===altEditor=== |
|||
*This is an editing tool that can read what is in dataTables, create what it calls a modal (a pop-up dialog) where rows can be added, edited, or deleted individually. |
|||
** MX when it added editing of log files at version 3.4.5 - Build 3069 (Friday 13 March 2020) adopted this software as it was free (although Mark Crossley said in his release notice: '''"The main thrust of this release is to add some log file editing capability to Cumulus MX. It works on all three log file types, but it is fairly basic at present. You can edit or delete lines in the files. The editing has to be done via pop-up dialog.''' |
|||
**'''I only found two libraries that support JQuery dataTables editing, one is very comprehensive - but costs $$$ - the other is free. The free version does not currently support in-line editing of the table which is a shame.''' |
|||
** '''If any web guru out there can come up with a better solution please post about it on the forum, or send a pull request."''') ''(By the way, it is possible to provide in-line editing and make it work with the existing api interface, but making it compatible with the obsolete software used by MX is hard).'' |
|||
*The single line of fields that is result of an edit or deletion done on the modal is sent back via another api to the server (the MX engine in our case) and that then regenerates the dataTables in the state after whatever action was done, sending back again up to 10 lines for the same page as before. |
|||
*As it happens there is another JQuery dataTables editing tool, but it has not been maintained since 2012. It is found at https://github.com/NicolasCARPi/jquery_jeditable, but the documentation is now only available in an archive at https://web.archive.org/web/20200615000000*/https://appelsiini.net/projects/jeditable. It is designed for editing table cells, so it does not involve any pop-up dialog. |
|||
===datepicker=== |
|||
#Although modern browsers generally will generate a calendar type interface when they meet an entry field defined as a date, this date picker software ensures all MX users see exactly the same interface for date selection needed for both the standard log and the extra sensors log which are monthly log files (a new one is created each month). It is used for picking which standard (monthly) log or extra (monthly) log is to be viewed by selecting a month and year only. |
|||
#It is also used for selecting individual days in the weather diary editor. |
|||
===editable grid=== |
|||
*As the name perhaps suggests MX only uses this for the extra web files screens where you can make selections within a grid like interface. |
|||
*I suspect it could enhance some other functionality in the future. |
|||
===handlebars=== |
|||
#Put simply this is a simple HTML generator based on templates. |
|||
#I have not found any file in the admin interface actually using this, but I am scared to delete it just in case it stops something working. |
|||
===x-editable=== |
|||
*Put simply, this allows in-place editing of web pages using bootstrap. |
|||
*In MX it is used for the record editing screens where you adjust the extreme values. |
|||
===jQuery=== |
|||
#The admin interface uses version 1.9.1 of this JavaScript based library. |
|||
#* At the time of typing this, the current jQuery is version 3.5.1. |
|||
# Of all the old versions of jQuery to choose, MX has picked the only version that the developers withdrew due to an error when they released it. |
|||
# Version 1.9.1, has a serious error in its code, because the developers accidentally combined code from two significantly different versions when they created the release file. |
|||
#*This reveals itself in two ways: |
|||
#*# The error handling does not work. |
|||
#*# It tries to load another script that does not match. |
|||
# Consequently, the developers quickly removed it, but it remained available from Contents Delivery Nodes, which is where MX has found it. |
|||
# Not surprisingly, the authors of jQuery strongly advise all 1.9.1 users to move to a later version. |
|||
#* Unfortunately, there are interdependencies between all the library code used by MX, so you cannot simply update this component (see next library item). |
|||
===Jquery Template=== |
|||
* This is also obsolete, and therefore will only work with old versions of jQuery (1.4.2 to 1.11.0). |
|||
*It was originally available from jQuery downloads, but they now offer '''jsRender.js''' for this functionality. |
|||
* It basically is used to bind the contents of objects (like array elements) into particular locations within HTML. |
|||
===SteelSeries=== |
|||
MX uses a modified version of the [[SteelSeries Gauges|steel series]] library made available by Mark Crossley for all the gauges (see dashboard and gauges pages of the admin interface) in MX. |
|||
===Highstocks=== |
|||
The odd one out is '''Highstocks''' (that includes HighCharts) |
|||
*This is loaded from a Contents Distribution Node (CDN), but it is still pinned to obsolete versions of the basic script and its themes. |
|||
*This means that the Charts page in the admin interface will only work when there is an internet connection working to permit download of this software |
|||
*If you need to view your admin interface where an internet connection is not available: |
|||
Then you need to edit the interface file... |
|||
'''<CMX_Folder>/Interface/charts.html''' ''Change lines 20,21 from'' |
|||
<pre><script src="https://code.highcharts.com/stock/8.0/h ... "></script> |
|||
<script src="https://code.highcharts.com/themes/grid.js"></script></pre> |
|||
''to'' |
|||
<pre><script src="webfiles/lib/highstock/js/highstock.js"></script> |
|||
<script src="webfiles/lib/highstock/js/themes/grid.js"></script></pre> |
|||
== Library software for your web server == |
|||
The '''webfiles/lib''' folder includes a number of software library items that are needed for the standard web pages included in the MX distribution. |
|||
This is fine if your web site is purely for the provided standard web pages, but if on your web server you also have web pages from third parties, or you have written your own web pages, then you may get conflict because all the library items used by MX are obsolete versions, and in one case MX uses a version of library software withdrawn by its originators due to compile error (so only available form some CDN who provide obsolete versions as its originators insist it must not be used)! |
|||
#'''Highstock''' |
|||
#*At the moment, as hinted in previous section, there is an old version of Highstocks included in the '''webfiles/lib''' folder. |
|||
#* However, that is not used, instead (like admin interface) an old version is loaded from a CDN. |
|||
#*Be careful if you also load the current version for use on web pages not produced by MX, that the browser does not try to reuse your latest version and not recognise that MX wants an older version. |
|||
#'''jQuery''' |
|||
#*Be aware that MX distribution is not consistent as different web pages use different obsolete versions of jQuery. |
|||
#*#'''gauges.htm''' includes <tt><script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script></tt> to load version 1.8.2 from the Google Content Delivery Node |
|||
#*#*You will find at the end of this web template, if you bother to read it, some instructions about how you can change that CDN load, but the instruction is neither grammatically correct, nor understandable for me. |
|||
#*#'''trends.htm''' includes <tt><script src="lib/jquery/jquery-latest.min.js"></script></tt> to load version 1.9.1 of jQuery loaded from this badly named file, there is nothing ''latest'' about it. |
|||
#*#*Ironically, the old version of jQuery that has been selected by MX to be included in the '''webfiles/lib''' folder is the '''only old version that jQuery themselves withdrew, and warn you not to use''' because the error handling content is from a different jQuery version to the correct handling content making the two parts of the code incompatible. In other words, this version only works if everything is set up correctly and no error handling is invoked. |
|||
#'''Steel Series''' this is treated as library software, as it is copies of that separate product by Mark Crossley. |
|||
#* Most of the files are exactly as that was in revision 0.14.13 made 30 January 2015. |
|||
#* '''gauges.js''' has one tweak for MX, it defaults to not having chart rollovers (for Cumulus 1, this file had default to showing the images of graphs that were uploaded by Cumulus 1). |
|||
= The provided web pages = |
|||
Setting up a web site is covered in [[Website_setup|this wiki page]] and the pages linked from there. I won't repeat that, but will try to explain below the MX context of the various files involved. MX will produce web pages locally even if you don't have a remote web site to display them on. You can view the web pages created in the web folder using a browser. |
|||
Cumulus MX provides a set of web templates, images, and json files, in '''\CumulusMX\web'''. The first of these are called templates (and have a 'T' at the end of the file name before the extension) because they include both text and web tags. MX will '''process''' these templates as it creates a web page (file name without a "T" in same web folder), during that processing, any text in the template is copied into new file without change, any web tags found in the template are parsed and the correct value is placed in that position in the output file. If you have a web server, then MX can process these files and upload them for you (by default using File Transfer Process) providing you specify the host, port, protocol, directory, username, and password, for the upload process to use. If you don't understand any of these terms, then this is not the place for explaining them, but generally if your web space is supplied by a provider, they will be able to tell you most of these settings, and you will choose the directory name. If you have set up a web server yourself, then you should know the required settings. |
|||
The web templates included are based on designs by Beth Loft used for Cumulus 1. Remember, Steve Loft (who wrote the original Cumulus software) said "''They exist because they're our web pages, and they're really only included with Cumulus as examples of how the web tags work. It never occurred to me that most people would simply use the supplied examples instead of creating their own pages!''" |
|||
The templates are written in fairly simple Hyper-Text Mark-up Language, designed to help people see how to write their own web templates. Here is a list of web templates provided: |
|||
*indexT .htm |
|||
*todayT.htm |
|||
*yesterdayT.htm |
|||
*thismonth.htm |
|||
*thisyear.htm |
|||
*recordT.htm |
|||
*monthlyrecordt.htm |
|||
*gaugesT.htm |
|||
*trendsT.htm |
|||
All these templates (except gaugesT.htm and trendsT.htm) include a table for showing values and styling that gives a graded background colour. The tables include a navigation row with links to the other pages in the set. '''That navigation line fixes the width of the table''', and you will realise it was designed in the days when all monitors were a standard shape. Therefore the standard web pages as provided cannot adapt to the range of devices we use for viewing web pages nowadays. There are a selection of alternative web page sets available [[:Category:User_Contributions#web_template_complete_set_replacements|on the User_Contributions page]], and some of these are responsive and adapt to the width of the device they are being viewed on. |
|||
The gaugesT.htm is a template similar to [[SteelSeries Gauges]] although the latter is designed to work with a range of software and the former is specific to MX. '''As supplied in MX if you mouse over the provided gauges appearing on your web site you will see a box with figures, not a graph as is seen with the general steel series gauges''', but there are some other differences such as how the figures are supplied for the displays. The remaining template '''trendsT.htm''', creates a structure that can display graphs. The data for all the graphs that can be displayed is contained in the various '''json''' files in '''\CumulusMX\web''', these files are also processed by Cumulus so latest values are added, and then uploaded so the web page produced by this template can use them. |
|||
The image that is provided in '''\CumulusMX\web''' is ''MoonBaseImage.png'', MX can be set to use that to generate (on MX start-up and on the hour) "moon.png" which it then can FTP to your web (also on the hour). |
|||
== To set up your web server for the first time == |
|||
When you first want to use Cumulus web pages on your web server, you need a number of static (unchanging) files to be put onto your web server. The web pages that MX uploads for you reference that static files and will not look right without them. The files that only have to be uploaded once are found in '''\CumulusMX\webfiles''' and its sub-folders. You don't create a folder called webfiles on your web server, but you put the files and sub-folders into position relative to where MX will upload the htm files. |
|||
To do this, you will must invoke a FTP process outside of Cumulus, MX does not include any functionality to do this one-off upload for you. The filezilla client is a popular choice as it has probably the most friendly graphical user interface, although other software to do this is also available. You may prefer a tool that lets you do the uploads from a command line without requiring working with a graphical interface. |
|||
#The static files to be uploaded include the standard styling file '''\CumulusMX\webfiles\weatherstyle.css''' which you place in the directory specified for the uploads. |
|||
#Next you have three sub-folders, each of those sub-folders need to be replicated '''within''' the directory specified for the uploads. |
|||
#*For example '''\CumulusMX\webfiles\images\picture.jpg''' will be stored in a "images" sub-directory of the upload directory and is used as the background image for web pages. |
|||
#**There is nothing to stop you creating your own "picture.jpg" (instead of uploading the supplied one) and then Cumulus web pages will use that for the background image on each page. |
|||
#*Similarly '''\CumulusMX\webfiles\js\cumuluscharts.js''' needs to be stored in a "js" sub-directory of your upload directory (this is the script that allows you to change the chart shown on the trends page and uses the appropriate json file to populate it with data). |
|||
# The "lib" sub-folder contains further levels of sub-folders all to be replicated on your web site. |
|||
# The '''trends.htm''' web page also loads some library software from an internet Content Delivery Network (cdn) to invoke the JavaScript based Highstocks library. |
|||
== Operating a web site with uploads from MX engine == |
|||
=== The standard web pages === |
|||
*If you want to operate the 'standard' web site, then just the same as with Cumulus 1, you will need to upload the ''contents'' of the '''webfiles''' folder from the zip file (don't upload the containing '''webfiles''' folder itself). |
|||
**Note that the MX web files are not the same as the ones for Cumulus 1, so make sure you upload the MX files if moving from Cumulus 1 to MX. |
|||
**The standard gauges are now the SteelSeries gauges. The default versions do not display a graph when you hover over a gauge as happened when you added the stand-alone Steel Series gauges to Cumulus 1. |
|||
**The trends web page in Cumulus 1 relied on that software generating graphs as images. In MX, the software generates files with time and value pairs, these are stored in json format, the trends page then uses a library package (Highstocks) to draw graphs from those data pairs. |
|||
=== Alternative ways to obtain web pages === |
|||
You can choose to use some of the alternative web pages available from third parties and described [[:Category:User Contributions|on User Contributions page]]. |
|||
=== Using your own web pages === |
|||
*Of course you can use your own web pages, instead of the standard ones. Assuming they need to include figures that are available as web tags, there are three alternative ways to implement this: |
|||
*#MX can process template files with a HTML structure and those web tags in the structure where values are required just as it does with the standard templates, and MX can upload the resulting web pages at either the real-time interval, the standard interval, or after end of day. All of this is covered on the [[Customised_templates|Customised templates]] page in this Wiki. |
|||
*#MX can process a file with a string of web tags mirroring the realtime.txt option in MX, and upload the resulting file so your web pages can use JavaScript for a one-off insert of the values or an Ajax routine to update the web page at a fixed interval. |
|||
*# Alternatively, you can use template scripts processed locally by MX that don't create web pages, but are uploaded by MX at either the real-time interval, the standard interval, or after end of day. These scripts simply initialise script variables with values obtained from web tags. You then independently have a set of web pages resident only on your web server (they don't exist where you run MX) using a combination of HTML and script content that bring in the script(s) with the variables by the appropriate syntax. All of this is covered on the [[Php_webtags|PHP web tags]] page in this wiki. As it suggests there, you might therefore have several files processed by Cumulus MX at these different intervals, converting the web tags into script variables, and then use AJAX (JavaScript that may use json format to bring in the variables) or PHP (using <tt>'require_once 'filename';</tt> syntax) to put those variables into a web page. |
|||
You may find [[PHP|this wiki page]] useful for understanding more about the different script languages. |
|||
= MX End of Day Process = |
|||
I have added this section, because this process has given me some headaches. |
|||
== My version == |
|||
If you write custom SQL, or have a template being processed at end of day, then what I find strange is that web tags related to system date report the new date, but other web tags report weather derivatives from the old day. Put another way, all the date related tags change at start of rollover, but all the weather related web tags change at end of rollover. |
|||
However, it is not quite as simple as that, the month and year are reset before the extra web files are processed (so they cannot use monthly web tags at end of month, nor yearly web tags at end of year). The complication is that in the extra web files you can use '''<currentlogfile>''' (and from build 3087 '''<currentextralogfile>''') and these pick up the old month/year. Now you see why I found it hard to digest, and why I wanted to write it here to make it easier for others. |
|||
If you use Custom SQL and therefore have to quote web tags, the SQL should use monthly and yearly web tags related to previous day, but all the weather tags it uses must be those for current conditions or today. Yes it is confusing. |
|||
As part of the so called "end of day" process, MX (just like Cumulus 1) creates a start of day back up in the daily sub-folder. So on first day of a new month (and new year sometimes), the backed standard log files (and extra sensor log files from build 3087) are for the month that has just started, there is no back-up of the old month. |
|||
== The official version == |
|||
Mark Crossley says the MX day reset does this (at version 3.5.x)... |
|||
<tt> |
|||
Reset midnight rain |
|||
Entering Day Reset (message about current day of month, at this stage web tag <#metdate> changes to new date) |
|||
Day Reset (message about date ending, time shown as 00:00:00 because time not defined, not because it is midnight, it might be 9am or 10am) |
|||
Run EOD custom SQL |
|||
Save dayfile entry (uses what is still in today.ini that includes old date, i.e. what is now in web tag <#metdateyesterday>) |
|||
Write monthly & yearly file entries |
|||
Write any new daily extreme records |
|||
if day of month = 1 then: copy month.ini to saved file, reset monthly figures |
|||
if day of month = 1 and month = 1 then: copy year.ini to saved file, reset yearly figures |
|||
Copy todays high/lows to yesterdays |
|||
Reset todays high/lows to current |
|||
Write today.ini & yesterday.ini |
|||
Create NOAA reports |
|||
Execute user daily external program |
|||
Process Extra EOD files |
|||
</tt> |
|||
But independent of above EOD thread that occurs on the rollover hour, the '''normal interval''' and '''hourly processes''' thread is seeking to run at same time, whether that happens at same time depends on processing capability and whether it can process multiple threads. |
|||
What actually happens in above list depends on your settings, and if your FTP interval is synchronised with the logging interval. |
|||
= SPECIAL CONSIDERATIONS - Text by Steve Loft = |
|||
== Restrictions in MX for decimal separators == |
|||
On the subject of decimal and list separators, there are a couple of issues which users of decimal commas may encounter. |
|||
#The first is that there may be an issue with some of the user interface not working correctly. Please report these issues and I will fix them. There may be aspects of the displays that I cannot change (because the package used does not support decimal commas) but it should be possible to at least get it working. |
|||
#The second issue with decimal separators only affects the Raspberry Pi (as far as I am aware). There is apparently an issue with a version (3.2.8) of the Mono package on Raspbian 'hard float' where it cannot parse values using decimal commas. If this does turn out to be an issue, there are a number of possible workarounds until the Raspbian package gets updated. One workaround is to use the 'soft float' version of Debian instead. Obviously, this will have performance issues, but is probably the easiest. The second workaround is to build Mono from the latest sources, see http://www.mono-project.com/docs/compiling-mono/linux/. I am told that this fixes the problem. Another possible workaround would be to find an already fixed binary package, but I don't know if one currently exists. |
|||
''PLEASE NOTE: The issues that Steve describes seem to have gone away with currently available versions of Mono; update your Mono if you are using an old version and encounter problems.'' Like any software, Mono might have bugs at a particular version, and sometimes you might need to swap to an older version if the current version has an outstanding issue. |
|||
== If you want to use your Cumulus 1 data with MX == |
|||
If you use decimal commas in your Cumulus 1 data, you will need to edit the .ini files to change the decimal commas into periods/full stops, because '''Cumulus MX always expects periods/full stops in .ini files''' ''regardless of the locale in use''. The other data files will be OK - assuming you are using the same decimal and list separators in MX as you used in Cumulus 1 (i.e. the same locale). If you try to switch to a different locale, then your data log files will of course no longer be in the correct format, so you would need to edit all of your files. You can select the locale for MX to use as a switch parameter when it starts up, see earlier on this page. |
|||
== A note to Davis owners == |
|||
I am experimenting with the use of the LOOP2 packet. The current code uses this for two purposes. First, it uses the 'peak 10-minute gust' value, to avoid the problem where a gust might be missed (although hopefully this will not be such an issue with Cumulus MX as it does not use the Davis DLL), and secondly it uses the 'absolute pressure' value to make calculation of 'altimeter pressure' easier and more accurate. This is mainly used if you upload to CWOP. |
|||
The LOOP2 packet is supported on the VP2 with firmware version 1.90 or later, and on the Vue. If you have a Vantage Pro (i.e. the original 'VP1'), or a VP2 with pre-1.90 firmware, or if you are using Virtual VP, none of these support the LOOP2 packet. In these cases, you should edit cumulus.ini and add a line to the [Station] section: |
|||
UseDavisLoop2=0 |
|||
With this setting, Cumulus will revert to calculating the 10-minute gust value itself from the individual wind speed readings, but it will not currently attempt to calculate altimeter pressure correctly, it will simply use the sea-level pressure instead. This is likely to be an issue if you are at high altitude and you upload to CWOP using Cumulus MX. |
|||
Also for Davis stations, I have assumed that people using millimetres in Cumulus have a metric rain gauge (0.2 mm per tip), and those using inches have a 0.01" rain gauge. This can be over-ridden by adding a line to the [Station] section of Cumulus.ini: |
|||
VPrainGaugeType=0 |
|||
or |
|||
VPrainGaugeType=1 |
|||
Where 0 is a 0.2mm gauge and 1 is a 0.01" gauge. Note that changing this after MX has already read some data may cause your rainfall reading for today etc to change considerably, so you will need to correct that. |
|||
= Web Tags and related features = |
|||
Almost all of the [[Webtags|web tags for all Cumulus flavours on this Wiki page]] that you could use in Cumulus 1 are also supported in Cumulus MX. |
|||
Each new build of the beta MX has increased the range of web tags it supports. Since MX has come out of beta, new versions have not only implemented the remaining tags from Cumulus 1, they have also added new tags not previously available. For full details see the [[Webtags#Differences_between_Cumulus_1_and_Cumulus_MX_.28Cumulus_3.29:|web tags]] article, but a quick précis follows in next few sections. |
|||
== All builds of MX == |
|||
The ''''format' parameter''' on the date/time output modifier for web tags is unfortunately different, because many of the characters used are different. See [[Webtags#List_of_allowed_modifiers|the modifiers list]] page of this Wiki. |
|||
Note that this difference in date/time modifiers also affects how you specify the '''NOAA report''' file names. For example in Cumulus 1 you can specify a 2 digit month number by either 'mm' or 'MM', but MX (later versions) has to change the former to the latter as MX uses 'mm' for minutes. The same applies to using 'mmm' or 'MMM' for 3 letter month abbreviation in Cumulus 1, only the latter works in MX, so MX (later versions) will adjust that. If you are using an older MX version, you should upgrade to latest as you are missing a lot of functionality, but while you use that old version, ensure that your file names for NOAA reports do use the correct modifiers for MX. |
|||
== Beta builds of MX == |
|||
The following web tags were not available or worked differently: |
|||
*The individual 'record set' tags such as <#TempRecordSet> etc did not work (because the interface then had no indicators for new records and no way to reset them). |
|||
*The <#newrecord> tag does work, but works differently, it turns itself off automatically after 24 hours. |
|||
*Some of the 'system status' web tags do not work: <#CpuName>, <#MemoryStatus>, <#DisplayMode>, <#DiskSize> and <#DiskFree> |
|||
*The <#txbattery> web tag has no content currently. Using it with a 'channel' parameter causes a 'token error'. |
|||
*The snow tags were not available as there was no '''Weather Diary''' |
|||
== Current builds of MX == |
|||
The web tags you have depend on which build you are using: |
|||
From beta version 3.0.0 - Build 3046 of 2 Jan 2019 |
|||
* added <#snowdepth> tag processing |
|||
* added '''diary.db''' file |
|||
From beta version 3.0.0 - build 3047 |
|||
* Web token parser updated to cope with html tag characters "<>" in the format string e.g. <#TapptempH format="dd' 'MMM' 'yyyy'<span class=\'xx\'> at 'HH:mm'</span>'"> |
|||
*All record Value tags should now return '---' and Date tags '----' until they are first set. |
|||
*<#MoonAge>, <#MoonPercent>, <#MoonPercentAbs> - all given new 'dp' and 'rc' parameters. |
|||
From version 3.1.1 - build 3054 |
|||
*Adds new web tags <#snowlying>, <#snowfalling>, both provide 1|0 responses |
|||
From version 3.2.0 - build 3056 of 19 November 2019: |
|||
* Enables alarms as per Cumulus 1 |
|||
**New Alarm page under Settings |
|||
**Alarms are shown visually on the dashboard |
|||
**Due to browser restrictions, alarm sounds on the browser page may require you to click a button on the first alarm in order to hear it. |
|||
***You can add the MX admin site to your browsers list of sites allowed to play sound automatically. Your browser should "learn" that you want to allow sounds to play automatically. |
|||
*** Alarm sound files should be placed in the /interface/sounds folder, they must be a browser compatible format (mp3 are good). The alarm settings for the sound file should be just the filename without any path |
|||
*Lots of new web tags not available in Cumulus 1, see release announcement for details |
|||
From Version 3.2.2 - build 3058 |
|||
*Implements the missing <#txbattery> web tag |
|||
From version 3.5.1 - build 3072 of 10 April 2020 |
|||
[[Category:Log Files]] |
|||
*Implements the tags that indicate when records are broken |
|||
* You configure whether if a record is set it turns off after 24 hours or a different period. |
|||
Revision as of 07:33, 3 September 2020
If you have any suggestions for improving this article, either edit this page yourself, or put your suggestion in the correct Support Sub-Forum. Thank you.
Introduction
If you have been using Cumulus 1, and now wonder whether to try using MX, be aware that:
- Currently, more people are still using Cumulus 1 than are using MX; but during 2020, many have experimented with MX, then successfully moved to MX, just a few have gone back!
- Cumulus 1 is a stable release, it has functionality that is not available in MX:
- Select a graph
- View period
- provided you are happy to keep running Cumulus 1 on a Windows pc, and use a weather station that is compatible with Cumulus 1, then you don't have to change over
- Cumulus MX is still under development, a change in a particular release may introduce bugs
- MX does however have additional features not available in Cumulus 1
- it can run on Unix-derived (Mac, Linux, Raspberry Pi operating systems) devices as well as Windows devices
- it can work with newer weather stations and newer sensors
- it reports feels like, Canadian Humidity Index
- it has built in updating of database tables
- it can output to more external sites
- it has MQTT, HTTP, and other extras, built in
Please see Moving from Cumulus 1 to MX for more information.
What does Cumulus MX do?
If you are new to Cumulus, then you will be wondering what benefits Cumulus MX has over other weather recording software. In that case start by reading the article that introduces Cumulus.
You may want to read that article first, that that will explain what Cumulus software can do and perhaps help you to:
- Learn what it can offer you if you are unsure whether Cumulus is right for you
- If you already use Cumulus, this might remind you of everything it offers.
If you do go off to read that article now, you will be linked back to this page.
This article
This Wiki article was originally created by Mark Crossley and contained exactly what Steve Loft said in the MX early builds support forum when he first started experimenting with Cumulus MX and access was restricted to those willing to experiment with his tests.
In this rewrite, somebody who used to use Cumulus 1, and experimented with MX, before moving over to MX first on a Windows PC and then on a Raspeberry Pi, has attempted to ensure this article contains everything I learnt from my experience as I explore more of the functionality of MX; and as I learn more from posts in the forum.
If you can correct any mistakes I make, have a suggestion as to how to make it clearer, can add anything I have not yet covered, or know something that I might not know, then please remember, anyone can update this article, I don't have any special access in the Wiki and any page I edit can be edited/corrected by anyone else.
During a period of my time in employment I was responsible for approving documentation on a large computerisation project, and later for supplying updated information for a public faced web site, and in both cases there were house style, and I probably continue to use that style. You might be afraid to add your contribution because my style is not the same as your natural one. Don't worry; as long as you use short paragraphs or bullet points, with lots of headings, then your contribution can blend in.
As this article has grown, I have been able to shorten it by moving material to new articles, hence you will see a lot of cross-references below. You may have suggestions for what else can be moved out of this article into separate articles?
If you have some ideas, but are unsure whether to apply changes, please use the discussion page first. If you have ideas, but want someone else to add them here, please write your suggestion in the support forum for wiki suggestions at https://cumulus.hosiene.co.uk/viewforum.php?f=38
Cumulus flavours
Cumulus 1
- Cumulus 1 was created in 2003 when Steve Loft moved to Wanlockhead and bought himself a weather station. Initial releases were very much tailored to his needs, changing what weather station it supported to match the make he was using.
- Subsequently, he made his software available to anybody, and enhanced what it did to suit not only his requirements buy also what other users asked for.
- The functionality of Cumulus grew and grew, there were a few bugs, and a few mistakes, but generally Cumulus software had a high reliability, and grew in popularity, especially when the internet made it easier for people to praise Cumulus on many unrelated sites.
- Increased popularity meant an increase in demand for more functionality, increased functionality made it more popular, and the spiral of development continued.
- Cumulus 1 had extensive help screens built into the package, it had an installation package, and produced a main screen when it was running that summarised the weather and gave access to all the settings and editors. This made Cumulus 1 very user friendly.
- Demand for enhancements soon exceeded the amount of spare time, Steve could devote to Cumulus outside his full-time job. A register of enhancement requests was created, so that Steve could track which he had implemented, but that register was lost when Steve moved to a different host.
- One widely supported request could not be implemented for Cumulus 1, to offer a version that would run on multiple operating systems, so Cumulus 2 was born (see below).
- Cumulus 1 development was halted for a while by the focus on Cumulus 2.
- Cumulus 1 development restarted when Cumulus 2 was aborted.
- The development environment for Cumulus 1 became obsolete, and Windows operating systems changed, making the development of Cumulus 1 more difficult, so Steve decided to have another go at a replacement (Cumulus 3).
- Development of Cumulus 1 ceased once Cumulus 3 (aka MX, see below) was able to work better than its C# predecessor Cumulus 2.
- Steve Loft decided not to release source code for Cumulus 1, so nobody else can develop Cumulus 1 any further.
- Consequently, Cumulus 1 functionality can not be changed, and without knowledge of how it was written, there is no ongoing support, just the experience of those who have used it, or are still using it.
- Fortunately, Steve Loft documented Cumulus 1 very well in the forum, and in the new Wiki that was started in August 2009, so Cumulus 1 continued to attract new users even when Cumulus 3 was made available as MX.
- When this article was first created in 2017, Cumulus 1 was still recommended for most users, and it would remain so for another couple of years.
- Even now in 2020, there are many, many more people using Cumulus 1 than are using MX.
Cumulus 2
- Steve Loft produced a Cumulus 2 where he tried to start again in September 2009. It was written in C# (which is the language used for MX), and it is fair to say that Steve did not find that new programming language easy, and in March 2010 he was really struggling to make Cumulus 2 work how he desired.
- Cumulus 2 did prove that a number of concepts (like separating "engine" from "admin interface") could work and it was a useful learning curve for when Steve decided to write Cumulus 3 (see below).
- One change that had been requested by several Cumulus 1 users was for better international viewing of web pages, with less dependence on time zones. To achieve this, one suggestion was that Cumulus should work in GMT (more widely known now as UTC). Cumulus 2 therefore read and logged all readings by UTC. Unfortunately, converting from local time used by weather stations, and most computer devices, never worked as smoothly as Steve Loft hoped, so this is one idea that was not adopted for Cumulus 3.
- Furthermore, Cumulus 2 never succeeded in getting some of the basic functionality like driving web pages to work, so it never offered much of the more useful functionality of Cumulus 1.
- But it was a good testing ground for new functionality and enhancements and regardless of whether they could be made to work fully in Cumulus 2, some were highlighting what Cumulus 1 lacked.
- In August 2010, the new features being tested in Cumulus 2 were added to Cumulus 1, and Cumulus 2 was discontinued.
Cumulus 3
- In 2015, Cumulus 3 also known as MX once it was made available to users, was experimental and it had limited functionality, much less than was available in Cumulus 1. This made MX innovative, but unfriendly.
- Consequently, at that time, most Cumulus users were using Cumulus 1, and just those wishing to take part in beta testing used MX.
- Steve Loft started development of MX while he was still in full-time employment, but as retirement approached he worked fewer days per week and was faced with the question as to whether to spend more time on MX or more time with his wife, Beth, exploring places.
- When he fully retired, a life on the road beckoned, and they started travelling. Work on MX decreased, and work on Cumulus 1 was no longer possible, as he was limited to what his laptop and internet connection at stops could cope with.
- Various people offered to help him with MX if he was willing to make his source code available. Initially, Steve did not want anyone else to interfere with his creation, but when he and his wife found a new home the priorities changed in favour of a focus on his new life, and he wanted to cease involvement with Cumulus software, its wiki and forum.
- Steve Loft who wrote and developed Cumulus 1 and MX while he was in Scotland, decided to cease to offer any support from his new home in France. After quite a while considering it, he decided to make the source code available.
- The various people who had offered to help develop MX now were able to see the source code and decide whether they really did want to get involved.
- One programmer launched Cumulus 4, a new approach. Work continued on this for a while, but as far as I know it never made it into a working system, and I believe like Cumulus 2, it is abandoned.
- Other programmers looked at the source and thought we can make MX useful by adding the missing functionality, both what Steve added in the source but never got into a public release; and the functionality that makes Cumulus 1 so popular but is missing in MX and makes it difficult for those who are using MX.
The MX future
- Mark Crossley was one of those who tried updating the MX source and producing a new release.
- In 2019, he made a successful first new release, and then focussed on adding some of the missing functionality. By 2020, he was not just adding in his own version of features that had been in Cumulus 1, he was also making MX talk to new weather station designs and deal with new sensors.
- During 2020 much extra functionality has been added to MX, and MX is now able to persuade Cumulus 1 users that it might be the right time to make the swap to MX.
- Cumulus 1 was designed to work with weather stations that were available when it was written, the technology used by stations, and the models available, have both been changing since then.
- The ongoing development is adding lots more functionality into MX, it can do a lot more with the the numbers it reads from weather stations, and it can be updated when weather station features change.
- Therefore, the advice to newcomers is to use Cumulus MX, sometimes called Cumulus 3, because there was a Cumulus 2 (that was abandoned) and sometime ago there was a start on a Cumulus 4.
- Similarly, the advice to established Cumulus 1 users is you should now consider a move to MX as you are now missing out on many features available only in MX.
- However, there are no instructions built into the MX package, so it is hoped that the update of this article will help people to understand MX sufficiently to use it both more easily and to maximum capability.
- Currently, Mark Crossley who has been responsible for all recent MX releases is able to answer questions in the support forum for recent MX releases, but this article will hopefully allow him to spend less time answering questions and more time improving MX (and more time for everything else in his life)
It would be wrong not to repeat what Mark has said here - MX is still not bug free, there is a lot more to correct as well as all the enhancements to cope with new weather station hardware.
There is a page (created in October 2018) listing MX Issues to be resolved, but I suspect it is out of date. If you look through the release announcements for 2020, yes there are a lot of new features being added, but there is even more work on resolving bugs.
Restrictions on who can use MX
MX makes extensive use of library packages:
- like bootstrap (cascade styling),
- datatables (display and manipulation of tables),
- JQuery (JavaScript package that provides code supports for multiple browsers and other libraries to work together),
- high stock (for drawing charts),
- datepicker (a JavaScript based routine for making date selection possible using a calendar type interface as not all browsers directly support that),
- handlebars (templates for generating HTML),
- alapaca (JavaScript from Gitana Software that generates interactive HTML5 forms),
- Steelseries (provides the gauges used),
- altEditor (for editing the log files) and a few more.
Fuller information on these is in Library software section.
Most of these are open software and free for personal use, but some have restrictions on commercial use requiring a licence. Consequently, MX does have to declare it is not for use on a commercial web site.
Message from Steve Loft about who can use MX
Note: The graphs used in Cumulus MX are drawn using Highcharts and they are free for non-commercial use only, i.e. you may not use them on a company web site, see http://shop.highsoft.com/faq/non-commercial for clarification. For this reason, and others, use of Cumulus MX in a commercial environment is expressly forbidden.
Please include a link to the Highstock web site (as the supplied web page does) if you use the charts under the terms of the non-commercial licence.
Documentation for MX
Cross reference to article on Moving from Cumulus 1 to MX (includes general issues regarding running on Windows)
Whether you have been using Cumulus 1, or are new to Cumulus, you may be tempted to install MX on a PC, or other device, running Windows Operating System; the article linked in the heading will help you.
You may be reading this article on MX as a Cumulus 1 user considering moving to MX, if so then it explains all issues when moving from Cumulus 1 to MX, including what functionality you will gain and what you will lose by moving to MX. The article will help you to decide whether to stick with Cumulus 1's stable release as many MX releases have bugs as the developer tries revising the code, to simplify it, as well as to introduce new features.
As a new Cumulus user, using MX on Windows (but with no experience of Cumulus 1), the separate article linked from the heading to this section will still interest you as it is a definitive guide to installing Cumulus MX on Windows.
Cross reference to article on Setting up Raspberry Pi for Cumulus
Whether you have been using Cumulus 1, or are new to Cumulus, you may be tempted to install MX on a Raspberry Pi. A Raspberry Pi (and similar devices from other manufacturers) is much simpler than a normal computer, but it can still run various operating systems that allow you to use it perhaps for both running MX and a web server. Find out more in the article linked in the heading.
Cross reference to article on Updating MX to new version
Whether you have not updated for a long time, or simply wonder whether you are updating the easiest way, follow the link in the heading for this section, to an article that focusses on all to do with updating from one MX version to another.
Critically, this linked article contains advice both for those updating each time there is a new release available and for those using an old version (perhaps when MX was still in beta) wishing to update to a new version skipping some in-between versions and need advice on whether to do it in multiple steps.
Cross reference to Cumulus MX FAQ
A new FAQ for MX has been started at another page. As I add the link here, the Cumulus MX FAQ is in a mess, but hopefully someone will have time to sort it out.
Meanwhile most MX specific questions are now covered by the updating of text on this "Cumulus MX" page.
Cross reference to MX Issues
The Cumulus MX Known Issues article was based on Steve Loft's forum post so based on an early beta version of MX, and there has been minimal attempt to update the contents as MX has been developed further and come out of the original beta. Anyone willing to check the linked article and do a further update?
Cross-reference to What to do when I have a problem with MX
The text that was here has been moved to a separate article, that makes it more accessible, please see What to do when I have a problem with MX article
Cross reference to Administrative Interface
This requires a whole topic to itself, and indeed it has an article to itself, reached from link in heading for this section.
Message from Steve Loft about documentation
The following was written by Steve Loft when he first made his MX beta available to Cumulus users. Although somewhat outdated it is preserved here.
There's quite a lot to read before you start - please do read all of this page and all the references it mentions, most of it is very important.
Note that most of the Cumulus 1 documentation also applies to Cumulus MX. MX specific documentation is currently in very early stages and some settings may not be obvious. Looking at the Frequently Asked Questions for Cumulus 1, Frequently asked questions for MX, and articles elsewhere in this wiki will help, as will looking at the Cumulus 1 help file, it is available on the Software Resources page. If you already use Cumulus 1, the help is part of the standard installation.
If MX is your first encounter with Cumulus, you will be at a disadvantage regarding documentation of many of the features, while those who have previously been familiar with Cumulus 1 will find most aspects of MX easier to pick up.
Message about this update to documentation
Although the message above from Steve Loft has been retained, it is no longer really true. When that was written on 2 January 2015, MX had been worked on for a year or so and had just opened up for beta testing.
Since then, of course MX has come out of beta and added a lot more functionality. More importantly, it has gained a large user base (although Cumulus 1 is still used by considerably more people than MX, there has been a recent surge of converts), and that means it is much better known, and consequently it is possible now to document it much better. The update made to this page draws on what has been said spread over lots of posts on the support forum and attempts to make it more accessible by repeating it on this page. Consequently, you don't now need to search in the way that Steve Loft's original text above implies.
In writing this update, I have drawn on my own experience of moving from Cumulus 1 to MX, and thus my knowledge of Cumulus is from over a decade of experience with this software and what it can do.
Before I swapped, I made a detailed study to check MX could do all I used to do with Cumulus 1 and much more. Before I add items to this article I play around with MX experimenting with what works and what does not work, but I have saved you the pain of where I went wrong, just telling you what is correct. I do need to add, that I don't have a separate testing environment, and therefore I am not willing to attempt anything that might muck up my collecting of weather information, plus my knowledge of modern technology is poor as I belong to that generation who did not have desktop computers, nor mobile devices, until some time into my working life. This all places restrictions on what I can test out, and therefore on the coverage of these notes.
If anyone else, can improve these notes, wants to split off more parts, or in any other way make the documentation better, then please do. This article already contains improvements that were suggested by others.
If this page, and those other Wiki pages it links to (e.g. Cumulus MX FAQ), do not answer all your questions then see the support forum for current Cumulus MX as that will let you see what other people have asked about, any posts I have not yet incorporated into this page, and there you get the opportunity to post your own query.
Installing and Running Cumulus MX
There is no automatic installer (this may change). Cumulus MX is supplied as a zipped package on a link from Software page.
Replacing Cumulus 1
See Moving from Cumulus 1 to MX article. If you wish to run MX on Windows, then you can unzip the contents of the download package over your cumulus 1 installation, i.e. so the same data and Reports folders continue to be used. But it would be best if you take a back-up copy of the Cumulus 1 installation first!
The package contains several extra .dll files, and everything else you need, to continue to read from your weather station, to load up the admin interface (there are some settings you will need to change), and some simple web templates (that replace the standard Cumulus 1 ones). You might want to read topics on the MX support forum to discover about other people's experiences.
Completely new MX installation
See earlier for links to other articles about installing on a Windows PC or a Raspberry Pi. Here only a brief indication of installation is covered.
- Create a new directory (recommended name CumulusMX) and unzip the contents of the download package into it.
- See notes below for extras required in various operating systems.
- The package contains everything else you need to read from your weather station (if it is a supported model), to load up the admin interface (for settings and some simple templates used to create web pages to see on a device connected to your home network). You might want to read topics on the MX support forum to discover about other people's experiences.
Running Cumulus MX
- Make sure your weather station (and any extra sensors) is connected to the device on which you have installed Cumulus MX, before you try to run Cumulus MX.
- Start Cumulus MX engine (command to do this varies between operating systems, so see sub-heading for your device below
- Start Admin Interface, it runs in a browser, by default on port 8998, see section below.
If you have been running Cumulus 1 before, then as instructed here your MX installation will require various files from your Cumulus 1 installation including all files in the data and Reports folder and all Configuration Files including Cumulus.ini and follow that link for details of a few of the parameters that you may need to change.
If you are running MX for the first time, without a configuration file (none is included in download package), see here for screen shots showing what you see as the engine starts running, and what you see in the admin interface where you set your weather station type. In that link there are more instructions.
.NET and Mono
The software currently called .NET was originally for all operating systems, but Microsoft then decided to restrict it to just Windows (and that was the position while MX was being developed and is still true in early 2020 when this article was rewritten), mostly to encourage greater dominance by Microsoft software and hardware.
Mono was then born based on .NET to work with all operating systems, Mono subsequently changed independently from .NET (although Microsoft still has a leading role).
More recently, Microsoft launched an alternative called .NET Core that took out of .NET the parts that were Windows specific, and it ceased work on further development of .NET beyond version 4.x.x.
Perhaps confusingly, in November 2020, there will be change around of names, and the multi-operating system .NET Core product will take over the .NET name as version 5. I don't pretend to understand the technical details, but the impression I get is that the new .NET in November will be similar to Mono, so apps designed for that will still work, but apps using .NET to make code designed for windows will stop working. Since the Cumulus code is currently coded to behave slightly differently using .NET and using MONO, I guess it is possible old versions of MX might stop working when the new .NET is installed via Windows Update.
Requirements for running on Windows
To run MX on Windows, you need .Net version of at least 4.5.2 installed. This is only available for Vista SP2, Windows 7 SP1, Windows 8, Windows 8.1.
For Windows 10 you need version 4.8 or later, this should already be installed by your windows update feature. The .Net download for version 4.8 should be here https://dotnet.microsoft.com/download/dotnet-framework/net48.
For those who downloaded the first MX Beta in January 2015, the code was only experimental and that version had to be run by a Windows Administrative User, but Steve Loft soon improved the code and now none of the code requires any elevated rights and it can be run by a normal user (or a user with administrative rights) without needing to be started by Run as administrator.
However, Cumulus MX initiates a web server, which is what runs the Admin Interface. To access that, all users need to be given elevated rights to the port on which the web server runs. By default this is port 8998, so that is used in the example below of the one-off command needed to give all users access to the port. You can use a -port=nnnn parameter when starting MX to make it use another port, if you use that then the command below needs revising accordingly.
To enter the command, first open a command window as administrator. One way to do this is to right click the windows symbol at the start of the windows task bar. The option to choose there is dependent on some settings which determine what appears when you right click:
- the normal default on Windows 10 is Windows PowerShell (admin),
- the normal default on earlier versions of Windows is Command Prompt (Administrator)
- an alternative is Windows Terminal
Whichever of these you can use, the result is it opens a new window on your monitor with a prompt for typing. In that window type the command:
netsh http add urlacl url=http://*:8998/ user=\users
You only need to do that once. If you do not issue this command, administrative rights are needed every time to access the port.
Talking about command windows, if you want to check that the port is open for listening (i.e. able to access the admin interface) type netstat -an | findstr 8998 into the command window.
The admin interface URL http://*:8998/ needs to have that wildcard "*" replaced by a precise location if we are to access the admin interface. The missing part of the URL depends on how your local network is set up. If you are accessing the admin interface on the same device as that running MX (and you don't have another web server on that device) the "*" can be replaced by "localhost", i.e. http://localhost:8998/ will be used to load the admin interface into your browser. In the more general case when you want to access the admin interface from anywhere on your local wired and wireless interface, then the "*" needs to be replaced by a string of 4 numbers representing what is called a IPv4 address (w.x.y.z) of the device you have installed MX on.
Look at your hub or router (this should have come with instructions on how to access its settings in your browser) and on one screen it should show what devices are connected to your LAN and wifi. Look for the IPv4 address, for example 192.168.1.64, it has assigned to the device where MX is running. That is what should replace the "*". However, there is one more complication, either the Windows networking settings may change, or else your hub or router may reconfigure, both can happen at any time and both can assign a different IPv4 address to the device running MX.
To give your Computer a fixed address for the MX admin interface,
- first find the network card via Network and Sharing Centre (Control Panel),
- then click on Change Adapter Settings,
- then Right click on Ethernet or WiFi Adapter,
- next select Properties
- In the window that opens, right click on Internet Protocol Version 4 (TCP/IP 4),
- Next select properties
- Onn that pop up screen tell the computer to "use the following IP address"
- Fill out the form with
- a subnet mask of 255.255.255.0 and
- gateway address between 192.168.1.1 and 192.168.1.254 (depending on the address of your hub/router).
Setting up for either manual or automatic running
To run Cumulus MX, Windows needs to know
- which .exe you want to run
- the path where all the required .dll files are located
Therefore it is best to always start MX using what Windows calls a shortcut, because when creating the shortcut you can enter all the required information into the properties. If you want MX to automatically start whenever you log into your PC, then the place to store your shortcut is C:\Users\...\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\Run_CumulusMX. Don't forget to put your Microsoft username where I have put ...
With this you might want to right click on that shortcut, select properties, then you can set the starting position for the command window, the colours and font it will use, and even choose to start minimised, amongst many other selections.
There are 3 ways on Windows to create a shortcut to run MX, as mentioned above you can't just click on the executable in file manager, because Windows needs to be told the path for loading all the related .dll files.
- Create a shortcut on your desktop (and/or the taskbar) for the CumulusMX.exe executable cmd.exe /C start CumulusMX C:\CumulusMX\CumulusMX.exe -debug, the "-debug" is optional, it starts the logging in debugging mode so the log created in MXDiags folder has more information. There are other optional parameters all listed later.
- In that shortcut define the path where the executable is located as the path to start in.
- Remember, if you have not done the netsh http add urlacl url=http://*:8998/ user=\users command mentioned above, you must run as administrator.
- Add any of the parameters that can be used with the executable, as listed later, such as specifying a different port for the admin interface , or starting with debugging.
- Choose whether to start as minimised or as a command window
- You can also choose text colour (foreground) and background colour, you can choose where the window appears (so if you have two monitors you can choose which one it appears on), and how big that window is. There is a forum post by water01 about this.
- Now you can click the shortcut to start the engine
- OR place the shortcut as defined above in the start up folder for the user account so MX automatically starts when you connect/log in (see a later section for how to find that start up folder).
- OR declare a task in the scheduler to start MX; in the Actions tab fill in fields as follows (the other tabs should be obvious):
- Action Start a program from drop-down
- Program/script cmd.exe (this is standard Windows environment to run something)
- Add arguments /C start Start_MX \CumulusMX\CumulusMX.exe -debug -port=nnnn (the "/C" means this task will close once it has started the task, the "Start_MX" is how the task will be labelled as it is running, the next argument "\CumulusMX\CumulusMX.exe" actually starts the executable and it does not need a drive prefix as that is in next box.
- Note in this example I have included next two optional parameters that can be used after the .exe call in that same box, here -debug (only include if you want full debugging logging) and -port=nnnn where nnnn is the port to be used for admin interface (only include if want to change from default 8998),
- all optional parameters are listed later
- Start in \CumulusMX (include a drive specifier if necessary)
Each time you want to run Cumulus MX on Windows:
- First start the engine in one of the 3 ways from last sub-section
- Next start the admin interface, it does not need to run all the time, but only when you need it (when you first use MX you will need it to access the settings where you tell MX what type of station you have and what units you want to use, and set various timing options), it normally runs on port 8998 (to vary that there is a -port parameter that is followed by required port and that port parameter has to be entered every time you start MX if you are not using the default port). More information on admin interface in separate article.
Try start /min C:\Cumulus\CumulusMX.exe to run MX as a minimised package (although in Windows you can change the properties of the shortcut you use to start minimised).
Stopping Cumulus MX on Windows pc
The recommended way is to click into the command window in which MX is running, hold down Control key and press C. It is normal for there to be a short wait, then a message "Cumulus Terminating" and then after another short wait, it will say "Cumulus Stopped" and immediately after that the command window will close.
Some people, click in the task bar and select close, or click the X button on top right of command window. Although these are not official advice, they do seem to work.
There are packages that can be programmed to send a control C to a running task, and to not continue until the task window has closed. Remember to also program in a subsequent delay in that package, to make sure the package waits for MX to close, or do a check that MX has released all the files it might need to update.
You should not issue a TASKKILL instruction, as that will prevent MX correctly writing out to all the files it should update on exit. Consequently, it will not restart correctly and may actually lose settings and data.
Requirements for running on Linux and OS X
You will need to install the Mono-complete runtime (the latest version of Mono should work with all functionality of latest MX in all locales). Mark Crossley says "There shouldn't be any outstanding issues with Mono, afaik they are all resolved - except for the Moon image rotation in the southern hemisphere which does not work with Mono 6.0 thru to the latest 6.8.0, only version 5.x works correctly atm for System Drawing."
- For OS X, you can download this here - http://www.mono-project.com/download/.
- How you install on Linux depends on the flavour of Linux you are running. There are download links for Linux at the same URL, but it is often easier to use a package manager, which will download and install it automatically.
- For example, in 'Raspbian' on the Raspberry Pi, you can install mono with the following commands, but first you need to have set up various pre-requisites (see Setting_up_Raspberry_Pi article for details):
sudo apt update && sudo apt upgrade sudo apt install mono-complete
Note that you do need to have the mono-complete package installed, not just the Mono for developers.
The "sudo" prefix gives the command 'root' privileges, that allows administrative commands like update and install to run.
To actually run MX
Open a terminal window, change to the Cumulus MX directory, and then type:
sudo mono CumulusMX.exe
There are some optional parameters you might need to use, as they also apply to windows they are covered later.
Next start the administrative interface, basically same as described for Windows above. More information on admin interface in separate article.
Other issues
There are lots of topics in the MX sub-forum about a multitude of issues about commands to use to install and check mono, how to stop MX and differences between different devices (including Mac) and different Linux versions. At the moment, there seems to be some uncertainty, and consequently, I have not attempted to include/summarise all the material I have found.
APPEAL - Please could any readers who have experience of running MX in a Linux or Mac environment please consider writing advice into this article. I want it to be a comprehensive accurate article.
Notes by Steve Loft
please note these notes ARE now obsolete, library routines have changed a lot since this was written in 2014
Any volunteers to replace this section with up to date information?
- If you have a Raspberry Pi 2, there is a later version of Mono available, which you may find works better that the one in the standard distribution, particularly if you use decimal commas. Mono 3.2.8 (which is the default in some Linux distributions) will not work if you use commas for decimals, as in some countries.
- On Linux you will need library libudev.so.0 which may not be installed by default. Installing package libudev0 may resolve this. There may be issues if you are using a 64-bit version of Linux. I'm not sure what the resolution is at the moment, if this is the case.
You need to specify something like /dev/ttyUSB0 for the connection for your weather station. This is set in the "station settings" and stored in the ComportName attribute in Cumulus.ini configuration file.
In some builds of MX you have to run as "root", there are ways of giving "root" like permissions when running MX as another user, see forum for details until this section has been updated.
Stopping MX
The best way to stop MX is by sending a control and C sequence to it. There is a start stop routine discussed in the support forum, where you will also find some topics about shutdown issues.
The MX source listing suggests it also accepts control and break, Close main window, user logoff, and system shutdown and should still attempt to stop running tidily. There are various tasks like writing final values to log files and recreating the configuration file, Cumulus.ini, that it needs to do as part of the MX closing routine, and obviously early closure of the device running MX (such as that caused by a power cut) will prevent a tidy end to MX.
Executables
The MX package, at time of typing this, includes two executables:
CumulusMX.exe
Whilst effectively MX is run by a CumulusMX.exe or sudo mono CumulusMX.exe depending on device, you actually need to ensure all the other components are loaded, so you either have a package that runs it for you, or you click a shortcut that includes the necessary path setting.
Optional parameters to add to the instruction to run the MX engine
Beta builds in MX version 3.0.0 had an optional parameter -wsport nnnn that determined which port (represented by a 4 digit number nnnn) was used for WebSockets. That parameter is now deprecated as WebSockets in all builds since 3045 uses the same port as the rest of the Admin Interface. The remaining parameters that are still available are described in subsequent sub-sections.
Parameter for changing Port
When Cumulus starts, it will display the URL of the user interface. It runs on port 8998 by default; if this is not suitable for some reason you can over-ride it using the '-port' parameter on the command line, e.g. to use port 9999 instead:
sudo mono CumulusMX.exe -port 9999
Parameter for adding debugging
MX has a default level of logging that stores in the MXDiags folder a log file that shows some of the interaction with the weather station and some of the output actions done as MX runs. A new log is started each time MX is restarted.
If there is a problem, then there is a great benefit in actually increasing the level of detail in these logs; and that is done either within the settings (options section of station settings) in admin interface while MX is running, or by adding 1 or 2 parameters when you start MX. Obviously this log file continues to grow, the longer MX is left running, and if debugging is switched on the file will grow in size must faster. Consequently, the default is not to add the extra debugging information and the settings can be used to switch it off again if you do have it switched on. Whether you start it with a parameter or enable it within settings, stopping MX will end the extra debugging, and on restart it will default back to no debugging unless turned on again with parameter or setting.
You can also add CumulusMX.exe -debug (to have full debugging of actions by MX turned on as MX starts), and/or CumulusMX.exe -Logging=1 (for the station to MX transfers to have increased debugging logging).
sudo mono CumulusMX.exe -debug -Logging=1
Since this parameter is applied when you start MX, it applies while MX continues to run. Obviously, it must be applied every time you start MX if you want this increased level of logging to continue every time you restart MX.
The comments in the MX source suggests -debug turns on both debug and data logging (see Station_Settings#Options in admin interface settings), but I believe that is wrong as per example above, there are 2 separate parameters.
Parameter for changing Locale
On Linux and (in particular) OS X, Cumulus MX may not be given the correct locale to use, and you may get the default US locale even if that is not your locale. It will output the local it is using when it starts; if it is not correct, close it down and start it again, this time specifying your locale on the command line, using the -lang parameter . For example, in the UK, on a non-Windows device type:
sudo mono CumulusMX.exe -lang en-GB
Other local examples: CumulusMX.exe Current culture: English (United States), CumulusMX.exe -lang de-DE, CumulusMX.exe -lang el-GR (this is one of the locales that reads numbers with integer,decimal format), CumulusMX.exe -lang nl-NL.
If you are not sure what value you need to supply for the -lang parameter, there is a list here - http://msdn.microsoft.com/en-gb/library/ee825488%28v=cs.20%29.aspx. You need to supply the code in the first column ("Language Culture Name") in that list.
Note that this does not affect the language used by Cumulus MX (although it may in the future), it affects the decimal separator and the list separator.
Note that you may need to supply your administrator password after typing the 'sudo ...' command line. The system will prompt you for this if it is needed.
ExportMySQL.exe
This second exe file has been available since the original MX package as Steve Loft developed this in April 2015, but sadly few people even notice it exists, and if they do, it is unlikely they know how to use it. Hopefully, some people will read this section and find out!
Obviously it was updated when Mark Crossley added the Feels Like fields to log files.
Put simply, this executable will read log files and insert (insert ignore) rows into an existing database table. Since it only does inserts, despite the name of this function, it is not just for MySQL tables, the included SQL should work with whatever database table type you have.
The executable has a mandatory single parameter that tells it which log files to read, there are only 3 possible parameters ("dayfile", "monthly", or path to a file). It needs to know what locale (or culture settings) it is to use to work out what character separates each item in the log file list. It also needs to read your Cumulus.ini file, as it takes these "input parameters" from MySQL section in that:
- Host
- Port
- User
- Pass
- Database
- MonthlyTable
- DayfileTable
Daily summary log file
- Use the feature in the admin interface:
- Settings menu
- MySQL settings page
- In Dayfile.txt upload section, give your database table a name, or accept default Dayfile.
- Click Save to ensure this setting is updated
- Now scroll down to Create database table (save settings first)
- Here click Create Dayfile
- Now you have a database table ready, you can use the executable to read all lines in your CumulusMX/data/dayfile.txt log file.
- Open a terminal display (if you are using Windows then, open a Command Window, a Windows Powershell window, or a Windows Terminal window)
- Run this executable in that terminal display (or command window) by using sudo mono ExportMySql.exe daily or ExportMySql.exe daily depending on device.
- In the terminal display (or command window) you will see Parameter = daily confirming what you entered and in the line below that a rapidly updating code that is the primary key displayed for each row it tries to insert into the table. If that primary key already exists in the table, it will still show the key, but no insert will take place.
- If you want MX to continue adding new rows to this database table, still in the admin interface, still in MySQL settings page:
- Return to Dayfile.txt upload section, and select Enable.
Standard Log files
- Use the feature in the admin interface:
- Settings menu
- MySQL settings page
- In Monthly log file upload section, give your database table a name, or accept default Monthly.
- Click Save to ensure this setting is updated
- Now scroll down to Create database table (save settings first)
- Here click Create Monthly
- If you want MX to continue adding new rows to this database table, still in the admin interface, still in MySQL settings page:
- In the same Monthly log file upload section, now select Enable.
- Now you have a database table ready, you can use the executable to read all lines in either one (if path to that file is in parameter), or every (if parameter is monthly) standard log file.
- If the parameter is "monthly" it will look in folder data for every file it can find with a file name of datestring + "log.txt" where datestring is a 3 letter code (in your locale) for each month (1 to 12) followed by a 2 digit year (from "00" to "99") so that is how it finds every standard log file in the folder.
- Open a terminal display (if you are using Windows then, open a Command Window, a Windows Powershell window, or a Windows Terminal window)
- Run this executable in that terminal display (or command window) by using sudo mono ExportMySql.exe monthly or ExportMySql.exe monthly depending on device.
- Alternatively, replace monthly parameter by a full path to a single standard log file, and it will process just that log file.
- In the terminal display (or command window) you will see Parameter = monthly confirming what you entered and in the line below that a rapidly updating code that is the primary key (omitting the first two digits of the year) displayed for each row it tries to insert into the table. If that primary key already exists in the table, it will still show the key, but no insert will take place. So you can run this again to pick up any additions to the latest log file since the original run. Also notice that if you use the parameter "monthly" the order in which it will process different standard log files is not predicable, they probably will not be in any particular order, but as one feature of SQL databases is that the row order is not able to be determined, it does not matter if rows are not added in chronological order.
- It is worth noting that it is safe to run this procedure while MX is also running, because this procedure only updates log entries that exist as this procedure reads the logs, and MX only adds new entries to the log and at the same time uploads that new entry (if enabled) to the database table.
Please be aware that the transfer to the database table adds two columns where bearings in the original log file given in degrees are output as compass directions, and these use up to 3 letters of how the compass directions are defined in the strings.ini file. Thus the number of columns in the database table will be at least 2 more than the number of fields in the log files. It is also important to stress that whilst the database table must contain one column defined for each field (plus the extra 2) being uploaded, you can add even more columns to your table if you want and populate those some other way. For example, I have added a Canadian Humidity Index (Humidex) column which is not in the standard logs, but is calculated by Cumulus, and can be calculated from columns that are uploaded from the standard log. Humidex is not uploaded by either ExportMySQL or the normal CumulusMX process, but neither objects to extra columns being there.
When testing this, I had some log files produced by various old versions of Cumulus 1 in my MX data folder as well as the log files in has generated since I swapped to MX. Plus I had used a PHP script to add feels like to those log files produced before version 3.6.0 and to correct feels like for those log entries made by versions 3.6.0 to 3.6.9 inclusive because they used a different formula to the one being used from version 3.6.10. This php script is a web page with a HTML form and can be obtained from the forum in Create Missing for MX
I notice that the database rows produced by those short log file lines produced by say version 1.9.0 had nulls entered for all subsequent columns, except Feels Like and this column was initialised at 0.0!
For those log files produced by the final version 1.9.4, all columns are populated although feels like is set to 0.0.
Library software
For most Cumulus users, this whole section can be skipped, but I have included it for those few users who have a technical slant and might want to understand more.
Cumulus MX uses library software (i.e. software written by others and made available by the provider and often also by other content delivery nodes or 'cdn') for a lot of the standard functionality. The library software for the admin interface and the separate library software for the standard web pages are both mostly included in the distribution zip, although some is used via a link to a cdn.
Many of the libraries included by MX are very obsolete. However, Mark Crossley, the current developer, said the following on 30 Sept 2019: "Chasing the latest versions of all the packages for the sake of it is a thankless task, and requires considerable effort to regression test each update. I am only updating packages when required to fix issues or for platform compatibility."
Just to mention the other side of this balance. It is difficult to code an addition to MX that works with these obsolete versions of libraries. All documentation provided by providers of the libraries relates to current versions of the packages (and what is documented to work now often will not work with obsolete versions). The documentation for the packages that are no longer supported is only available in archive sites if available at all.
Library Software for the MX engine
The distribution zip contains various .dll files and these are the libraries used by MX itself. The exact mix of libraries included has varied at various times, the list below is a snapshot of those included at the version that was investigated when this article was extended to include this section, and may not be right for the current MX version.
Devart
The two files used are both related to the database functionality of MX.
FluentFTP
As the name suggests, this is used by MX for controlling the file transfer functionality. This component was first introduced at version 3.0.0 build 3045 (adding more functionality than that available from System.Net.FtpClien, which was used at earlier MX builds), and FluentFTP.dll has been updated at some subsequent MX releases, see the announcements for details.
Linq
Language INtegrated Query is used to work with sequences of items and pick the ones that are needed, putting them into output format required. MX uses two files in connection with preparing output for Twitter. There is a third Linq file for other processing.
MQTT
When MX added capability to talk to other devices using the MQTT protocol, it added this component for that optional functionality.
Newtonsoft
This component is used for processing JSON strings. It is a very popular choice for developers, and used therefore very widely. However, SystemText has superseded it, so MX is using an obsolete method.
Renci SSH
This component is server connection software, it is what processes the host name, password, and so on.
SQLite3
This is used for all interactions with the weather diary.
HTTP
These files handle the optional HTTP functionality.
Unosquare
The EmbedIO file is open-source software that handles the web-sockets functionality of MX. The Swan file is open-source software that handles JSON formatting and threading of tasks in MX.
Library software for admin interface
As the following sections reveal, MX uses external libraries rather than writing its own code whenever possible.
- However, that does not mean MX is good at meeting development standards.
- MX only implements small parts of the functionality of most libraries, the minimum to make a feature work, not all the features available to make it work well
- MX does not use the latest versions of libraries
- MX does not attempt to obey guidance for good user interaction, and although validation is being added, many parts of MX do depend on user understanding what is valid
- MX does not make provision for screen readers and other accessibility aids.
Alpaca
- Alpaca software is effectively a programming language extension to help people design forms like those MX uses for all its settings, and as a Cumulus user you really don't need to worry about it.
- It is used for most settings screens. See http://www.alpacajs.org/ for more information. The latest version there is 1.5.27 released on 14 May 2019.
- MX uses Alpaca Release 1.1.3 from https://github.com/gitana/alpaca which was first released on 15 May 2014. Although some individual components have been updated on that github URL, and elsewhere, MX has not incorporated these.
Bootstrap
- Also known by some as Twitter Bootstrap which gives a clue as to its developer and to its origins as an internal tool for those building Twitter, that company still keep making updates as it is now the most popular styling library of all those available widely.
- The simplest way to think about this package is as a standard set of styling promoting easy responsive (means adapts to screen dimensions) web site design.
- To give just a few examples, it defines a standard way to represent buttons, form components, lists, navigation, and breadcrumbs.
- MX uses Bootstrap version 3.3.7, which is very restricted in what it offers
- Bootstrap version 5 is available (http://getbootstrap.com), so MX is using an obsolete library
- Bootstrap version 4.5.0 (Bootstrap 4 released as alpha in August 2015, beta in August 2017, and with fully working releases frequently from January 2018) was very widely praised for its improved functionality, and ability to work with latest jQuery and multiple modern devices/browsers.
- MX does not implement key features of Bootstrap like colouring text according to what it represents (primary, secondary, information, warning etc.)
dataTables
- When MX sends out multiple lines of a log file to view or edit, the application programming interface (api) that transfers the information from the MX engine sends it in dataTables format for display on the web page in the admin interface.
- Thus dataTables does all the work of providing the ability to present the data in a HTML table, the functionality to move between multiple pages needed (as MX sends only up to 10 lines of a log file at a time to the admin interface).
- The free version of dataTables used by MX lacks the most useful functionality that needs a subscription licence. For example, its editing functionality requires a subscription.
altEditor
- This is an editing tool that can read what is in dataTables, create what it calls a modal (a pop-up dialog) where rows can be added, edited, or deleted individually.
- MX when it added editing of log files at version 3.4.5 - Build 3069 (Friday 13 March 2020) adopted this software as it was free (although Mark Crossley said in his release notice: "The main thrust of this release is to add some log file editing capability to Cumulus MX. It works on all three log file types, but it is fairly basic at present. You can edit or delete lines in the files. The editing has to be done via pop-up dialog.
- I only found two libraries that support JQuery dataTables editing, one is very comprehensive - but costs $$$ - the other is free. The free version does not currently support in-line editing of the table which is a shame.
- If any web guru out there can come up with a better solution please post about it on the forum, or send a pull request.") (By the way, it is possible to provide in-line editing and make it work with the existing api interface, but making it compatible with the obsolete software used by MX is hard).
- The single line of fields that is result of an edit or deletion done on the modal is sent back via another api to the server (the MX engine in our case) and that then regenerates the dataTables in the state after whatever action was done, sending back again up to 10 lines for the same page as before.
- As it happens there is another JQuery dataTables editing tool, but it has not been maintained since 2012. It is found at https://github.com/NicolasCARPi/jquery_jeditable, but the documentation is now only available in an archive at https://web.archive.org/web/20200615000000*/https://appelsiini.net/projects/jeditable. It is designed for editing table cells, so it does not involve any pop-up dialog.
datepicker
- Although modern browsers generally will generate a calendar type interface when they meet an entry field defined as a date, this date picker software ensures all MX users see exactly the same interface for date selection needed for both the standard log and the extra sensors log which are monthly log files (a new one is created each month). It is used for picking which standard (monthly) log or extra (monthly) log is to be viewed by selecting a month and year only.
- It is also used for selecting individual days in the weather diary editor.
editable grid
- As the name perhaps suggests MX only uses this for the extra web files screens where you can make selections within a grid like interface.
- I suspect it could enhance some other functionality in the future.
handlebars
- Put simply this is a simple HTML generator based on templates.
- I have not found any file in the admin interface actually using this, but I am scared to delete it just in case it stops something working.
x-editable
- Put simply, this allows in-place editing of web pages using bootstrap.
- In MX it is used for the record editing screens where you adjust the extreme values.
jQuery
- The admin interface uses version 1.9.1 of this JavaScript based library.
- At the time of typing this, the current jQuery is version 3.5.1.
- Of all the old versions of jQuery to choose, MX has picked the only version that the developers withdrew due to an error when they released it.
- Version 1.9.1, has a serious error in its code, because the developers accidentally combined code from two significantly different versions when they created the release file.
- This reveals itself in two ways:
- The error handling does not work.
- It tries to load another script that does not match.
- This reveals itself in two ways:
- Consequently, the developers quickly removed it, but it remained available from Contents Delivery Nodes, which is where MX has found it.
- Not surprisingly, the authors of jQuery strongly advise all 1.9.1 users to move to a later version.
- Unfortunately, there are interdependencies between all the library code used by MX, so you cannot simply update this component (see next library item).
Jquery Template
- This is also obsolete, and therefore will only work with old versions of jQuery (1.4.2 to 1.11.0).
- It was originally available from jQuery downloads, but they now offer jsRender.js for this functionality.
- It basically is used to bind the contents of objects (like array elements) into particular locations within HTML.
SteelSeries
MX uses a modified version of the steel series library made available by Mark Crossley for all the gauges (see dashboard and gauges pages of the admin interface) in MX.
Highstocks
The odd one out is Highstocks (that includes HighCharts)
- This is loaded from a Contents Distribution Node (CDN), but it is still pinned to obsolete versions of the basic script and its themes.
- This means that the Charts page in the admin interface will only work when there is an internet connection working to permit download of this software
- If you need to view your admin interface where an internet connection is not available:
Then you need to edit the interface file...
<CMX_Folder>/Interface/charts.html Change lines 20,21 from
<script src="https://code.highcharts.com/stock/8.0/h ... "></script> <script src="https://code.highcharts.com/themes/grid.js"></script>
to
<script src="webfiles/lib/highstock/js/highstock.js"></script> <script src="webfiles/lib/highstock/js/themes/grid.js"></script>
Library software for your web server
The webfiles/lib folder includes a number of software library items that are needed for the standard web pages included in the MX distribution.
This is fine if your web site is purely for the provided standard web pages, but if on your web server you also have web pages from third parties, or you have written your own web pages, then you may get conflict because all the library items used by MX are obsolete versions, and in one case MX uses a version of library software withdrawn by its originators due to compile error (so only available form some CDN who provide obsolete versions as its originators insist it must not be used)!
- Highstock
- At the moment, as hinted in previous section, there is an old version of Highstocks included in the webfiles/lib folder.
- However, that is not used, instead (like admin interface) an old version is loaded from a CDN.
- Be careful if you also load the current version for use on web pages not produced by MX, that the browser does not try to reuse your latest version and not recognise that MX wants an older version.
- jQuery
- Be aware that MX distribution is not consistent as different web pages use different obsolete versions of jQuery.
- gauges.htm includes <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script> to load version 1.8.2 from the Google Content Delivery Node
- You will find at the end of this web template, if you bother to read it, some instructions about how you can change that CDN load, but the instruction is neither grammatically correct, nor understandable for me.
- trends.htm includes <script src="lib/jquery/jquery-latest.min.js"></script> to load version 1.9.1 of jQuery loaded from this badly named file, there is nothing latest about it.
- Ironically, the old version of jQuery that has been selected by MX to be included in the webfiles/lib folder is the only old version that jQuery themselves withdrew, and warn you not to use because the error handling content is from a different jQuery version to the correct handling content making the two parts of the code incompatible. In other words, this version only works if everything is set up correctly and no error handling is invoked.
- gauges.htm includes <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script> to load version 1.8.2 from the Google Content Delivery Node
- Be aware that MX distribution is not consistent as different web pages use different obsolete versions of jQuery.
- Steel Series this is treated as library software, as it is copies of that separate product by Mark Crossley.
- Most of the files are exactly as that was in revision 0.14.13 made 30 January 2015.
- gauges.js has one tweak for MX, it defaults to not having chart rollovers (for Cumulus 1, this file had default to showing the images of graphs that were uploaded by Cumulus 1).
The provided web pages
Setting up a web site is covered in this wiki page and the pages linked from there. I won't repeat that, but will try to explain below the MX context of the various files involved. MX will produce web pages locally even if you don't have a remote web site to display them on. You can view the web pages created in the web folder using a browser.
Cumulus MX provides a set of web templates, images, and json files, in \CumulusMX\web. The first of these are called templates (and have a 'T' at the end of the file name before the extension) because they include both text and web tags. MX will process these templates as it creates a web page (file name without a "T" in same web folder), during that processing, any text in the template is copied into new file without change, any web tags found in the template are parsed and the correct value is placed in that position in the output file. If you have a web server, then MX can process these files and upload them for you (by default using File Transfer Process) providing you specify the host, port, protocol, directory, username, and password, for the upload process to use. If you don't understand any of these terms, then this is not the place for explaining them, but generally if your web space is supplied by a provider, they will be able to tell you most of these settings, and you will choose the directory name. If you have set up a web server yourself, then you should know the required settings.
The web templates included are based on designs by Beth Loft used for Cumulus 1. Remember, Steve Loft (who wrote the original Cumulus software) said "They exist because they're our web pages, and they're really only included with Cumulus as examples of how the web tags work. It never occurred to me that most people would simply use the supplied examples instead of creating their own pages!"
The templates are written in fairly simple Hyper-Text Mark-up Language, designed to help people see how to write their own web templates. Here is a list of web templates provided:
- indexT .htm
- todayT.htm
- yesterdayT.htm
- thismonth.htm
- thisyear.htm
- recordT.htm
- monthlyrecordt.htm
- gaugesT.htm
- trendsT.htm
All these templates (except gaugesT.htm and trendsT.htm) include a table for showing values and styling that gives a graded background colour. The tables include a navigation row with links to the other pages in the set. That navigation line fixes the width of the table, and you will realise it was designed in the days when all monitors were a standard shape. Therefore the standard web pages as provided cannot adapt to the range of devices we use for viewing web pages nowadays. There are a selection of alternative web page sets available on the User_Contributions page, and some of these are responsive and adapt to the width of the device they are being viewed on.
The gaugesT.htm is a template similar to SteelSeries Gauges although the latter is designed to work with a range of software and the former is specific to MX. As supplied in MX if you mouse over the provided gauges appearing on your web site you will see a box with figures, not a graph as is seen with the general steel series gauges, but there are some other differences such as how the figures are supplied for the displays. The remaining template trendsT.htm, creates a structure that can display graphs. The data for all the graphs that can be displayed is contained in the various json files in \CumulusMX\web, these files are also processed by Cumulus so latest values are added, and then uploaded so the web page produced by this template can use them.
The image that is provided in \CumulusMX\web is MoonBaseImage.png, MX can be set to use that to generate (on MX start-up and on the hour) "moon.png" which it then can FTP to your web (also on the hour).
To set up your web server for the first time
When you first want to use Cumulus web pages on your web server, you need a number of static (unchanging) files to be put onto your web server. The web pages that MX uploads for you reference that static files and will not look right without them. The files that only have to be uploaded once are found in \CumulusMX\webfiles and its sub-folders. You don't create a folder called webfiles on your web server, but you put the files and sub-folders into position relative to where MX will upload the htm files.
To do this, you will must invoke a FTP process outside of Cumulus, MX does not include any functionality to do this one-off upload for you. The filezilla client is a popular choice as it has probably the most friendly graphical user interface, although other software to do this is also available. You may prefer a tool that lets you do the uploads from a command line without requiring working with a graphical interface.
- The static files to be uploaded include the standard styling file \CumulusMX\webfiles\weatherstyle.css which you place in the directory specified for the uploads.
- Next you have three sub-folders, each of those sub-folders need to be replicated within the directory specified for the uploads.
- For example \CumulusMX\webfiles\images\picture.jpg will be stored in a "images" sub-directory of the upload directory and is used as the background image for web pages.
- There is nothing to stop you creating your own "picture.jpg" (instead of uploading the supplied one) and then Cumulus web pages will use that for the background image on each page.
- Similarly \CumulusMX\webfiles\js\cumuluscharts.js needs to be stored in a "js" sub-directory of your upload directory (this is the script that allows you to change the chart shown on the trends page and uses the appropriate json file to populate it with data).
- For example \CumulusMX\webfiles\images\picture.jpg will be stored in a "images" sub-directory of the upload directory and is used as the background image for web pages.
- The "lib" sub-folder contains further levels of sub-folders all to be replicated on your web site.
- The trends.htm web page also loads some library software from an internet Content Delivery Network (cdn) to invoke the JavaScript based Highstocks library.
Operating a web site with uploads from MX engine
The standard web pages
- If you want to operate the 'standard' web site, then just the same as with Cumulus 1, you will need to upload the contents of the webfiles folder from the zip file (don't upload the containing webfiles folder itself).
- Note that the MX web files are not the same as the ones for Cumulus 1, so make sure you upload the MX files if moving from Cumulus 1 to MX.
- The standard gauges are now the SteelSeries gauges. The default versions do not display a graph when you hover over a gauge as happened when you added the stand-alone Steel Series gauges to Cumulus 1.
- The trends web page in Cumulus 1 relied on that software generating graphs as images. In MX, the software generates files with time and value pairs, these are stored in json format, the trends page then uses a library package (Highstocks) to draw graphs from those data pairs.
Alternative ways to obtain web pages
You can choose to use some of the alternative web pages available from third parties and described on User Contributions page.
Using your own web pages
- Of course you can use your own web pages, instead of the standard ones. Assuming they need to include figures that are available as web tags, there are three alternative ways to implement this:
- MX can process template files with a HTML structure and those web tags in the structure where values are required just as it does with the standard templates, and MX can upload the resulting web pages at either the real-time interval, the standard interval, or after end of day. All of this is covered on the Customised templates page in this Wiki.
- MX can process a file with a string of web tags mirroring the realtime.txt option in MX, and upload the resulting file so your web pages can use JavaScript for a one-off insert of the values or an Ajax routine to update the web page at a fixed interval.
- Alternatively, you can use template scripts processed locally by MX that don't create web pages, but are uploaded by MX at either the real-time interval, the standard interval, or after end of day. These scripts simply initialise script variables with values obtained from web tags. You then independently have a set of web pages resident only on your web server (they don't exist where you run MX) using a combination of HTML and script content that bring in the script(s) with the variables by the appropriate syntax. All of this is covered on the PHP web tags page in this wiki. As it suggests there, you might therefore have several files processed by Cumulus MX at these different intervals, converting the web tags into script variables, and then use AJAX (JavaScript that may use json format to bring in the variables) or PHP (using 'require_once 'filename'; syntax) to put those variables into a web page.
You may find this wiki page useful for understanding more about the different script languages.
MX End of Day Process
I have added this section, because this process has given me some headaches.
My version
If you write custom SQL, or have a template being processed at end of day, then what I find strange is that web tags related to system date report the new date, but other web tags report weather derivatives from the old day. Put another way, all the date related tags change at start of rollover, but all the weather related web tags change at end of rollover.
However, it is not quite as simple as that, the month and year are reset before the extra web files are processed (so they cannot use monthly web tags at end of month, nor yearly web tags at end of year). The complication is that in the extra web files you can use <currentlogfile> (and from build 3087 <currentextralogfile>) and these pick up the old month/year. Now you see why I found it hard to digest, and why I wanted to write it here to make it easier for others.
If you use Custom SQL and therefore have to quote web tags, the SQL should use monthly and yearly web tags related to previous day, but all the weather tags it uses must be those for current conditions or today. Yes it is confusing.
As part of the so called "end of day" process, MX (just like Cumulus 1) creates a start of day back up in the daily sub-folder. So on first day of a new month (and new year sometimes), the backed standard log files (and extra sensor log files from build 3087) are for the month that has just started, there is no back-up of the old month.
The official version
Mark Crossley says the MX day reset does this (at version 3.5.x)...
Reset midnight rain Entering Day Reset (message about current day of month, at this stage web tag <#metdate> changes to new date) Day Reset (message about date ending, time shown as 00:00:00 because time not defined, not because it is midnight, it might be 9am or 10am) Run EOD custom SQL Save dayfile entry (uses what is still in today.ini that includes old date, i.e. what is now in web tag <#metdateyesterday>) Write monthly & yearly file entries Write any new daily extreme records if day of month = 1 then: copy month.ini to saved file, reset monthly figures if day of month = 1 and month = 1 then: copy year.ini to saved file, reset yearly figures Copy todays high/lows to yesterdays Reset todays high/lows to current Write today.ini & yesterday.ini Create NOAA reports Execute user daily external program Process Extra EOD files
But independent of above EOD thread that occurs on the rollover hour, the normal interval and hourly processes thread is seeking to run at same time, whether that happens at same time depends on processing capability and whether it can process multiple threads.
What actually happens in above list depends on your settings, and if your FTP interval is synchronised with the logging interval.
SPECIAL CONSIDERATIONS - Text by Steve Loft
Restrictions in MX for decimal separators
On the subject of decimal and list separators, there are a couple of issues which users of decimal commas may encounter.
- The first is that there may be an issue with some of the user interface not working correctly. Please report these issues and I will fix them. There may be aspects of the displays that I cannot change (because the package used does not support decimal commas) but it should be possible to at least get it working.
- The second issue with decimal separators only affects the Raspberry Pi (as far as I am aware). There is apparently an issue with a version (3.2.8) of the Mono package on Raspbian 'hard float' where it cannot parse values using decimal commas. If this does turn out to be an issue, there are a number of possible workarounds until the Raspbian package gets updated. One workaround is to use the 'soft float' version of Debian instead. Obviously, this will have performance issues, but is probably the easiest. The second workaround is to build Mono from the latest sources, see http://www.mono-project.com/docs/compiling-mono/linux/. I am told that this fixes the problem. Another possible workaround would be to find an already fixed binary package, but I don't know if one currently exists.
PLEASE NOTE: The issues that Steve describes seem to have gone away with currently available versions of Mono; update your Mono if you are using an old version and encounter problems. Like any software, Mono might have bugs at a particular version, and sometimes you might need to swap to an older version if the current version has an outstanding issue.
If you want to use your Cumulus 1 data with MX
If you use decimal commas in your Cumulus 1 data, you will need to edit the .ini files to change the decimal commas into periods/full stops, because Cumulus MX always expects periods/full stops in .ini files regardless of the locale in use. The other data files will be OK - assuming you are using the same decimal and list separators in MX as you used in Cumulus 1 (i.e. the same locale). If you try to switch to a different locale, then your data log files will of course no longer be in the correct format, so you would need to edit all of your files. You can select the locale for MX to use as a switch parameter when it starts up, see earlier on this page.
A note to Davis owners
I am experimenting with the use of the LOOP2 packet. The current code uses this for two purposes. First, it uses the 'peak 10-minute gust' value, to avoid the problem where a gust might be missed (although hopefully this will not be such an issue with Cumulus MX as it does not use the Davis DLL), and secondly it uses the 'absolute pressure' value to make calculation of 'altimeter pressure' easier and more accurate. This is mainly used if you upload to CWOP.
The LOOP2 packet is supported on the VP2 with firmware version 1.90 or later, and on the Vue. If you have a Vantage Pro (i.e. the original 'VP1'), or a VP2 with pre-1.90 firmware, or if you are using Virtual VP, none of these support the LOOP2 packet. In these cases, you should edit cumulus.ini and add a line to the [Station] section:
UseDavisLoop2=0
With this setting, Cumulus will revert to calculating the 10-minute gust value itself from the individual wind speed readings, but it will not currently attempt to calculate altimeter pressure correctly, it will simply use the sea-level pressure instead. This is likely to be an issue if you are at high altitude and you upload to CWOP using Cumulus MX.
Also for Davis stations, I have assumed that people using millimetres in Cumulus have a metric rain gauge (0.2 mm per tip), and those using inches have a 0.01" rain gauge. This can be over-ridden by adding a line to the [Station] section of Cumulus.ini:
VPrainGaugeType=0
or
VPrainGaugeType=1
Where 0 is a 0.2mm gauge and 1 is a 0.01" gauge. Note that changing this after MX has already read some data may cause your rainfall reading for today etc to change considerably, so you will need to correct that.
Almost all of the web tags for all Cumulus flavours on this Wiki page that you could use in Cumulus 1 are also supported in Cumulus MX.
Each new build of the beta MX has increased the range of web tags it supports. Since MX has come out of beta, new versions have not only implemented the remaining tags from Cumulus 1, they have also added new tags not previously available. For full details see the web tags article, but a quick précis follows in next few sections.
All builds of MX
The 'format' parameter on the date/time output modifier for web tags is unfortunately different, because many of the characters used are different. See the modifiers list page of this Wiki.
Note that this difference in date/time modifiers also affects how you specify the NOAA report file names. For example in Cumulus 1 you can specify a 2 digit month number by either 'mm' or 'MM', but MX (later versions) has to change the former to the latter as MX uses 'mm' for minutes. The same applies to using 'mmm' or 'MMM' for 3 letter month abbreviation in Cumulus 1, only the latter works in MX, so MX (later versions) will adjust that. If you are using an older MX version, you should upgrade to latest as you are missing a lot of functionality, but while you use that old version, ensure that your file names for NOAA reports do use the correct modifiers for MX.
Beta builds of MX
The following web tags were not available or worked differently:
- The individual 'record set' tags such as <#TempRecordSet> etc did not work (because the interface then had no indicators for new records and no way to reset them).
- The <#newrecord> tag does work, but works differently, it turns itself off automatically after 24 hours.
- Some of the 'system status' web tags do not work: <#CpuName>, <#MemoryStatus>, <#DisplayMode>, <#DiskSize> and <#DiskFree>
- The <#txbattery> web tag has no content currently. Using it with a 'channel' parameter causes a 'token error'.
- The snow tags were not available as there was no Weather Diary
Current builds of MX
The web tags you have depend on which build you are using:
From beta version 3.0.0 - Build 3046 of 2 Jan 2019
- added <#snowdepth> tag processing
- added diary.db file
From beta version 3.0.0 - build 3047
- Web token parser updated to cope with html tag characters "<>" in the format string e.g. <#TapptempH format="dd' 'MMM' 'yyyy' at 'HH:mm''">
- All record Value tags should now return '---' and Date tags '----' until they are first set.
- <#MoonAge>, <#MoonPercent>, <#MoonPercentAbs> - all given new 'dp' and 'rc' parameters.
From version 3.1.1 - build 3054
- Adds new web tags <#snowlying>, <#snowfalling>, both provide 1|0 responses
From version 3.2.0 - build 3056 of 19 November 2019:
- Enables alarms as per Cumulus 1
- New Alarm page under Settings
- Alarms are shown visually on the dashboard
- Due to browser restrictions, alarm sounds on the browser page may require you to click a button on the first alarm in order to hear it.
- You can add the MX admin site to your browsers list of sites allowed to play sound automatically. Your browser should "learn" that you want to allow sounds to play automatically.
- Alarm sound files should be placed in the /interface/sounds folder, they must be a browser compatible format (mp3 are good). The alarm settings for the sound file should be just the filename without any path
- Lots of new web tags not available in Cumulus 1, see release announcement for details
From Version 3.2.2 - build 3058
- Implements the missing <#txbattery> web tag
From version 3.5.1 - build 3072 of 10 April 2020
- Implements the tags that indicate when records are broken
- You configure whether if a record is set it turns off after 24 hours or a different period.