Category:Cumulus MX: Difference between revisions

If you are new to Cumulus, then you will be wondering what benefits Cumulus MX has over other weather recording software. In that case start by reading [[About Cumulus|the article that introduces Cumulus]].  

This Wiki article was originally created by Mark Crossley and contained exactly what Steve Loft said in the [https://cumulus.hosiene.co.uk/viewforum.php?f=39 MX early builds support forum] when he first started experimenting with Cumulus MX and access was restricted to those willing to experiment with his tests.

In this rewrite, somebody who used to use Cumulus 1, and experimented with MX, before moving over to MX first on a Windows PC and then on a Raspeberry Pi, has attempted to ensure this article contains everything I learnt from my experience as I explore more of the functionality of MX; and as I learn more from posts in the forum.  

If you can correct any mistakes I make, have a suggestion as to how to make it clearer, can add anything I have not yet covered, or know something that I might not know, then please remember, anyone can update this article, I don't have any special access in the Wiki and any page I edit can be edited/corrected by anyone else.  

During a period of my time in employment I was responsible for approving documentation on a large computerisation project, and later for supplying updated information for a public faced web site, and in both cases there were house style, and I probably continue to use that style. You might be afraid to add your contribution because my style is not the same as your natural one. Don't worry; as long as you use short paragraphs or bullet points, with lots of headings, then your contribution can blend in.

As this article has grown, I have been able to shorten it by moving material to new articles, hence you will see a lot of cross-references below. You may have suggestions for what else can be moved out of this article into separate articles?

If you have some ideas, but are unsure whether to apply changes, please use the discussion page first. If you have ideas, but want someone else to add them here, please write your suggestion in the support forum for wiki suggestions at https://cumulus.hosiene.co.uk/viewforum.php?f=38

 

You may be reading this article on MX as a Cumulus 1 user considering moving to MX, if so then it explains all issues when moving from Cumulus 1 to MX, including what functionality you will gain and what you will lose by moving to MX. The article will help you to decide whether to stick with Cumulus 1's stable release as many MX releases have bugs as the developer tries revising the code, to simplify it, as well as to introduce new features.

As a new Cumulus user, using MX on Windows (but with no experience of Cumulus 1), the separate article linked from the heading to this section will still interest you as it is a definitive guide to installing Cumulus MX on Windows.

Whether you have been using Cumulus 1, or are new to Cumulus, you may be tempted to install MX on a Raspberry Pi. A Raspberry Pi (and similar devices from other manufacturers) is much simpler than a normal computer, but it can still run various operating systems that allow you to use it perhaps for both running MX and a web server. Find out more in the article linked in the heading.

Before I swapped, I made a detailed study to check MX could do all I used to do with Cumulus 1 and much more.  Before I add items to this article I play around with MX experimenting with what works and what does not work, but I have saved you the pain of where I went wrong, just telling you what is correct. I do need to add, that I don't have a separate testing environment, and therefore I am not willing to attempt anything that might muck up my collecting of weather information, plus my knowledge of modern technology is poor as I belong to that generation who did not have desktop computers, nor mobile devices, until some time into my working life. This all places restrictions on what I can test out, and therefore on the coverage of these notes.

'''If anyone else, can improve these notes, wants to split off more parts, or in any other way make the documentation better, then please do.  This article already contains improvements that were suggested by others.'''

== Replacing Cumulus 1 ==

See [[Moving from Cumulus 1 to MX]] article. If you wish to run MX on Windows, then you can unzip the contents of the download package over your cumulus 1 installation, i.e. so the same data and Reports folders continue to be used. But it would be best if you take a back-up copy of the Cumulus 1 installation first!

The package contains several extra .dll files, and everything else you need, to continue to read from your weather station, to load up the admin interface (there are some settings you will need to change), and some simple web templates (that replace the standard Cumulus 1 ones).  You might want to read topics on the MX support forum to discover about other people's experiences.

== Completely new MX installation ==

See earlier for links to other articles about installing on a Windows PC or a Raspberry Pi. Here only a brief indication of installation is covered.

*Create a new directory (recommended name CumulusMX) and unzip the contents of the [[Software|download package]] into it.

*See notes below for extras required in various operating systems.

*The package contains everything else you need to read from your weather station (if it is a supported model), to load up the admin interface (for settings and some simple templates used to create web pages to see on a device connected to your home network). You might want to read topics on the MX support forum to discover about other people's experiences.

== Running Cumulus MX ==

 

If you are running MX for the first time, without a configuration file (none is included in download package), see [[Cumulus.ini#Cumulus_MX|here]] for screen shots showing what you see as the engine starts running, and what you see in the admin interface where you set your weather station type. In that link there are more instructions.

The software currently called .NET was originally for all operating systems, but Microsoft then decided to restrict it to just Windows (and that was the position while MX was being developed and is still true in early 2020 when this article was rewritten), mostly to encourage greater dominance by Microsoft software and hardware.  

'''Perhaps confusingly, in November 2020, there will be change around of names, and the multi-operating system .NET Core product will take over the .NET name as version 5. I don't pretend to understand the technical details, but the impression I get is that the new .NET in November will be similar to Mono, so apps designed for that will still work, but apps using .NET to make code designed for windows will stop working'''. Since the Cumulus code is currently coded to behave slightly differently using .NET and using MONO, I guess it is possible old versions of MX might stop working when the new .NET is installed via Windows Update.

You only need to do that once. If you do not issue this command, administrative rights are needed every time to access the port.

# Next '''start the admin interface''', it does not need to run all the time, but only when you need it (when you first use MX you will need it to access the settings where you tell MX what type of station you have and what units you want to use, and set various timing options), it normally runs on port 8998 (to vary that there is a '''-port''' parameter that is followed by required port and that port parameter has to be entered every time you start MX if you are not using the default port). More information on admin interface [[MX Administrative Interface|in separate article]].

**For example, in 'Raspbian' on the Raspberry Pi, you can install mono with the following commands, but '''first you need to have set up various pre-requisites''' (see [[Setting_up_Raspberry_Pi]] article  for details):

 

Note that you do need to have the '''mono-complete''' package installed, not just the Mono for developers.

Next start the administrative interface, basically same as described for Windows above. More information on admin interface [[MX Administrative Interface|in separate article]].

=== Notes by Steve Loft ===

''please note these notes ARE now obsolete, library routines have changed a lot since this was written in 2014''

'''Any volunteers to replace this section with up to date information?'''

**If you have a Raspberry Pi 2, there is a later version of Mono available, which you may find works better that the one in the standard distribution, particularly if you use decimal commas. Mono 3.2.8 (which is the default in some Linux distributions) will not work if you use commas for decimals, as in some countries.

**On Linux you will need library '''libudev.so.0''' which may not be installed by default. Installing '''package libudev0''' may resolve this. There may be issues if you are using a 64-bit version of Linux. I'm not sure what the resolution is at the moment, if this is the case.

You need to specify something like '''/dev/ttyUSB0''' for the connection for your weather station. This is set in the "station settings" and stored in the [[Cumulus.ini#station|ComportName attribute]] in Cumulus.ini configuration file.

In some builds of MX you have to run as "root", there are ways of giving "root" like permissions when running MX as another user, see forum for details until this section has been updated.

The best way to stop MX is by sending a '''control and C''' sequence to it. There is a start stop routine discussed in the support forum, where you will also find some topics about shutdown issues.

The MX source listing suggests it also accepts ''control and break'', '''Close main window''', ''user logoff'', and '''system shutdown''' and should still attempt to stop running tidily. There are various tasks like writing final values to log files and recreating the configuration file, [[Cumulus.ini]], that it needs to do as part of the MX closing routine, and obviously early closure of the device running MX (such as that caused by a power cut) will prevent a tidy end to MX.

 

 

Whilst effectively MX is run by a '''CumulusMX.exe''' or '''sudo mono CumulusMX.exe''' depending on device, you actually need to ensure all the other components are loaded, so you either have a package that runs it for you, or you click a shortcut that includes the necessary path setting.

 

=== Optional parameters to add to the instruction to run the MX engine ===

 

Beta builds in MX version 3.0.0 had an optional parameter <tt>-wsport nnnn</tt> that determined which port (represented by a 4 digit number ''nnnn'') was used for '''WebSockets'''. That parameter is now deprecated as WebSockets in all builds since 3045 uses the same port as the rest of the [[MX_Administrative_Interface#The_API_interface|Admin Interface]]. The remaining parameters that are still available are described in subsequent sub-sections.

==== Parameter for changing Port ====

When Cumulus starts, it will display the URL of the user interface. It runs on port 8998 by default; if this is not suitable for some reason you can over-ride it using the '-port' parameter on the command line, e.g. to use port 9999 instead:

<pre>sudo mono CumulusMX.exe -port 9999</pre>

MX has a default level of logging that stores in  the [[MXDiags]] folder a log file that shows some of the interaction with the weather station and some of the output actions done as MX runs. A new log is started each time MX is restarted.

If there is a problem, then there is a great benefit in actually increasing the level of detail in these logs; and that is done either within the settings ('''options''' section of ''station settings'') in admin interface  while MX is running, or by adding 1 or 2 parameters when you start MX.  Obviously this log file continues to grow, the longer MX is left running, and if debugging is switched on the file will grow in size must faster. Consequently, the default is not to add the extra debugging information and the settings can be used to switch it off again if you do have it switched on. Whether you start it with a parameter or enable it within settings, stopping MX will end the extra debugging, and on restart it will default back to no debugging unless turned on again with parameter or setting.

You can also add  '''CumulusMX.exe -debug''' (to have full debugging of actions by MX turned on as MX starts), and/or '''CumulusMX.exe -Logging=1''' (for the station to MX transfers to have increased debugging logging).  

<pre>sudo mono CumulusMX.exe -debug -Logging=1</pre>

Since this parameter is applied when you start MX, it applies while MX continues to run. Obviously, it must be applied every time you start MX if you want this increased level of logging to continue every time you restart MX.

The comments in the MX source suggests -debug turns on both debug and data logging (see [[MX_Administrative_Interface#Options|Station_Settings#Options]] in admin interface settings), but I believe that is wrong as per example above, there are 2 separate parameters.

On Linux and (in particular) OS X, Cumulus MX may not be given the correct locale to use, and you may get the default US locale even if that is not your locale. It will output the local it is using when it starts; if it is not correct, close it down and start it again, this time specifying your locale on the command line, using the -lang parameter . For example, in the UK, on a non-Windows device type:

<pre>

sudo mono CumulusMX.exe -lang en-GB

</pre>

If you are not sure what value you need to supply for the -lang parameter, there is a list here - http://msdn.microsoft.com/en-gb/library/ee825488%28v=cs.20%29.aspx. You need to supply the code in the first column ("Language Culture Name") in that list.  

Note that this does not affect the language used by Cumulus MX (although it may in the future), it affects the decimal separator and the list separator.

Note that you ''may'' need to supply your administrator password after typing the 'sudo ...' command line. The system will prompt you for this if it is needed.

== ExportMySQL.exe ==

Obviously it was updated when Mark Crossley added the Feels Like fields to log files.

Put simply, this executable will read log files and insert (insert ignore) rows into an existing database table. Since it only does inserts, despite the name of this function, it is not just for MySQL tables, the included SQL should work with whatever database table type you have.

The executable has a mandatory single parameter that tells it which log files to read, there are only 3 possible parameters ("dayfile", "monthly", or path to a file). It needs to know what locale (or culture settings) it is to use to work out what character separates each item in the log file list. It also needs to read your Cumulus.ini file, as it takes these "input parameters" from MySQL section in that:

#* In '''Dayfile.txt upload''' section, give your database table a name, or accept default ''Dayfile''.

# Now scroll down to '''Create database table (save settings first)'''

#* Here click '''Create Dayfile'''

# Now you have a database table ready, you can use the executable to read all lines in your '''CumulusMX/data/dayfile.txt''' log file.

# Run this executable in that terminal display (or command window) by using  '''sudo mono ExportMySql.exe daily''' or <tt>ExportMySql.exe daily</tt> depending on device.

#* Click '''Save''' to ensure this setting is updated

#*If the parameter is "monthly" it will look in folder '''data'''  for every file it can find with a file name of datestring + "log.txt" where datestring is a 3 letter code (in your locale) for each month (1 to 12) followed by a 2 digit year (from "00" to "99") so that is how it finds every standard log file in the folder.

# Open a terminal display (if you are using Windows then, open a Command Window, a Windows Powershell window, or a Windows Terminal window)

# Run this executable in that terminal display (or command window) by using  '''sudo mono ExportMySql.exe monthly''' or <tt>ExportMySql.exe monthly</tt> depending on device.

Please be aware that the transfer to the database table adds two columns where bearings in the original log file given in degrees are output as compass directions, and these use up to 3 letters of how the compass directions are defined in the '''strings.ini''' file. Thus the number of columns in the database table will be at least 2 more than the number of fields in the log files. It is also important to stress that whilst the database table must contain one column defined for each field (plus the extra 2) being uploaded, you can add even more columns to your table if you want and populate those some other way. For example, I have added a Canadian Humidity Index (Humidex) column which is not in the standard logs, but is calculated by Cumulus, and can be calculated from columns that are uploaded from the standard log. Humidex is not uploaded by either ExportMySQL or the normal CumulusMX process, but neither objects to extra columns being there.

When testing this, I had some log files produced by various old versions of Cumulus 1 in my MX data folder as well as the log files in has generated since I swapped to MX. Plus I had used a PHP script to add feels like to those log files produced before version 3.6.0 and to correct feels like for those log entries made by versions 3.6.0 to 3.6.9 inclusive because they used a different formula to the one being used from version 3.6.10.  This php script is a web page with a HTML form and can be obtained from the forum in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 Create Missing for MX]

I notice that the database rows produced by those short log file lines produced by say version 1.9.0 had nulls entered for all subsequent columns, except '''Feels Like''' and this column was initialised at 0.0!

For those log files produced by the final version 1.9.4, all columns are populated although feels like is set to 0.0.

For most Cumulus users, this whole section can be skipped, but I have included it for those few users who have a technical slant and might want to understand more.

Cumulus MX uses '''library''' software (i.e. software written by others and made available by the provider and often also by other content delivery nodes  or 'cdn')  for a lot of the standard functionality. The library software for the admin interface and the separate library software for the standard web pages are both mostly included in the distribution zip, although some is used via a link to a cdn.

Many of the libraries included by MX are very obsolete. However, Mark Crossley, the current developer, said the following on 30 Sept 2019:

"Chasing the latest versions of all the packages for the sake of it is a thankless task, and requires considerable effort to regression test each update. I am only updating packages when required to fix issues or for platform compatibility."

Just to mention the other side of this balance.  It is difficult to code an addition to MX that works with these obsolete versions of libraries.  All documentation provided by providers of the libraries relates to current versions of the packages (and what is documented to work now often will not work with obsolete versions). The documentation for the packages that are no longer supported is only available in archive sites if available at all.

==Library Software for the MX engine==

===Devart===

The two files used are both related to the database functionality of MX.

===FluentFTP===

As the name suggests, this is used by MX for controlling the file transfer functionality. This component was first introduced at version 3.0.0 build 3045 (adding more functionality than that available from System.Net.FtpClien, which was used at earlier MX builds), and FluentFTP.dll has been updated at some subsequent MX releases, see the announcements for details.

Language INtegrated Query is used to work with sequences of items and pick the ones that are needed, putting them into output format required. MX uses two files in connection with preparing output for Twitter. There is a third Linq file for other processing.

===MQTT===

When MX added capability to talk to other devices using the MQTT protocol, it added this component for that optional functionality.

===Newtonsoft===

This component is used for processing JSON strings. It is a very popular choice for developers, and used therefore very widely. However, '''SystemText''' has superseded it, so MX is using an obsolete method.

===Renci SSH===

This component is server connection software, it is what processes the host name, password, and so on.

===HTTP===

These files handle the optional HTTP functionality.

The EmbedIO file is open-source software that handles the web-sockets functionality of MX. The Swan file is open-source software that handles JSON formatting and threading of tasks in MX.

#Also known by some as ''Twitter Bootstrap'' which gives a clue as to its developer and to its origins as an internal tool for those building Twitter, that company still keep making updates as it is now the most popular styling library of all those available widely.

#Thus dataTables does all the work of providing the ability to present the data in a HTML table, the functionality to move between multiple pages needed (as MX sends only up to 10 lines of a log file at a time to the admin interface).

#The free version of dataTables used by MX lacks the most useful functionality that needs a subscription licence. For example, its editing functionality requires a subscription.

*This is an editing tool that can read what is in dataTables, create what it calls a modal (a pop-up dialog) where rows can be added, edited, or deleted individually.