Category:Cumulus MX: Difference between revisions

If you are new to Cumulus, then you will be wondering what benefits Cumulus MX has over other weather recording software. In that case start by reading [[About Cumulus|the article that introduces Cumulus]].  

This Wiki article was originally created by Mark Crossley and contained exactly what Steve Loft said in the [https://cumulus.hosiene.co.uk/viewforum.php?f=39 MX early builds support forum] when he first started experimenting with Cumulus MX and access was restricted to those willing to experiment with his tests.

In this rewrite, somebody who used to use Cumulus 1, and experimented with MX, before moving over to MX first on a Windows PC and then on a Raspeberry Pi, has attempted to ensure this article contains everything I learnt from my experience as I explore more of the functionality of MX; and as I learn more from posts in the forum.  

If you can correct any mistakes I make, have a suggestion as to how to make it clearer, can add anything I have not yet covered, or know something that I might not know, then please remember, anyone can update this article, I don't have any special access in the Wiki and any page I edit can be edited/corrected by anyone else.  

During a period of my time in employment I was responsible for approving documentation on a large computerisation project, and later for supplying updated information for a public faced web site, and in both cases there were house style, and I probably continue to use that style. You might be afraid to add your contribution because my style is not the same as your natural one. Don't worry; as long as you use short paragraphs or bullet points, with lots of headings, then your contribution can blend in.

As this article has grown, I have been able to shorten it by moving material to new articles, hence you will see a lot of cross-references below. You may have suggestions for what else can be moved out of this article into separate articles?

If you have some ideas, but are unsure whether to apply changes, please use the discussion page first. If you have ideas, but want someone else to add them here, please write your suggestion in the support forum for wiki suggestions at https://cumulus.hosiene.co.uk/viewforum.php?f=38

As a new Cumulus user, using MX on Windows (but with no experience of Cumulus 1), the separate article linked from the heading to this section will still interest you as it is a definitive guide to installing Cumulus MX on Windows.

Whether you have been using Cumulus 1, or are new to Cumulus, you may be tempted to install MX on a Raspberry Pi. A Raspberry Pi (and similar devices from other manufacturers) is much simpler than a normal computer, but it can still run various operating systems that allow you to use it perhaps for both running MX and a web server. Find out more in the article linked in the heading.

=== Cross reference to article on [[Updating MX to new version]] ===

Whether you have not updated for a long time, or simply wonder whether you are updating the easiest way, follow the link in the heading for this section, to an article that focusses on all to do with updating from one MX version to another.

Critically, this linked article contains advice '''both''' for those updating each time there is a new release available '''and''' for those using an old version (perhaps when MX was still in beta) wishing to update to a new version skipping some in-between versions and need advice on whether to do it in multiple steps.

A new FAQ for MX has been started at [[Cumulus_MX_FAQ|another page]]. As I add the link here, the Cumulus MX FAQ is in a mess, but hopefully someone will have time to sort it out.

Meanwhile most MX specific questions are now covered by the updating of text on this "Cumulus MX" page.

The [[MX Issues|Cumulus MX Known Issues]] article was based on [https://cumulus.hosiene.co.uk/viewtopic.php?f=39&t=12943 Steve Loft's forum post] so based on an early beta version of MX, and there has been minimal attempt to update the contents as MX has been developed further and come out of the original beta. <big>Anyone willing to check the linked article and do a further update?</big>

 

 

 

 

''The following was written by Steve Loft when he first made his MX beta available to Cumulus users. Although somewhat outdated it is preserved here.''

Before I swapped, I made a detailed study to check MX could do all I used to do with Cumulus 1 and much more.  Before I add items to this article I play around with MX experimenting with what works and what does not work, but I have saved you the pain of where I went wrong, just telling you what is correct. I do need to add, that I don't have a separate testing environment, and therefore I am not willing to attempt anything that might muck up my collecting of weather information, plus my knowledge of modern technology is poor as I belong to that generation who did not have desktop computers, nor mobile devices, until some time into my working life. This all places restrictions on what I can test out, and therefore on the coverage of these notes.

'''If anyone else, can improve these notes, wants to split off more parts, or in any other way make the documentation better, then please do.  This article already contains improvements that were suggested by others.'''

== Replacing Cumulus 1 ==

See [[Moving from Cumulus 1 to MX]] article. If you wish to run MX on Windows, then you can unzip the contents of the download package over your cumulus 1 installation, i.e. so the same data and Reports folders continue to be used. But it would be best if you take a back-up copy of the Cumulus 1 installation first!

The package contains several extra .dll files, and everything else you need, to continue to read from your weather station, to load up the admin interface (there are some settings you will need to change), and some simple web templates (that replace the standard Cumulus 1 ones).  You might want to read topics on the MX support forum to discover about other people's experiences.

== Completely new MX installation ==

See earlier for links to other articles about installing on a Windows PC or a Raspberry Pi. Here only a brief indication of installation is covered.

*Create a new directory (recommended name CumulusMX) and unzip the contents of the [[Software|download package]] into it.

*The package contains everything else you need to read from your weather station (if it is a supported model), to load up the admin interface (for settings and some simple templates used to create web pages to see on a device connected to your home network).  You might want to read topics on the MX support forum to discover about other people's experiences.

== Running Cumulus MX ==

 

If you are running MX for the first time, without a configuration file (none is included in download package), see [[Cumulus.ini#Cumulus_MX|here]] for screen shots showing what you see as the engine starts running, and what you see in the admin interface where you set your weather station type. In that link there are more instructions.

The software currently called .NET was originally for all operating systems, but Microsoft then decided to restrict it to just Windows (and that was the position while MX was being developed and is still true in early 2020 when this article was rewritten), mostly to encourage greater dominance by Microsoft software and hardware.  

'''Perhaps confusingly, in November 2020, there will be change around of names, and the multi-operating system .NET Core product will take over the .NET name as version 5. I don't pretend to understand the technical details, but the impression I get is that the new .NET in November will be similar to Mono, so apps designed for that will still work, but apps using .NET to make code designed for windows will stop working'''. Since the Cumulus code is currently coded to behave slightly differently using .NET and using MONO, I guess it is possible old versions of MX might stop working when the new .NET is installed via Windows Update.

For those who downloaded the first MX Beta in January 2015, the code was only experimental and that version had to be run by a Windows Administrative User, but Steve Loft soon improved the code and now none of the code requires any elevated rights and it can be run by a normal user (or  a user with administrative rights) without needing to be started by '''Run as administrator'''.

 

However, Cumulus MX initiates a web server, which is what runs the [[MX_Administrative_Interface|Admin Interface]]. To access that, all users need to be given elevated rights to the port on which the web server runs. By default this is port 8998, so that is used in the example below of the one-off command needed to give all users access to the port. You can use a '''-port=nnnn''' parameter when starting MX to make it use another port, if you use that then the command below needs revising accordingly.

 

To enter the command, first open a command window as administrator. One way to do this is to right click the windows symbol at the start of the windows task bar. The option to choose there is dependent on some settings which determine what appears when you right click:

*the normal default on Windows 10 is '''Windows PowerShell (admin)''',

*the normal default on earlier versions of Windows is  '''Command Prompt (Administrator)'''  

Whichever of these you can use, the result is it opens a new window on your monitor with a prompt for typing. In that window type the command:

You only need to do that once. If you do not issue this command, administrative rights are needed every time to access the port.

Talking about command windows, if you want to check that the port is open for listening (i.e. able to access the admin interface) type <tt>netstat -an | findstr 8998</tt> into the command window.

Look at your hub or router (this should have come with instructions on how to access its settings in your browser) and on one screen it should show what devices are connected to your LAN and wifi. Look for the IPv4 address, for example 192.168.1.64, it has assigned to the device where MX is running. That is what should replace the "*". However, there is one more complication, either the Windows networking settings may change, or else your hub or router may reconfigure, both can happen at any time and both can assign a different IPv4 address to the device running MX.

 

To give your Computer a fixed address for the MX admin interface,

#first find the network card via Network and Sharing Centre (Control Panel),

# In the window that opens, right click on Internet Protocol Version 4 (TCP/IP 4),  

#Next select properties

# Onn that pop up screen tell the computer to "use the following IP address"

#Fill out the form with

#* a subnet mask of 255.255.255.0 and  

#*gateway address between 192.168.1.1 and 192.168.1.254 (depending on the address of your hub/router).

 

 

 

 

There are 3 ways on Windows to create a shortcut to run MX, as mentioned above you can't just click on the executable in file manager, because Windows needs to be told the path for loading all the related .dll files.

# Next '''start the admin interface''', it does not need to run all the time, but only when you need it (when you first use MX you will need it to access the settings where you tell MX what type of station you have and what units you want to use, and set various timing options), it normally runs on port 8998 (to vary that there is a '''-port''' parameter that is followed by required port and that port parameter has to be entered every time you start MX if you are not using the default port). More information on admin interface [[MX Administrative Interface|in separate article]].

**For example, in 'Raspbian' on the Raspberry Pi, you can install mono with the following commands, but '''first you need to have set up various pre-requisites''' (see [[Setting_up_Raspberry_Pi]] article  for details):

 

Note that you do need to have the '''mono-complete''' package installed, not just the Mono for developers.

Next start the administrative interface, basically same as described for Windows above. More information on admin interface [[MX Administrative Interface|in separate article]].

=== Notes by Steve Loft ===

 

 

 

 

**If you have a Raspberry Pi 2, there is a later version of Mono available, which you may find works better that the one in the standard distribution, particularly if you use decimal commas. Mono 3.2.8 (which is the default in some Linux distributions) will not work if you use commas for decimals, as in some countries.

 

 

In some builds of MX you have to run as "root", there are ways of giving "root" like permissions when running MX as another user, see forum for details until this section has been updated.

The best way to stop MX is by sending a '''control and C''' sequence to it. There is a start stop routine discussed in the support forum, where you will also find some topics about shutdown issues.

The MX source listing suggests it also accepts ''control and break'', '''Close main window''', ''user logoff'', and '''system shutdown''' and should still attempt to stop running tidily. There are various tasks like writing final values to log files and recreating the configuration file, [[Cumulus.ini]], that it needs to do as part of the MX closing routine, and obviously early closure of the device running MX (such as that caused by a power cut) will prevent a tidy end to MX.

= Executables =

The MX package, at time of typing this, includes two executables:

== CumulusMX.exe ==

Whilst effectively MX is run by a '''CumulusMX.exe''' or '''sudo mono CumulusMX.exe''' depending on device, you actually need to ensure all the other components are loaded, so you either have a package that runs it for you, or you click a shortcut that includes the necessary path setting.

=== Optional parameters to add to the instruction to run the MX engine ===

Beta builds in MX version 3.0.0 had an optional parameter <tt>-wsport nnnn</tt> that determined which port (represented by a 4 digit number ''nnnn'') was used for '''WebSockets'''. That parameter is now deprecated as WebSockets in all builds since 3045 uses the same port as the rest of the [[MX_Administrative_Interface#The_API_interface|Admin Interface]]. The remaining parameters that are still available are described in subsequent sub-sections.

When Cumulus starts, it will display the URL of the user interface. It runs on port 8998 by default; if this is not suitable for some reason you can over-ride it using the '-port' parameter on the command line, e.g. to use port 9999 instead:

<pre>sudo mono CumulusMX.exe -port 9999</pre>

==== Parameter for adding debugging ====

MX has a default level of logging that stores in  the [[MXDiags]] folder a log file that shows some of the interaction with the weather station and some of the output actions done as MX runs. A new log is started each time MX is restarted.

If there is a problem, then there is a great benefit in actually increasing the level of detail in these logs; and that is done either within the settings ('''options''' section of ''station settings'') in admin interface  while MX is running, or by adding 1 or 2 parameters when you start MX.  Obviously this log file continues to grow, the longer MX is left running, and if debugging is switched on the file will grow in size must faster. Consequently, the default is not to add the extra debugging information and the settings can be used to switch it off again if you do have it switched on. Whether you start it with a parameter or enable it within settings, stopping MX will end the extra debugging, and on restart it will default back to no debugging unless turned on again with parameter or setting.

You can also add  '''CumulusMX.exe -debug''' (to have full debugging of actions by MX turned on as MX starts), and/or '''CumulusMX.exe -Logging=1''' (for the station to MX transfers to have increased debugging logging).  

<pre>sudo mono CumulusMX.exe -debug -Logging=1</pre>

Since this parameter is applied when you start MX, it applies while MX continues to run. Obviously, it must be applied every time you start MX if you want this increased level of logging to continue every time you restart MX.

The comments in the MX source suggests -debug turns on both debug and data logging (see [[MX_Administrative_Interface#Options|Station_Settings#Options]] in admin interface settings), but I believe that is wrong as per example above, there are 2 separate parameters.

sudo mono CumulusMX.exe -lang en-GB

</pre>

If you are not sure what value you need to supply for the -lang parameter, there is a list here - http://msdn.microsoft.com/en-gb/library/ee825488%28v=cs.20%29.aspx. You need to supply the code in the first column ("Language Culture Name") in that list.

Note that this does not affect the language used by Cumulus MX (although it may in the future), it affects the decimal separator and the list separator.

Note that you ''may'' need to supply your administrator password after typing the 'sudo ...' command line. The system will prompt you for this if it is needed.

== ExportMySQL.exe ==

This second exe file has been available since the original MX package as Steve Loft developed this in April 2015, but sadly few people even notice it exists, and if they do, it is unlikely they know how to use it. Hopefully, some people will read this section and find out!

Obviously it was updated when Mark Crossley added the Feels Like fields to log files.

The executable has a mandatory single parameter that tells it which log files to read, there are only 3 possible parameters ("dayfile", "monthly", or path to a file). It needs to know what locale (or culture settings) it is to use to work out what character separates each item in the log file list. It also needs to read your Cumulus.ini file, as it takes these "input parameters" from MySQL section in that:

#* In '''Dayfile.txt upload''' section, give your database table a name, or accept default ''Dayfile''.

# Now scroll down to '''Create database table (save settings first)'''

# Now you have a database table ready, you can use the executable to read all lines in your '''CumulusMX/data/dayfile.txt''' log file.

# Run this executable in that terminal display (or command window) by using  '''sudo mono ExportMySql.exe daily''' or <tt>ExportMySql.exe daily</tt> depending on device.

#* Here click '''Create Monthly'''

#* In the same '''Monthly log file upload''' section, now select '''Enable'''.

# Now you have a database table ready, you can use the executable to read all lines in either one (if path to that file is in parameter), or every (if parameter is monthly) standard log file.

#*If the parameter is "monthly" it will look in folder '''data'''  for every file it can find with a file name of datestring + "log.txt" where datestring is a 3 letter code (in your locale) for each month (1 to 12) followed by a 2 digit year (from "00" to "99") so that is how it finds every standard log file in the folder.

# Run this executable in that terminal display (or command window) by using  '''sudo mono ExportMySql.exe monthly''' or <tt>ExportMySql.exe monthly</tt> depending on device.

Please be aware that the transfer to the database table adds two columns where bearings in the original log file given in degrees are output as compass directions, and these use up to 3 letters of how the compass directions are defined in the '''strings.ini''' file. Thus the number of columns in the database table will be at least 2 more than the number of fields in the log files. It is also important to stress that whilst the database table must contain one column defined for each field (plus the extra 2) being uploaded, you can add even more columns to your table if you want and populate those some other way. For example, I have added a Canadian Humidity Index (Humidex) column which is not in the standard logs, but is calculated by Cumulus, and can be calculated from columns that are uploaded from the standard log. Humidex is not uploaded by either ExportMySQL or the normal CumulusMX process, but neither objects to extra columns being there.

When testing this, I had some log files produced by various old versions of Cumulus 1 in my MX data folder as well as the log files in has generated since I swapped to MX. Plus I had used a PHP script to add feels like to those log files produced before version 3.6.0 and to correct feels like for those log entries made by versions 3.6.0 to 3.6.9 inclusive because they used a different formula to the one being used from version 3.6.10.  This php script is a web page with a HTML form and can be obtained from the forum in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 Create Missing for MX]

I notice that the database rows produced by those short log file lines produced by say version 1.9.0 had nulls entered for all subsequent columns, except '''Feels Like''' and this column was initialised at 0.0!

For those log files produced by the final version 1.9.4, all columns are populated although feels like is set to 0.0.

For most Cumulus users, this whole section can be skipped, but I have included it for those few users who have a technical slant and might want to understand more.

Many of the libraries included by MX are very obsolete. However, Mark Crossley, the current developer, said the following on 30 Sept 2019:

"Chasing the latest versions of all the packages for the sake of it is a thankless task, and requires considerable effort to regression test each update. I am only updating packages when required to fix issues or for platform compatibility."

Just to mention the other side of this balance. It is difficult to code an addition to MX that works with these obsolete versions of libraries. All documentation provided by providers of the libraries relates to current versions of the packages (and what is documented to work now often will not work with obsolete versions). The documentation for the packages that are no longer supported is only available in archive sites if available at all.

==Library Software for the MX engine==

The distribution zip contains various '''.dll''' files and these are the libraries used by MX itself. The exact mix of libraries included has varied at various times, the list below is a snapshot of those included at the version that was investigated when this article was extended to include this section, and may not be right for the current MX version.

===Devart===

===FluentFTP===

As the name suggests, this is used by MX for controlling the file transfer functionality. This component was first introduced at version 3.0.0 build 3045 (adding more functionality than that available from System.Net.FtpClien, which was used at earlier MX builds), and FluentFTP.dll has been updated at some subsequent MX releases, see the announcements for details.

===Linq===

Language INtegrated Query is used to work with sequences of items and pick the ones that are needed, putting them into output format required. MX uses two files in connection with preparing output for Twitter. There is a third Linq file for other processing.

When MX added capability to talk to other devices using the MQTT protocol, it added this component for that optional functionality.

===Newtonsoft===

This component is used for processing JSON strings. It is a very popular choice for developers, and used therefore very widely. However, '''SystemText''' has superseded it, so MX is using an obsolete method.

===Renci SSH===

This component is server connection software, it is what processes the host name, password, and so on.

===SQLite3===

This is used for all interactions with the [[Weather Diary|weather diary]].

===Unosquare===

The EmbedIO file is open-source software that handles the web-sockets functionality of MX. The Swan file is open-source software that handles JSON formatting and threading of tasks in MX.

#Also known by some as ''Twitter Bootstrap'' which gives a clue as to its developer and to its origins as an internal tool for those building Twitter, that company still keep making updates as it is now the most popular styling library of all those available widely.

*MX uses Bootstrap version 3.3.7, which is very restricted in what it offers

**Bootstrap version 5 is available (http://getbootstrap.com), so MX is using an obsolete library

** Bootstrap version 4.5.0 (Bootstrap 4 released as alpha in August 2015, beta in August 2017, and with fully working releases frequently from January 2018) was very widely praised for its improved functionality, and ability to work with latest jQuery and multiple modern devices/browsers.   

*MX does not implement key features of Bootstrap like colouring text according to what it represents (primary, secondary, information, warning etc.)

#Thus dataTables does all the work of providing the ability to present the data in a HTML table, the functionality to move between multiple pages needed (as MX sends only up to 10 lines of a log file at a time to the admin interface).

#The free version of dataTables used by MX lacks the most useful functionality that needs a subscription licence. For example, its editing functionality requires a subscription.

*This is an editing tool that can read what is in dataTables, create what it calls a modal (a pop-up dialog) where rows can be added, edited, or deleted individually.

** MX when it added editing of log files at version 3.4.5 - Build 3069 (Friday 13 March 2020) adopted this software as it was free (although Mark Crossley said in his release notice: '''"The main thrust of this release is to add some log file editing capability to Cumulus MX.  It works on all three log file types, but it is fairly basic at present. You can edit or delete lines in the files.  The editing has to be done via pop-up dialog.'''

**'''I only found two libraries that support JQuery dataTables editing, one is very comprehensive - but costs $$$ - the other is free.  The free version does not currently support in-line editing of the table which is a shame.'''

**  '''If any web guru out there can come up with a better solution please post about it on the forum, or send a pull request."''') ''(By the way, it is possible to provide in-line editing and make it work with the existing api interface, but making it compatible with the obsolete software used by MX is hard).''

#* Unfortunately, there are interdependencies between all the library code used by MX, so you cannot simply update this component (see next library item).

===Jquery Template===

* This is also obsolete, and therefore will only work with old versions of jQuery (1.4.2 to 1.11.0).

*It was originally available from jQuery downloads, but they now offer '''jsRender.js''' for this functionality.

* It basically is used to bind the contents of objects (like array elements) into particular locations within HTML.