Cumulus Wiki

Your Own Server and Software: Difference between pages

Page Discussion
(Difference between pages)

Software (view source)

Revision as of 19:37, 20 August 2022

9,495 bytes removed ,  20 August 2022
m
no edit summary
 
mNo edit summary
 
[[Category:Cumulus MX]] This Wiki page has been created for those who use Cumulus MX.
{{Template:WorkInProgressBanner}}
This page describes how I set up a web server and a database server on my Raspberry Pi computer.
 
= What is Cumulus MX? =
For more information about running MX in a Linux environment please see [[MX on Linux]].
 
: a cross platform version of Cumulus software that runs on Microsoft Windows, all varieties of UNIX and Linux (including the Raspberry Pi Operating System), and Mac OS X.
For more information about Raspberry Pi computers see [[Raspberry Pi computer page]].
 
'''NOTE relating to MX Release v3.9.1 or later: On Linux/RPi/macOS, Mono version 5, or 6, is needed for CreateMissing.Exe, ExportToMySQL.exe, and CumulusMX.exe. Mono v4 is subject to random crashes.'''
For more information on log files that you might want to convert to database tables, please see [[Diary.db|MX Weather Diary]], [[Log.xml|Legacy Weather Diary]], [[Dayfile.txt|daily summary log (dayfile.txt)]], [[Speciallog.txt]], [[Standard log files]], [[Extra Sensor Files]], and [[Air Link Log.txt]].
 
=Introduction=
 
==CumulusMX.exe Usage Restriction==
This page records the experience of setting up a web server and database server on my computer.
 
Both the mandatory [[MX Administrative Interface|MX Interface]], and the optional [[New Default Web Site Information|default MX web site]], include the [https://www.highcharts.com HighStock] product.
Most web servers run on Linux, although you can still install them on a Microsoft Windows PC.
 
Are you using MX for a non-profit organisation, personal website, or school project?
It is important to say that many different web servers exist, I have chosen to mention a few, and I have tried various ones. Since “apache2” is very well known, and proabably the most widely used server, I will mostly focus on that. Sometimes, you might choose a simple server that is what is described as light, it uses the minimum resources, but also probably offers the least functionality. In contrast, you might choose apache, because it offers the most functionality, and because it is used so widely there are many people who can help you.
Then download MX and start using it with Highcharts today.
 
Are you representing any commercial organisation running a facility that others use? If so, then you cannot use Cumulus MX to share weather data on behalf of your organisation without buying a licence from the High Charts organisation.
=Web server and database server built into Cumulus MX=
 
'''HighCharts rules for usage of their products are described ''' [https://shop.highcharts.com/ on their licence purchase web page], and ''it is your responsibility to ensure compliance''.
==Web Server built into Cumulus MX==
 
== Help with ‘Stable’ MX releases ==
The [[MX Administrative Interface|Admin Interface]] is a set of web pages that allow you to look at your weather data and change settings.
 
If you need any help with Cumulus MX, please first consult, in this Wiki, [[What_to_do_when_I_have_a_problem_with_MX]].
The MX code will create the necessary web server, it uses the Internet Protocol (ip) address of the computer that is running MX and a port (:8998 by default, but MX has an optional parameter to make it use a different port). If you are viewing the interface on the same computer as runs MX, “localhost” can replace the ip address.
 
It is recommended you check you have complied with any instructions in the announcement for the latest MX release in [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=17887 the forum] before seeking help (to see what changes in any release affect you, and what actions you may need to take).
As an aside, there are two different ways to specify ip addresses (IPv4 and IPv6), but this is not place to explain the difference.
 
If you wish to research the documentation on MX to try to resolve your problem yourself, use the entry point of the [[:Category:Cumulus MX]] page for pointers to other Wiki pages connected with MX that may help.
==Databases built into Cumulus MX==
 
If you cannot answer your questions from information in this Wiki (and MX changes so quickly any information here can be very out of date), then use the [https://cumulus.hosiene.co.uk/viewforum.php?f=40 correct MX support forum] and the developer or Cumulus community will offer assistance.
You do not need to know what is in these databases to use MX. I mention them here to explain MX has a built in database server.
 
==Upgrading==
Cumulus MX includes two SQLite databases;
#The first database was added by Steve Loft, but he never documented what the first table in that database was to be used for (see [[Cumulusmx.db]]). The second table in that database is used for [[Recent history]].
#The second database is the Weather Diary [[Weather Diary|documented here]] added by Mark Crossley. Also see [https://cumulus.hosiene.co.uk/viewtopic.php?f=37&t=17919 in the support forum] for a topic comparing the differences between Cumulus 1 and Cumulus MX weather diaries.
#* The [[MX Administrative Interface|Admin Interface]] provides a page to view, or edit, a single date in the weather diary. You can install another tool that edits SQLite databases if you want (see next section).
 
We strongly advise [[Updating_MX_to_new_version#Introduction_to_upgrading_MX|upgrading]] to the latest release:
The legacy Cumulus software used a [[Log.xml|XML file]] to store the weather diary.
* You may wish to wait a few days after a new release comes out, just in case a new release has some bugs
** MX is extremely complex, the developers work on MX edits in their spare time, have settings that suit them, and do not have the full set of sensors that MX can support, so pre-release testing is never perfect
* By switching to latest release, you gain any extra functionality it may offer (although it may not be useful to you), but more importantly you gain fixes of bugs your existing release may have
* Also the main developer can most easily offer support if you are using the build he most recently released.
 
There are inter-dependencies between different files in the MX release distribution:
===Reading content of SQLite databases===
* ''Therefore you must always install into your MX folders, every file included in the download zip''.
** You can take a back-up of your existing installation before you copy in any new files, that will provide you with the ability to regress to the earlier release, if the new release does cause you an issue
* If you customise any file that is in the distribution, either give the new file a new name (perhaps add a prefix of "_" in front of files you edit) or place your tailored version in a different folder
** Either approach will ensure your tailored files are not overwritten by files from the download, and let you compare your tailored file with the new file in the download to see if you need to tweak your tailored file
* It is strongly recommended that you install all files in the download for a new release of MX directly over your existing installation.
** You may end up with some files no longer needed by the new release, left over from older release, but this is better than risking missing some vital files
 
: There is an [[Updating_MX_to_new_version#Installer_Option|installer utility]] available to help with upgrading.
Obviously, Cumulus MX can read and update both of these databases, but there are a number of software packages available that can be used outside MX to access these database files, and to minimise technical detail, two will be named as possible candidates, you will find others online.
 
*Important, please read all of the release notes (in the [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=17887 appropriate forum announcement] threads) for the builds since the version you are upgrading from, especially noting when there are references to necessary once-off actions.
On Microsoft Windows, install phpLiteAdmin (the significant part of that name is in the middle) to read (you can display the whole table very easily) and edit (via a friendly GUI) either, or both, of the SQLite databases that MX uses as mentioned above.
*If you have never upgraded before, or you are currently running a fairly old release of MX, you are advised to read the [[Updating MX to new version]] page in this Wiki,
** you may be recommended to upgrade in stages, as certain releases do essential one off actions that are missed if you skip that release; e.g. rewriting [[Cumulus.ini]] or changes to columns in database tables
** overwriting an existing installation will add any new files, but it won't remove obsolete files, and sometimes [[MX_Basic_info#Library_software|replacement files]] have different names to their predecessors so don't overwrite the old files
*If you are upgrading from a previous version of the legacy Cumulus 1, then you might want to read [[Migrating from Cumulus 1 to MX]] page; and if you need more justification read [[Compare_C1_and_MX|Compare legacy Cumulus and MX]] page
** However, it is likely that information on those pages was written for earlier MX releases, and might be out of date, so like many other people you might want to ask in the [https://cumulus.hosiene.co.uk/viewforum.php?f=40 correct MX support forum] to see if others have more up-to-date advice.
 
Note by Steve Loft: you may find that anti-virus software flag a new release of Cumulus as a ‘threat’, for a few days or weeks after a new version has been released, even though it does not actually contain a virus of any kind. This is due to a system they call ‘reputation’ where they automatically block any versions of software that have recently been released. You may want to consider complaining to the provider of your anti-virus software about this ridiculous system, or better still, use some better anti-virus software.
In Linux use <code>sudo apt install sqlitebrowser</code> to install easy-to-use software to read/edit these databases.
 
== Current Release ==
You may need to change file permissions.
 
===Latest build distribution download===
Note that sqlite databases use sequential file structure, not a relational database design. This implies for updates, rows are physically identified by row number (order of creation), (not by the primary key that SQL accesses use with relational database designs).
 
~ Download [https://github.com/cumulusmx/CumulusMX/releases/download/b3196/CumulusMXDist3196.zip Cumulus MX v3.19.3 build 3196, 25-July-2022]
= Adding a web server and a database server for your own purposes =
 
===Latest MX Source Code===
MX has (optional) functionality that can upload to a web server, it also has (optional) functionality that can update one or more database tables on a database server.
 
~ View/download [https://github.com/cumulusmx/CumulusMX Cumulus MX Current Source Code]
Frequently, people will use an external web server. This might be one of the Wunderground, Windy, Awekas, WeatherCloud, PWSweathe, WOW, APRS, OpenWeatherMap, and WindGuru external sites that MX can automatically update to. It might be you have subscribed to a commercial web server (and optionally database server). that an external provider supplies to you to run either the [[New_Default_Web_Site_Information|3.10.1 onwards default web site]], or it might be for pages you or a [[:Category:WebTools|third party]] have designed.
 
This source code base is updated for all releases resulting from development by M Crossley, it includes some contributions from other developers.
If you have a computer that is left running all the time without consuming a lot of electricity), you ''might'' want to add a web server and database server to that device. For example, see [[Raspberry_Pi_computer_page]] for a computer with low power consumption that a number of Cumulus users have used for their web server and database server.
 
===Raspberry Pi Image===
As with a commercial web server, you can use your own web server to run the default web pages available in the MX download package. Alternatively, you might want this option, if you are creating your own web pages, and want to try them out without exposure to the public over the internet.
 
~ Download [//{{SERVERNAME}}/Downloads/rpi-CumulusMX-lite-3.19.0.zip Raspberry Pi Image for Cumulus MX 3.19.0]
You might select this option to save the subscription costs of a commercial web server (and optionally database server).
 
As at April 2022, the image includes Raspberry Pi. O. S. 11 "Bullseye" version of Linux.
== All in one versus separate components ==
 
<br>
To get maximum benefit when using with MX, you need multiple components:
# web server – there are various options listed below; beware "apache2" is huge as it is very complicated, but secure; other options consume fewer resources, and may be easier to use, but they will also be more restrictive
# PHP script - from [https://windows.php.net/download/] - this provides a way of controlling what SQL is run depending on other factors
# Database server - MySQL Community edition from [mysql.com] or MariaDB from [https://downloads.mariadb.org/].
# Integrating components like ‘’’phpMyAdmin’’ (there are alternatives) from [https://www.phpmyadmin.net/downloads/] which combines PHP and MySQL
 
# This is a pre-built disk image file for your first installation onto a Raspberry Pi computer.
There are a number of packages that combine all of these:
# Do read the instructions for using this image on the [[Raspberry_Pi_Image|Raspberry Pi Image]] page to see what is in the image and how to use it
*If designed for Microsoft Windows and including Apache2, PHP, and either MariaDB or MySQL, the package is often described as ‘’’WAMP’’’
# It is of course possible that sometimes the image provided may have issues for you,
*If designed for UNIX (including Linux), then XAMP will include Apache2, PHP, and either MariaDB or MySQL
#* it is set up so that MX starts running as a service immediately you boot up your Raspberry Pi, rather than letting you first run MX interactively (see [[MX_on_Linux#Running_MX]]);
#* it is produced with a Raspberry Pi model 3, so with other models you may need to replace the Mono software included in image, and consider what updating rates your model can support;
#* it is produced using a GB locale and GB time-zone, so that automatically started MX may not initially be running correctly for your locale (however as you need to restart MX after changing settings you should be okay when MX starts recording data).
 
: Please note, this image may not always contain latest MX release, but once you have installed a release of MX on your Raspberry Pi, [[Updating MX to new version|upgrading]] is done using main software download link, as none of the other components in the image need to be upgraded (unless the release announcement states that the new release needs a specific version of Mono).
Obviously, there are other web servers, some come on their own, others come as a package.
 
I will give a few examples, but there are many more, each has its supporters, but also each has people who criticise it. You can search on-line and find both all the (good and bad) comments, and clear instructions.
 
==Older releases==
This Wiki page will only describe one installation (on a Raspberry Pi) in any detail, otherwise it will become far too complex.
 
It is always possible that a new release introduces a bug, and you decide to regress to an earlier release.
=== Install a web server ===
 
Find link to earlier MX release distributions [https://github.com/cumulusmx/CumulusMX/releases here].
I will first mention 3 possible web servers than can run on a Microsoft Windows personal computer (or server box):
#You might choose '''Apache 2''' as it is probably the most comprehensive, so if you have enough space on your PC, install it following instructions at the official site [https://httpd.apache.org/download.cgi] (or one of many other web sites that give instructions for installing apache2 on Microsoft Windows).
#You might choose ‘’’Uniform Server’’’, which is extremely simple as it comes as a package including PHP script, and MariaDB database in one package from [http://www.uniformserver.com/], although you may need to update to a newer release of PHP from [https://windows.php.net/download/] and a newer phpMyAdmin from [https://www.phpmyadmin.net/downloads/] as Uniform Server tends to issue upgrades very rarely.
#You might choose to install the Abyss web server from [aprelium.com]
 
(You can search using the history tab on this Wiki page to see which past builds were made available as Raspberry Pi images, as these are not created for all releases, but there is no way to tell if the old filenames can still be downloaded. Generally, it is easiest to download a different normal MX release distribution over whatever MX release was in the image you downloaded.)
If you get components separately, then there is extra work to get them to work together, but you will find clear explanations on download sites. Generally, if you have an all-in-one package, it seems easier; and normally upgrading one component is also easy (providing its a minor version upgrade, it might be more complex with a major version upgrade).
 
= Other utilities =
I will next mention 3 possible web servers than can run on a Linux computer:
#You might choose '''nginx''', as it is a quite popular web server for small computers (like the RPi), install this web server with <code>sudo apt-get install nginx -y</code>.
#You might choose '''Apache 2''' as it is probably the most comprehensive, so if you have enough space, install it using <code>sudo apt install apache2 -y</code>.
#*You might want to add Fast CGI to your apache server (this is a way of interfacing interactive programmes with a web server, but if you don’t have any idea what that might mean, skip this)
#* To add this use <code>sudo apt install libapache2-mod-fcgid</code>.
#You might choose '''lighttpd''', as a web server that is designed to use as little space on your Linux computer as possible, install it with <code>sudo apt-get install lighttpd -y</code>.
 
==By Mark Crossley==
For a Raspberry Pi, installing two of these is covered at [https://www.raspberrypi.org/documentation/remote-access/web-server/]
 
The following download zips do not include the ''*.dll'' files ("software libraries") needed to run the respective utility, these are provided in the MX release zip that is stated alongside download link. The different utilities vary with regard to which software libraries they require, and those with technical understanding can check the code sources.
=== Install PHP Hypertext Pre-processor ===
 
If you are using an earlier MX release than that stated, upgrade MX to bring in the new "*.dll" files needed, because the utility will not work with the versions of "software libraries" included with earlier MX releases.
As already mentioned, some web servers come in a package that includes this script language. Again, as mentioned before, such packages may be rarely upgraded so the included PHP might be an old version with bugs or security concerns.
 
If you are using a later MX release than that stated, check [https://cumulus.hosiene.co.uk/viewtopic.php?t=17887 the MX release announcements] for all releases since that stated to see if the "software libraries" have been upgraded, if they have then the respective utility because it has been compiled with older versions of the respective .dll files will probably not work. (Sometimes the names of new .dll files are different to the names of the old .dll files, so an upgrade of MX done by copying files over an existing installation will leave the old files there, in that case the utility might still work by picking up old names).
At time of updating this Wiki page, PHP 8.1 is stable version, PHP 8.2 is available to test, and PHP 9.0 is planned. Older versions 7.x, and 5.x are largely obsolete, although some web providers may still make them available.
 
=== Create Records ===
* PHP is not the only script language available, but it is quite comprehensive
** PHP can be used either in a fairly simple way by non-technical people
** or in an object-oriented way for those more technical to achieve success with more complex scripts.
* To check which PHP modules have been installed by any command, type <code>php -m</code>.
* To test that php is installed, type <code>php -v</code> into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
* If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least '''mysqli''' and '''mbstring'''.
 
This utility is not yet available.
 
A utility to create, or recreate, missing [[Correcting_Extremes|extreme records]] from Cumulus MX by parsing your [[dayfile.txt|daily summary (dayfile.txt)]].
 
=== Create Missing ===
 
~ The CreateMissing.exe ''v1.2.1'' download dated 20 May 2022 (works with release of MX 3.14.0 onwards) [https://github.com/cumulusmx/CreateMissing/releases/download/v1.2.1/CreateMissing-v1.2.1.zip is here].
====Microsoft Windows - download/install PHP====
 
Check compatibility with the MX release you have running as per notes above. (Although earlier releases of "Create Missing" exist on "github.com/cumulusmx/CreateMissing/releases", their software bugs imply they should not be used). Install into same directory as "CumulusMX.exe".
Straight forward, just down load a zip from from [https://windows.php.net/download/]. You will want a “multi-threaded” version to use with a web server, as several PHP web pages may need to run at the same time.
 
This MX utility, affects [[Category:Files with Comma Separated Values|Cumulus CSV files]]:
*PHP is not the only script language available, but it is quite comprehensive
# It will create a new [[dayfile.txt|daily summary (dayfile.txt)]]. If you already have a "dayfile.txt", that is renamed, the new file should include individual fields, and lines, missing from your old file.
**being able to be used either in a fairly simple way by non-technical people
# It will read, and where appropriate update, your [[Standard log files|month by month (MMMyylog.txt)]] files.
**or in an object-oriented way for those more technical to achieve success with more complex scripts.
#* The utility reads the [[Calculate_Missing_Values#Source_value|source values]] in as input values. This utility has the ability to calculate many of the [[Calculate_Missing_Values#Derived_value|derived fields]], so will add those derived values to your month by month files as it proceeds.
*To check which PHP modules have been installed by the above command, type <code>php -m</code>.
#* Therefore the utility should be used to add the [[Standard_log_files#Number_of_fields_per_line_varies_by_release|extra fields]] for those months where the input file was produced by Cumulus 1 or an earlier release of MX (which might have populated fewer fields or used a [[Feels_Like#How_to_express_Feels_Like_for_highest_temperatures|different calculation]]).
*If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least '''mysqli''' and '''mbstring'''.
 
 
Usage information by the author of the utility can be [https://github.com/cumulusmx/CreateMissing/blob/master/README.md found here.]
To test that php is installed, type <code>php -v</code> into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
* Learn more about this utility by reading the [[Calculate_Missing_Values#CreateMissing.exe|calculate missing]] page in this Wiki.
====Linux operating systems - best repository====
** Elsewhere on that same page learn when this utility might be useful, and what else you can try.
* Check in the support forum for information about status of bugs in this utility, at time of last updating this paragraph there was an outstanding issue concerning storing the line in dayfile.txt for the first of some months if your rollover time is not midnight.
* For full information about solving problems with your '''dayfile.txt''' file read the [[Amending dayfile|amending daily summary file]] page.
 
=== Export ''To'' MySQL ===
The default repositories (see [[Preparing_your_Linux_computer_for_MX#The_various_components_to_commands_for_installation|here]]) may not contain up-to-date versions of all software, or may contain some software that is not compatible with other software.
 
~ The ExportToMySQL.exe ''v1.4.1'' download dated 21 June 2022 (works with release of MX 3.18.0) [https://github.com/cumulusmx/ExportToMySQL/releases/download/v1.4.1/ExportToMySQL-v1.4.1.zip is here.]
You can add another repository that generally has reasonably new PHP versions (a few weeks behind the official [https://www.php.net/downloads.php] site) using
<pre>sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update</pre>
 
Check compatibility with the MX release you have running as per notes above. Note that compatibility is needed both with the .dll files and with the columns present in the table to be updated.
 
If you are not using the MX release named above, then other release zips (and source code) for this utility are available at [https://github.com/cumulusmx/ExportToMySQL/releases ExportToMySQL/releases] and each release there is labelled as to which MX release creates default tables with correct columns.
* On Linux, the simplest instruction to install it is <code>sudo apt install php -y</code>, although in practice this will not install all the PHP modules you actually need
** A warning, which version of PHP that command gives you depends on which php version is found first in the [[MX_on_Linux#The_various_components_to_commands_for_installation|respositories]] your computer has in its source list, so you may end up with the oldest!
* A better variation is to actually list the version we want:
<code>sudo apt install php''V.x''</code> (where V is major version, and x is minor version e.g. 7.4 or 8.1)
** This may still not give us the modules we want, but it does control which version is obtained
* Therefore the best variation is to list both the modules we want, and the version we want (here "x" denotes minor version, adjust to what is latest stable (e.g. change to php8.1-common etc.):
<code>sudo apt install -y php8.x-common php8.x-cli php8.x-json php8.x-xml php8.x-readline php8.x-mbstring php8.xmysql php8.x-curl php8.x-gd pdp8.x-psqlite php8.x-pdo -y</code>
* The Linux package manager will ensure that any components, that are required by dependencies, but not listed specifically, are also installed
** Loading from the "packages.sury.org/php" repository has one disadvantage, the dependencies there don't automatically integrate the PHP with your web server (if you are using apache 2 for web server, add "libapache2-mod-php8.x" to list);
 
This utility reads [[MySqlConnect|Settings]] that are also used by "CumulusMX.exe". Parameters supplied to this utility enable it to do ''one of the following actions'' affecting a [[Your Own Server|MySQL, or MariaDB, database server]]:
====Linux operating systems - standard repository====
# The parameter used is "dayfile", will update a database table that is called "dayfile" (by default, the user can change the name of that table) that (as minimum) has columns named as per the MX default for that table (at CumulusMX.exe release quoted for which the download applies) from the contents of [[dayfile.txt|daily summary (dayfile.txt)]] file
# Update a database table that is called "Monthly" (by default, the user can change the name of that table) that (as minimum) has columns named as per the MX default for that table (at CumulusMX.exe release quoted for which the download applies) from the contents of the single file of [[Standard log files|month by month (MMMyylog.txt)]] whose relative path "data/..." is named in the parameter
# The parameter used is "monthy", will update a database table that is called "Monthly" (by default, the user can change the name of that table) that (as minimum) has columns named as per the MX default for that table (at CumulusMX.exe release quoted for which the download applies) from the contents of all files of type [[Standard log files|month by month (MMMyylog.txt)]] found in the [[Data folder|data sub-folder]].
 
Installation and usage information by the author of the utility can be [https://github.com/cumulusmx/ExportToMySQL/blob/main/README.md found here.]
* On Linux, the simplest instruction to install PHP is <code>sudo apt install php -y</code>, although in practice this will not install all the PHP modules you actually need
** A warning, which version of PHP that command gives you depends on which php version is found first in the [[MX_on_Linux#The_various_components_to_commands_for_installation|respositories]] your computer has in its source list, so you may end up with the oldest!
* A better variation is to actually list the version we want:
<code>sudo apt install php7.4</code>
** This still does not give us the modules we want, but it does control which version is obtained
*A still better variation is to list the components we want:
<code>sudo apt install php7.4 php7.4-cli php7.4-common php7.4-json php7.4-xml php7.4-opcache php7.4-readline php7.4-mbstring php7.4mysql php7.4-curl php7.4-gd php7.4-sqlite php7.4-pdo -y</code>
** Add ”libapache2-mod-php7.4” as just part of what integrates apache 2 and PHP
** The included “php7.4-cli” is what allows us to run scripts in batch
** “php7.4-common” and “php7.4-mbstring” are pre-requisites for various other modules, and the second is used by ‘’’phpMyAdmin’’’ and other tools
** “php7.4-json” (see [[Php_webtags#Option_3:_JavaScript_Object_Notation|here]] and [[:Category:JSON_Files#Data_Transfer_Format]] and “php7.4-xml” (see [[Php_webtags#Option_1:_eXtensible_Markup_Language_alternative|this section]]) are used to handle particular file types
** “php7.4mysql” integrates php and MariaSQL/MySQL
** "php7.4-sqlite php7.4-pdo" are needed to access the weather diary or recent history databases
** “php7.4-curl” “php7.4-gd” may be needed as these are used by some third-party routines designed by Cumulus contributors
* There are advantages in using default repositories:
** installing a web server and PHP from the default repositories will pick up joint dependencies and thus integrate the web server and php.
 
''This new utility replaces the "ExportMySQL.exe" (no "To" in that name) utility included by Steve Loft in his MX beta distributions.'' That is not compatible with any recent MX releases.
===Creating a Home web page on your web server===
 
==By Steve Loft==
You may wish to create a index.php web page at (Linux Apache server) /var/www/html which is the web server root for browsing, or of course you may wish to copy or FTP here all your existing web pages. To view a php web page, go into a browser and type in a url with the same IPv4 address as you use for the admin interface, omit the port (:8998) and instead type in your web page name (e.g. //192.168.1.'''xy'''/index.php, where you need to determine digits that replace '''xy''').
 
Source code for these utilities are not available, therefore they cannot be altered or upgraded.
===Finding your PHP configuration file===
 
=== Utility for Fine Offset Stations ===
For sake of simplicity in this article, from now on I will assume the web server you installed is "apache2", change that segment (in the paths quoted below here), if you installed a different web server.
 
*[//{{SERVERNAME}}/Downloads/SetLogger.zip Set Logger Interval] A utility by Steve Loft to adjust the internal logging interval on Fine Offset stations
The ''web server'' main '''php.ini''' is found at ''/etc/php/m.n/apache2/php.ini'' (where 'm.n' depends on which PHP version was installed, that was found out in last sub-section).
 
You may need to edit this file for example to specify where your include files are stored (if not in same folder as script with require/include). Replace "apache2" by "cli" in the path for the ''batch'' '''php.ini''' file that you may also need to edit.
 
=== Weather Display Converter ===
To run a php script in batch, type <code>php - f <file_name></code>. You can redirect the output by adding a greater than symbol and the destination file (i.e. > log_file) on the same terminal line.
 
This utility was provided by Steve to convert '''Weather Display''' log files to Cumulus 1 format, download [https://cumuluswiki.org/a/File:WDconverter.zip here].
If you want the MX external commands to run a PHP script for you, use something like "sh" as the program to run (i.e. run shell script); and in the parameters something like
'''/home/pi/CumulusMX/MXutils/autoEOD.sh''' will run a script "autoEOD.sh" you have added to the "MXutils" folder. In that script, you put something like (assuming you have added a folder 'batch' with a subfolder 'log'):
<pre>#!/bin/sh
# This MX batch command file is initiated automatically by Cumulus MX software during last stage of processing the end of a meteorological day
echo "It stores feedback in log file CumulusMXDailyBatch (file name ends with day of month)"
sudo php -f the_path_and_filename_goes_here.php > /home/pi/CumulusMX/batch/log/DailyBatch_Day$(date +%d).log
</pre>
 
Please read the readme file in the zip.
 
There is a [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=10118 forum thread about using this converter with the legacy Cumulus here].
 
<br>
=== Install Maria database ===
 
At time of creating this utility, the functionality offered by MX was limited, and this utility was not useful with MX. MX now has much more functionality, and depending on the release you are using, this utility may be useful. Remember this utility will not support all fields used by MX. It is advised you use [[#Create Missing]] (see above) after using this utility to populate the other fields. If you do find this utility does not work with your MX release, you may need to manually recreate records, etc. using Cumulus 1 formats, and then [[Migrating from Cumulus 1 to MX|Migrate all your files from Cumulus 1 to MX]].
MySQL database software is controlled by Oracle and not made available for inclusion in Raspberry Pi repository.
 
=== WeatherLink Converter ===
MariaDB is an alternative that has largely similar command syntax so is likely to work with code (like MX) designed to work with MySQL. Since the MX developer (Mark Crossley) actually uses MariaDB, we can have plenty of confidence it is suitable.
 
This utility was provided by Steve to convert Davis '''WeatherLink''' log files to Cumulus format, download [https://cumuluswiki.org/a/File:WLconverter.zip here].
To install this database server, we type <code>sudo apt install mariadb-server php-mysql -y</code>.
 
Note this has nothing to do with Davis ''Weather Link Live'' despite confusing similarity in name.
Our database files will be stored at (Linux) '''/var/lib/mysql''' by default. Our MariaDB configuration is stored at '''/etc/mysql/mariadb.conf.d/50-server.cnf''', and it is the ''datadir='' entry that controls where the database files are stored, so edit that to store files somewhere else, but make sure the storage location has right ownership permissions. This Wiki page will not give you the full technical details, so only those with some technical knowledge do such changes.
 
Please read the '''readme''' file in the zip.
=== Making your database secure ===
 
* There is a recent [https://cumulus.hosiene.co.uk/viewtopic.php?f=6&t=20017 forum thread about using this converter with MX here]
We need to assign passwords to control access to the database by typing into terminal <code>sudo mysql_secure_installation</code>. That brings up a screen:
* There is an older [https://cumulus.hosiene.co.uk/viewtopic.php?t=11349 forum thread about using this converter with the legacy Cumulus here].
# where we are asked to type current password for the root (as no password has yet been set, simply press Enter),
# next type '''Y''' to signify we are going to set a new password for '''root''',
# next type in a new password that you will not forget,
# next as explained on the screen we are going to say whether users must select a user name as well as a password, type '''Y''' to ''Remove anonymous users''
# next we have to decide whether we will only be logging into the database on our Pi (using Localhost) or we might be logging in remotely; type Y or N respectively, but if you choose N remember someone other than you might guess there is a root user and might guess the password you set,
# next we have another option of whether to retain or delete a test database, answer N or Y respectively, I would keep the test database for now as you can play with it and then remove it later,
# finally you type in another Y as that will '''Reload the privilege tables''' and ensure all is set up for your access to the database later.
 
== Third Party Contributions by other Cumulus Users ==
The message, when the process successfully completes, is "Thanks for using MariDB".
 
For files, and add-on's, contributed by Cumulus users, see these Categories (where third party authors may have mentioned what they offer):
# [[:Category:3rd Party Apps]] - contributions for various devices,
# [[:Category:AddOns]] - adding extra functionality,
# [[:Category:WebTools]] - for your web server, and
# [[:Category:User_Contributions]] - key contributions from the user community
 
===Getting web and database servers ready for use===
 
=Current MX Developed from =
We need to create a user for PhpMyAdmin (or adMiner or whatever) to access our database and another for Cumulus to use to access the database tables. At the moment our database access has the single '''root@localhost''' user we created when we installed MariaDB. The initial password was set then, and we need to use it to get access to MariaDB monitor where we can insert some SQL commands to create these two users.
 
Cumulus 3 (MX) software was developed as a beta with limited functionality by Steve Loft.
Most tools, like PhpMyAdmin, on first start up will ask for username (here I choose "admin") and password, thereafter the tool should use same log-in (by default you see log-in screen each time you restart or if it is left idle for a long time), let us create a user called 'admin' for it.
* Steve Loft previously developed Cumulus 2 (where he tested using the C# language, now used for MX)
* Steve Loft created the original Cumulus software (now known as legacy cumulus 1) just for his use, before making it generally available, he then developed it over the next decade to include requirements suggested by thousands of users.
* Steve Loft, the author of Cumulus/Cumulus MX has since fully retired, and stopped providing support, or updates, to any of his software.
 
==Steve's final beta MX==
The database name (here I choose "cumulus" for the data base name), user name (here I choose "weather" for the user name), and password, must all match those set in MX using the [[MX_Administrative_Interface#MySQL_settings|MySQL_settings]] in the admin interface. You could of course use PhpMyAdmin (or AdMiner) to create additional user names, and to create the database, but I assume all is done in the following bit of SQL. Remember, the Windows operating system is not case sensitive, but all Linux based operating systems are case sensitive (so whatever pattern of capitals and lowercase you choose must be used every time. Also all names must start with a letter, can contain only letters or digits (no punctuation), and must not be a reserved word ("password" is a reserved word, so you cannot use that for a password, nor for a column name).
 
You can download below the final Cumulus 3 beta release, as made available by Steve Loft, and also his ''subsequent'' final source code.
Obviously, these names might not be what you use, but you can amend commands below accordingly. For each line with SQL, it must end with a semicolon (;) as shown. After you press "Enter" key you will get a response saying "OK" if you have remembered the semi-colon. You can actually use "\G" or "\g" instead of a semi-colon, but here we will keep it simple and stick to semi-colon.
 
If you are able to understand his source code (linked below), and compare it to the latest source code (linked above), you can appreciate the transformation that has taken place in recent years.
The SQL lines have a prompt of a greater than symbol (>) while the command lines have a prompt showing current path. Note that "identified by" is followed by a password enclosed in single quotes.
 
*[//{{SERVERNAME}}/Downloads/CumulusMXDist3043.zip Cumulus MX v3.0.0 beta build 3043, 20-Jan-2017] Cumulus MX, a cross platform version of Cumulus 3 which runs on Windows, Linux (including the Raspberry Pi) and Mac OS X (Steve Loft - final release), single zip that includes both "CumulusMX.exe" and "ExportMySQL.exe".
<pre>sudo mysql --user=root --password=InitialPassword
*[//{{SERVERNAME}}/Downloads/CumulusMXSource.zip subsequent updated Cumulus MX Source Code] from Steve Loft. He made the following comments, as he handed the project over to "the community":
create user admin@localhost identified by 'PhpMyAdminPassword';
**This version of the code is post the last release I made at build 3043 as it includes some extra changes that I have not finalised.
create user weather@localhost identified by 'MXPassword';
**My source code is offered as my parting gift, completely '''without support''', in the hope that it might be useful to future developers.
grant all privileges on *.* to admin@localhost;
**My source code is very badly structured due to the 'Frankenstein' way I cobbled it together from various places. Some of it is a machine translation of parts of Cumulus 1.
grant all privileges on *.* to weather@localhost;
FLUSH PRIVILEGES;
create database cumulus;
exit;</pre>
 
= Legacy releases =
As I type this, Cumulus MX has no exception handling if the username and password defined in the settings do not exist in the database, therefore in this situation it will crash out (with message press Enter to close).
 
See [[Downloads| downloads page]] for full details.
=== Installing Adminer, or PhpMyAdmin ===
 
== Cumulus 1 ==
ExportMySQL.exe and CumulusMX.exe (see [[Cumulus_MX#Executables]] for details) both create SQL for updating tables in a MySQL database, such as the one our MariaDB software package we have installed can create.
 
Cumulus 1 software was popular, when it went public in 2008, and its [[Cumulus_Users|usage]] grew very rapidly in the subsequent decade. Even now (in mid-2022) some people are still using the final release version (and a quite substantial number of public viewable weather sites are still driven by earlier builds).
This option is about installing a package to do the tasks that go with operating a database, that MX may not be able to do in the way you require.
 
Be aware, the longer you wait before swapping from legacy to Cumulus MX software; the more that MX will have developed away from the legacy. MX's development has effectively abandoned its former focus on compatibility, so you might encounter more difficulty with [[Migrating from Cumulus 1 to MX|transferring your data files when you migrate]].
====PhpMyAdmin====
 
In all months of 2020, (when the year selection drop-down provided in all releases of original software reached its limit, and MX gained a substantial increase in functionality), and continuing into mid-2021, there was an avalanche of people who used to use the legacy software successfully swapping to MX. As of mid-2022, the usage of this legacy software is in fast decline, implying fewer people can offer assistance.
'''PhpMyAdmin''' is one tool that can be used to manipulate your MySQL like database (that is the significance of the "My" bit in the middle of the tools's name).
 
The '''installer for the FINAL release of the legacy Cumulus 1''' is available here:
You may like '''PhpMyAdmin''' as this offers:
#[//{{SERVERNAME}}/Downloads/CumulusSetup.exe Cumulus 1099 installer] Full set of files within an installer.
* a graphical approach (you see a table on screen and navigate to the row or cell you want to work on)
#* On the "this period" type, and snow index, screens you can still manually enter any year,but you have to over-type with the full year yourself, outside range 2008 to 2020.
* a SQL approach (you can try out any SQL here, before adding it to a script that you might use in a web page)
#* You cannot manually regenerate NOAA reports after 2020, but there is no problem with automatic generation.
* a selection approach (you select a database, then select a table, then select an action)
#[//{{SERVERNAME}}/Downloads/cumulus.1099.2.zip Cumulus 1.9.4 build 1099.2 patch] Patched version of one file (cumulus.exe) to replace that file from above installer
#* Note this is not the full installation package, just one file to replace in the suite of files created by the installer.
#* This patch modifies the drop-down year selectors, (on "this period" screens, for NOAA reports, and for Snow Index) making it easier to choose date entries (patch version 1099.2 was released 28 Jan 2020) up to the year 2030.
 
=== Legacy Cumulus 1 Resources ===
WINDOWS:
#Download from [https://www.phpmyadmin.net/downloads/ official download page]
#You need to do some configuration (PLEASE EXPAND THIS).
 
*[//{{SERVERNAME}}/Downloads/Cumulus%20basic%20installation%20guide.pdf Cumulus installation guide] Instructions for installing Cumulus 1
LINUX:
*[//{{SERVERNAME}}/Downloads/readme.txt Cumulus readme.txt] Please read this before installing or updating Cumulus
#The version of phpmyadmin in the default repository is not compatible with PHP7.2 and above
*[//{{SERVERNAME}}/Downloads/Cumulus.chm Cumulus help file] The Cumulus 1 help file (this also gets installed when Cumulus is installed)
# Visit the phpMyAdmin [https://www.phpmyadmin.net/downloads/ official download page] and look for the .tar.gz URL and download it using wget.
*[//{{SERVERNAME}}/Downloads/CumulusFR.chm Cumulus help file in French] The Cumulus 1 help file translated into French
# The following steps assume version 5.2.0, released in 2022 is to be downloaded. (If a later version is now available, make sure to change the commands below to match)
*[//{{SERVERNAME}}/Downloads/CumulusRealtime.zip Cumulus Realtime] An obsolete Silverlight (Silverlight is no longer developed or supported by Microsoft) application for displaying 2 extra dials showing real time wind data on the standard Cumulus gauges page (with temperature and rainfall in histogram style) for a web site
# Decide whether you just want English, or need another language
#* For English only, the wget command is <code>sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.2.0/phpMyAdmin-5.2.0-english.tar.gz</code>
#* For other languages, the wget command is <code>sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.2.0/phpMyAdmin-5.2.0-all-languages.tar.gz</code>
# Change to folder where we will extract our download, this could be <code>cd /usr/share/</code> or some other directory perhaps on an extra storage drive
#* For English only, extract your download into the current folder using <code>sudo tar xzf phpMyAdmin-5.2.0-english.tar.gz</code>
#* For other languages, extract your download into a sub-folder of the current folder using <code>sudo tar xzf phpMyAdmin-5.2.0-all-languages.tar.gz</code>
# Create a folder to hold the download within the folder where we did our download <code>sudo mkdir phpmyadmin_5.2.0</code>
# Now we will move the contents from the sub-folder created by the "tar" command into the folder we just created
#* For English only, the move command is <code>sudo mv phpMyAdmin-5.2.0-english/* ./phpmyadmin_5.2.0</code>
#* For other languages, the move command is <code>sudo mv phpMyAdmin-5.2.0-all-languages/* ./phpmyadmin_5.2.0</code>
# There is some configuration to do, for simplicity here assume the destination directory ("./phpmyadmin_5.2.0") is actually "/usr/share/phpmyadmin" and that we will use '/var/lib/' and '/etc/' to store other directories used by phpmyadmin:
#* First <code>sudo nano /usr/share/phpmyadmin/libraries/vendor_config.php</code> to edit (with nano) the vendor configuration
#** change the path shown in the existing line mentioning 'TEMP_DIR' so it becomes <code>define('TEMP_DIR', '/var/lib/phpmyadmin_5.2.0/tmp/');</code>
#** change the path shown in the existing line mentioning 'CONFIG_DIR' so it becomes <code>define('CONFIG_DIR', '/etc/phpmyadmin/');</code>
#The following sequence of commands will
#* add the mysqli module to our php install, <code>sudo phpenmod mysqli</code>
#*will restart apache (change if you are using a different web server), <code>sudo service apache2 restart</code>
#*will create a symbolic link for the phpadmin installation in the server web root so it can be seen (and used) in our browser, <code>sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin</code>, note the first path is to directory where we stored our phpmyadmin files, and the second path/file is the default web directory and the symbolic link file name, adjust these paths to what is needed for you.
#*will give the standard user (pi) ownership of the database files and the web pages: <code>sudo chown -R pi:www-data /var/www/html/</code>
# phpMyAdmin now has the files where it expects to see them, and should work
#* To log in requires a username (default "admin") and password (defined in your database set-up - see previous section)
#* The remaining configuration can be done within phpMyAdmin, follow cues on each screen as work through all configuration tabs.
 
=== Special Variants ===
 
It is '''STRONGLY RECOMMENDED''' that users of Cumulus '''1.9.4 variants''' migrate to Cumulus MX, which has ongoing support by Mark Crossley. The baud rate can be selected within MX as a configuration setting.
If you have any difficulty, there are some instructions at https://devanswers.co/manually-upgrade-phpmyadmin/ and some at https://docs.phpmyadmin.net/en/latest/setup.html#quick-install
 
*[//{{SERVERNAME}}/Downloads/CumulusSetup1100.zip Cumulus 1.9.4 build 1100 setup] '''Use ONLY if you are an Instromet user''', and your logger operates at 115200 baud, you should use build 1100
*[//{{SERVERNAME}}/Downloads/CumulusSetup1101.zip Cumulus 1.9.4 build 1101 setup] '''Use ONLY if you are an Instromet user''', and your logger operates at 19200 baud, you should use build 1101
*'''NOTE:''' '''These Cumulus 1.9.4 variants will not have any updates in the future.'''
* The patch for Cumulus.exe, cannot be used with these variants
* The menu items with year drop-down selectors will not display years beyond 2020.
**For example, this will happen on manual generation of NOAA-style reports.
**This does not affect any automatic generation of NOAA reports, but does prevent re-generation.
 
====adminer== Cumulus 2 ==
 
The software for Cumulus 2 alpha release is no longer available.
You may prefer to install a different tool, perhaps '''adminer''' that works using a drill down approach. A drill down approach is when you select the database, then select the table, then select the row, then select the column, then select the action. This logical step by step approach is a popular approach, but does not suit everyone.
 
Install this drill down package with <code>sudo apt install adminer</code> (I leave you to work out the commands needed after that).
 
 
===Viewing web pages on our new web server===
 
You can view any index.php or PHPMyAdmin web page in your browser by prefixing the address with your Pi IPv4 address e.g. a URL like '''http://192.168.1.xyz/phpmyadmin''' where ''xyz'' is 2 or 3 digits you look up as mentioned before where FTP was described. If your Pi has both Ethernet and wireless connection, there will be two different values of ''xyz'' for you to choose one from. On first run of PhpMyAdmin, as already mentioned, you will see a '''PHP MyAdmin log-on page''' where you type username and password we have just set up.
 
===Populating your database tables on your Pi===
 
Cumulus MX has functionality to update database tables at one of several intervals:
# fixed SQL at real-time
# fixed SQL at standard interval
# fixed or custom SQL at end of day
# custom number of seconds
# custom number of minutes
 
MX now has ability to run SQL updates when you edit a log file, provided you use the predefined column and table names. MX also has the ability to store failed SQL and to try to run it again at the next rollover. This will fail again and again until MX is stopped, if the SQL is specified wrongly, but is designed to cope with correct SQL, that fails because the database server is down, and assumes the database server may be working again by next rollover.
 
The database tables can use the column names in the schema pre-defined by Cumulus MX for the predefined queries, or in a custom schema (where you specify the column names) if you specify the required SQL. The settings are all in [[Cumulus.ini#Optional_Structured_Query_Language_Settings|SQL settings]], so read that cross-reference to find out more.
 
You might have started using MX before you set up your database. There is a option in that part of the admin interface to create database tables (as required) for each of the first 3 updating intervals. For example, the default name for the table updated at the standard interval is "monthly", but you could give it a name of "standard" or whatever you like.
 
There is an optional utility that can be installed. Type <code>cd CumulusMX && sudo mono ExportToMySQL.exe parameter</code> to run the executable. The mandatory parameter specifies the log file name to read (dayfile.txt or standard monthly log file).
# for each line in the log file the executable will try to insert a row in the database table
#*the SQL syntax used is "insert ignore", so if the row for that log file line already exists in the table, it will skip onto next line of log file.
 
There is no way to use this executable for insert of past rows into the real-time table.
 
If you have been running Cumulus on another device previously and already have database tables, the next section explains how you can create SQL to export your old database and use that SQL to populate the new table.
 
===Transferring database tables to your Pi===
 
If you have been using Cumulus before (and already have a database) then you can use PhpMyAdmin on your old device to '''export''' out all the '''Cumulus tables as SQL''' in a zip file, FTP that zip file across to your Pi, then use PhpMyAdmin on your Pi to '''import''' that zip file.
 
Providing you selected the right options for what SQL you created in your export, the import will contain SQL to create the tables and to insert all the rows into each Cumulus table on your Pi. Please note that there is a limit of around 1000 rows that can be imported in one action, so for your bigger tables, you will only create the table once, but you will do several '''Replace''' row actions (export from old device, then import to Pi) each transferring just one thousand rows, until the whole table is on the Pi. You may prefer to use "ExportMySQL.exe" as described in previous section to recreate your bigger tables from the Cumulus log files.
 
You might want to also '''export/import the tables in the PhpMyAdmin database''' (as they contain your preferences for each of your tables) in a separate zip, although these might need some extra transformations, as they are specific to a particular version of the database server (and the old database server version may not match the MariaDB version on your Pi). Some PhpMyAdmin tables do change for different versions of the tool, so that too may make export/import of its tables more complicated.
 
=== Restarting Web Server ===
 
After all these installs, we need to restart your web server (so it loads the PHP and MariaDB), if you have installed apache 2 do this by typing <code>sudo systemctl restart apache2</code> or (if we just want the Apache configuration reloaded) by typing <code>sudo systemctl reload apache2</code>. Similar commands apply for other web servers.
 
We will need to restart Apache (or whichever web server we installed) any time we change our php.ini files, database passwords, and anything else that is checked when the web server starts.
 
 
[[Category:WebTips]]
Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/10583...10628"