2,032
edits
CumulusUtils: Difference between revisions
m
→MAC: Using crontab (advised)
| (45 intermediate revisions by the same user not shown) | |||
|
== Introduction ==
This is the first page of the CumulusUtils Wiki. From here you should be able to find everything you need to know for this tool.
If needed you can always go to the CumulusUtils Category page (see bottom of this page).
== Prerequisites ==
This paragraph describes what is required to use CumuluUtils (To be modified/extended by user experience).
#A working CumulusMX environment on Windows, Linux (RPi) or MacOS
#For the ''[[Charts_-_Misc_charts|Miscellaneous Charts]]'' more data - a year starting on jan 1 - is required to make these useful
#From CMX4 and CMX5 change in the handling of datetime in the logs. Corresponding CUtils versions are available. It is advised to upgrade to the last version of CMX/CUtils as fast as possible because of support.
#For longer data series it will be useful to run the [[Software#Create_Missing|CMX supporting application ''CreateMissing.exe'']] beforehand.
== Goals ==
The goal of ''CumulusUtils'' (abbr: ''Cutils'' or ''CUtils'') is to facilitate website creation for users of CumulusMX without exposure to coding in e.g. PHP or javascript. Having a meteo-website should not be privileged to users with great IT-skills. Making charts should not be demanding for programming skills in SQL or making it otherwise impossible to create charts without diving deep into the technique involved in CumulusMX (which actually is also the case with SQL). Note that this does not mean you can't use IT skills: the ''user configurable menu'' offers an opening to expand as far as you wish.
''CumulusUtils'' positions itself as a configurable application for which the understanding of how to run it and how to configure it is the most important thing to know. For configuration ''CumulusUtils'' uses [[Cumulusutils.ini|''cumulusutils.ini'']] file which resides in the CumulusMX directory. If that file does not exist, it will be created. The same holds for the CUstringsXX.ini file (containing your translations for CUtils text strings). All other files related to ''CumulusUtils''
The charts are created using [https://www.highcharts.com/ HighCharts] using their [https://shop.
== Assumptions ==
== Installation ==
# ''CumulusUtils'' is available in a distribution which can be downloaded from the [https://cumulus.hosiene.co.uk/viewtopic.php?f=44&t=17998 forum]. In the CumulusMX directory the user must create a directory ''utils'' (case dependent) and must copy the files of the distribution (including the subdirectories) to that directory.
# Rename ''CutilsCharts-default-for-use.def'' to ''CutilsCharts.def''
# Then, on the domain for the website, the user needs to make two directories in the webroot: 'lib' and 'css'.
# Updating ''CumulusUtils'' is done by completely overwriting the contents of the installation directory utils as if it were an installation. If you want to save the old installation, make a copy of the utils directory.
# The language setting of CumulusUtils is for display/language handling only and does not affect the reading of the data in any way apart for the monthly log file name.
# From CMX version 4, the logfiles are written as invariant and the above striked through note has become invalid.
# From CMX version 4.7 (.NET 10; later rename version 5.0) there are changes in the database which require CMX to run before CUtils 8.2.0 can run.
When installed you are ready to run. The first run after an install or an update MUST be without [[Thrifty - Cutils Command Qualifier|Thrifty]].
|--- lib (JavaScript libraries, uploaded by CumulusUtils during a run)
|--- css (css files, uploaded by CumulusUtils during a run)
=== Updating CumulusUtils ===
Beside the libraries the distribution contains:
#CUserAbout-example.txt : as an example of the user about file - when used rename to ''CUserAbout.txt''
#
#CutilsCharts-examples.def : not intended for direct use, contains some more elaborate examples of the [[ChartsCompiler]] possibilities
To avoid confusion it is left to the user to edit and maintain the files for use either on the website or on the CumulusMX machine.
When creating website the file ''CutilsHead.def'' may have importance to you. See [[Website Generator|Website]]
=== Manually installing without FTP ===
NOTE: Not using
All required installation and generated files are present in the utils directory and copied with ''FTP'' or ''PHP upload'' to the server during a run of CumulusUtils.
If you don't use ''FTP'' or ''PHP upload'' you will have to copy manually.
The PHP upload is by far preferred.
=== Files involved ===
| index.html || <webroot>
|-
|
|-
|
|-
| HighchartsDefaults.js || <webroot>/lib
| suncalc.js || <webroot>/lib
|-
|
|-
|
|-
|
|-
|
|-
|
|-
! Generated File !! Destination !! Remark
|-
| extrasensorsdata.json || <webroot>
|-
| customlogsRecentdata.json || <webroot>
|-
| customlogsDailydata.json || <webroot>
|-
| CUserdataRECENT.json || <webroot>
|-
| extrasensorsrealtime.txt || <webroot> || When Extra Sensor devices are configured [2]
|-
| customlogsrealtime.txt || <webroot> || When Extra Sensor devices are configured [2]
|-
| meteocamrealtime.txt || <webroot> || When Extra Sensor devices are configured [2]
|}
[1] : These files may be served by CMX in another directory (just like the CMX JSON data files). In that case the relative location is specified in the inifile parameter ''CumulusRealTimeLocation''.<br/>
== Running CumulusUtils ==
CumulusUtils is - up to v6.x.y - a [https://www.mono-project.com/ ''mono''] executable.
From v7.0.0 and up it runs under .NET 8 [https://learn.microsoft.com/en-gb/dotnet/core/install/linux-scripted-manual#scripted-install ''Follow this link when on Linux''].
From v8.2.0 and up it runs under .NET 10 [https://learn.microsoft.com/en-gb/dotnet/core/install/linux-scripted-manual#scripted-install ''Follow this link when on Linux''] (and this has changes to the database which require CMX to run before CUtils can successfully run).
All environments are very different and not interchangeable. However running CumulusUtils is pretty similar on all environments and mono and dotnet (8 and 10) can co-exist on the same machine. See the CMX installation on how to install either mono or dotnet. CumulusUtils assumes the correct installation of either environment.
''CumulusUtils'' runs on any operating system CumulusMX runs on and it '''MUST''' run in the CumulusMX directory (as ''working directory'').
Running ''CumulusUtils'' is done from the commandline in a command window (under any OS) or from a script.
''CumulusUtils'' takes one or more commandline parameters (in short: commands) and must be like (see NOTE 1 below; square brackets means ''optional'' so DON'T TYPE THEM):
''dotnet utils/bin/cumulusutils.dll [command]'' (the ''.NET'' syntax - ''dotnet'' is obligatory on the commandline for UNIX. Note the '''dll''')
For UNIX: Note you will have to carry this on in scripts as well. Using the dotnet command in crontab requires using the explicit path to the dotnet command or defining the PATH environment variable in the crontab script which does not know the environment (a bit awkward).
If no commands are given the application responds with:
[ [[pwsFWI]] ][ [[Records - Top10|Top10]] ][ [[Graphs]] ][ [[Yadr]] ][ [[Records]] ]
[ [[NOAA]] ][ [[Records - DayRecords|DayRecords]] ][ [[AirLink]] ][ [[UserAskedData]] ]
[ [[ChartsCompiler|CompileOnly]] ] [ [[Extra Sensors|ExtraSensors]] ] [ [[Custom Logs|CustomLogs]] ]
| [ [[Thrifty - Cutils Command Qualifier|Thrifty]] ]
OR (in case you use the website generator):
Commands to CumulusUtils - the module names - are case independent but the author prefers [https://en.wikipedia.org/wiki/Camel_case Upper Camel Case] (or Pascal Case).
Copying to the webroot can also be done automatically through the FTP account of the user, much like ''CumulusMX'' itself copies e.g. the realtime.txt or the datafiles to the webroot. The FTP account and domain used are the same as present in the [[Cumulus.ini]] configuration. The PHP upload as exists in CMX can also be used with CUtils and it works exactly the same.
=== Considerations with CMX configuration ===
Please note that CumulusUtils requires live data from CumulusMX to display what you expect. To accomplish that you need to configure CMX as follows:
The things it is looking for are realtime.txt and realtimegauges.txt. Those are sent by CMX to the directory you configure in ''Settings=>Internet Settings=>Web/
In ''Settings=>
In ''Settings=>
In ''Settings=>
''availabledata.json'', ''dailyrain.json'', ''dailytemp.json'', ''sunhours.json'', ''airquality.json'', ''extra*.json'', ''soil*.json'', ''user*.json'',''co2*.json'' and ''leaf*.json''
In ''Settings=>
In ''Settings=>
In ''Settings=>Internet Settings=>Interval Configuration=>Real time Interval Settings'': Tick both ''realtime.txt'' and ''realtimegauges.txt'' for Upload, use ''Create Local'' only when needed.
In ''Settings=>Web/Upload Site=>General Settings'': Tick ''UTF-8 encoding''
For the ''[[ChartsCompiler]]'' (you may be more selective later when understanding what and how):
After selecting the required tables you need to select the variables they may contain:
'''NOTE: You may disable some fields later when you are more acquainted with the system. Disabling tables and field can be especially useful when you are worried about size of transfer and provider limits.'''
You may want to read about and understand the [[Website_Generator#CumulusRealTimeLocation|''CumulusRealTimeLocation'']]
Running CumulusUtils results in output which represents a static view of the data in a table or graphic format for display on your website (it may even create a complete website). The fact that the output is a static view requires the output to be regenerated when new data is available. Rerunning CumulusUtils is also required when you change anything in the configuration and can't wait to see that change reflected on the site. In general: changes in data and configuration require a rerun.
Reruns for changes of data are typically once
There are some exceptions to the daily rule: Sysinfo and [[UserAskedData]]
Automating the run of CumulusUtils on the Raspberry Pi is typically done through [https://man7.org/linux/man-pages/man5/crontab.5.html ''crontab'']. Below is the crontab of the author of CumulusUtils to be used as an '''example'''!
===== For dotnet =====
1-51/10 * * * * cd /home/CumulusMX; /usr/share/dotnet/dotnet utils/bin/cumulusutils.dll sysinfo UserAskedData UserReports
==== Windows: Using the scheduler ====
===== How to create a scheduled task =====
Prior to doing this we need to also create the script that the task will use.
Open notepad or your favourite text editor (preferred use: Notepad++).
You can copy paste the below and save as cumulusutils.bat
(It can be a filename of choice but must have extension of .bat, if using windows notepad, underneath the File name: option, will need to set Save as type: option to All Files (*.*). This way when the File name: is set as cumulusutils.bat it will save it as that and not cumulusutils.bat.txt)
====== Batch file code of cumulusutils.bat ======
'''''For v6 and below (running under mono):'''''<br/><br/>
REM This batch file script checks for the process CumulusMX.exe is running, if yes, then runs the "cumulusutils.exe", otherwise exit</br>
REM Set the current working session to the folder C:\CumulusMX</br>
cd\</br>
cd C:\CumulusMX</br>
REM Check for CumulusMX.exe is running, if yes goto :runutils, otherwise goto :end</br>
tasklist /FI "IMAGENAME eq CumulusMX.exe" 2>NUL | find /I /N "CumulusMX.exe">NUL</br>
if "%ERRORLEVEL%"=="0" GOTO runutils</br>
GOTO end</br>
REM Set as working directory C:\CumulusMX and run cumulusutils.exe in folder C:\CumulusMX\utils\bin, with the desired command or commands as per the wiki</br>
:runutils</br>
START /D C:\CumulusMX C:\CumulusMX\utils\bin\cumulusutils.exe Website</br>
:end</br>
exit</br>
NOTE: The command shown here is ''Website''. The user can put any command he likes in here (with or without ''Thrifty'') depending on what he wants to achieve. Possibly he has to create multiple tasks (e.g. UserAskedData specifically runs on a higher frequency than website). These commands then require their own batch files.
'''''For v7 and up (running under dotnet):'''''<br/><br/>
REM This batch file script checks for the process CumulusMX.exe is running, if yes, then runs the "cumulusutils.exe", otherwise exit<br/>
REM Set the current working session to the folder C:\CumulusMX<br/>
cd\<br/>
cd C:\CumulusMX<br/>
REM Check for CumulusMX.exe is running, if yes goto :runutils, otherwise goto :end<br/>
tasklist /FI "IMAGENAME eq CumulusMX.exe" 2>NUL | find /I /N "CumulusMX.exe">NUL<br/>
if "%ERRORLEVEL%"=="0" GOTO runutils<br/>
GOTO end<br/>
REM Set as working directory C:\CumulusMX and run cumulusutils.exe in folder C:\CumulusMX\utils\bin, with the desired command or commands as per the wiki<br/>
:runutils<br/>
START /D C:\CumulusMX dotnet C:\CumulusMX\utils\bin\cumulusutils.dll Website<br/>
:end<br/>
exit<br/>
===== Task creation process =====
Within the Task Scheduler Library
#Create “New Task…” (actions pane), or right click middle pane and choose “Create New Task” (not Create Basic Task).
##On the General tab of the new task (General information about the task).
##Set desired name for the Task (Example name CumulusUtils, can also set description if desired).
##Click “Change User or Group…”.
##Type in here the word system (click on “Check Names” if desired, to confirm it is correct and it should then display as SYSTEM in capitals).
##Click OK to use this user account.
##Note in Security options part of the new task creation process it should now state under. When running the task, use the following user account: NT AUTHORITY\SYSTEM <br/>(Convenient reason for this is that when the task runs, it runs is running as system account and so is not seen popping up on screen when it is running.)
##“Configure for:” not required to be changed, can be changed to version of operating system.
#On the Triggers tab of the new task (Setting when and how often the task runs).
##Click on “New”, to create a new launch time and frequency of the task.
##Begin the task: option should default to “On a schedule”, this is the desired option.
##Under the settings section choose “Daily” option.
##Start: date can be left as is, set the desired start time of the task. Set to “12:00:15 am”.</br>(Reason for this setting with 15 second delay is trying not to compete with CumulusMX upload tasks).
##In the advanced part, enable Repeat task every: and choose desired interval.</br>With the option for a duration of: default setting of “1 day” is used.
##Set Stop task if it runs longer than: to “30 minutes” (if the task runs longer than the time set, it is stopped. This applies to the individually triggered task, and not the overall routine of this scheduled task, with it’s repeating trigger times).
##Confirm that the Enabled options is ticked to make active this trigger routine.
#On the Actions tab (Setting what the task is to do).
##Click on “New…” to create a new action.
##Action: should default to “Start a program” this is correct.
##In the Program/Script: part, type the path to the program to run, or click “Browse” to navigate and select the program to run. (example C:\CumulusMX\batch_files\cumulusutils.bat).
#On the Conditions tab (setting other conditions).
##Under Power, untick Start the task only if the computer us on AC power</br>(This then enables the task to start and run even if on battery. It will automatically untick Stop if the computers switches to batter power.)
#On the Settings tab (additional other settings for the task).
##Confirm ticked for Allow task to be run on demand.
##By default Stop the task if it runs longer than: is already ticket and set as 3 days.Change to 1 hour (the behaviour here is the same as instruction 3.f.)
##Confirm ticked for If the running ask does not end when requested, for it to stop.
##Default setting of “Do not start a new instance” Is OK.
#Click on OK down the bottom to complete and commit the creation of this new task.
===== Customisation =====
One can also customise/fine tune the task further by exporting it, default export format is XML. Then can edit in favourite text editor and reimport when done. May be useful in particular for fine tuning the repetition times of the task.
==== MAC: Using crontab (advised) ====
Automating the run of CumulusUtils on the MAC is typically done through [https://share.google/aimode/CMq6jMIq3FvXLT8bn ''crontab'']. The link is a Google Gemini shortcut which offers more batch options on the MAC. The advise is to use ''crontab'' to have support available. Below is the crontab on RPi of the author of CumulusUtils to be used as an '''example'''!
===== For dotnet =====
05 1 * * * cd /home/CumulusMX; /usr/share/dotnet/dotnet utils/bin/cumulusutils.dll thrifty website
1-51/10 * * * * cd /home/CumulusMX; /usr/share/dotnet/dotnet utils/bin/cumulusutils.dll sysinfo UserAskedData UserReports
== Additional remarks ==
#CumulusUtils is written in C#, HTML and javascript. All HTML and javascript is embedded in the C# code and generated on execution of ''CumulusUtils''. The cumuluscharts.txt charts library was based on the cumuluscharts.js originally distributed with ''CumulusMX'', probably the version a bit before April 2020 and has been modified since. With the appearance of the [[ChartsCompiler]] the name cumuluscharts.txt has been preserved but the technique of creating the charts has been changed.
#As CumulusUtils itself runs in the Cumulus directory, the data directory (data) is one level below. CumulusUtils currently uses - a copy of - dayfile.txt, the monthly logfiles and incidentally the alltime.ini file. It also takes information from Cumulus.ini. All output is written to the utils/ subdirectory. All logs are created in the utils/utilslog/ subdirectory. It also uses the AirLink.log and the ExtraLog files in a similar way when the user has such devices.
#CumulusUtils can output for many languages. See the specific page on [[Language in CumulusUtils|language in CumulusUtils]].
#Development of CumulusUtils started in 9<sup>th</sup> of August 2019 with an initial commit of the Top10 written in C. Development changed rapidly to C# with an initial commit of Top10 on November 19 2019 of version 0.7.0. The rest of the history can be read in the [[Release Notes|release notes]].
#
#In case of problems there exists [[Errorlogging]].
'''NOTE''': comments to this series of articles is welcome.
[[Category:WebTools]][[Category:User Contributions]][[Category:CumulusUtils]]
| |||