Dear Inigoflores, on windows i started gettin a division by zeror erro. Do you have any idea why? It was working some weeks ago.

Using logs in folder processlogs\logs
Warning: Division by zero in processlogs\processlogs.php on line 168

General Witnesses Overview  
----------------------------------
Total witnesses                   =     1   (INF/hour)
Succesfully delivered             =     0     (0%)
Failed                            =     1   (100%) 
  ├── Max retry    =    0     (0%) 
  └── Crash/reboot =    1   (100%) 

Max Retry Failure Reasons 
----------------------------------
Timeout                           =     0     (0%) 
Not Found                         =     0     (0%) 
Other challenger issues           =     0     (0%) 

Challengers 
----------------------------------
Not Relayed                       =     0     (0%) 
Relayed                           =     0     (0%) 
Unknown (Probably Not Relayed)    =     1   (100%) 

console.log

maybe you can take some additional info from it to enhance yours :)

#!/bin/bash
#!/bin/sh
#
# spot for varaiables that will be needed
miner_vm_name=$(balena ps | egrep "miner_" | awk '{print $NF}')
miner_animal_name=$(balena exec $miner_vm_name miner info name)
get_console_log=$(find /mnt/data -name "console.log")
total_witnesses=$(cat $get_console_log | egrep -w '@miner_onion_server:decrypt:' | grep -c ':')
successful_witnesses=$(cat $get_console_log | grep -c 'successfully sent witness to challenger')
failedtodial_witnesses=$(cat $get_console_log | grep -c 'failed to dial challenger')
resending_witnesses=$(cat $get_console_log | egrep -w '@miner_onion_server:send_witness:' | grep -c 're-sending')
sending_witnesses=$(cat $get_console_log | egrep -w '@miner_onion_server:send_witness:' | grep -c 'sending')
failedtosendresend_witnesses=$(cat $get_console_log | grep -c 'failed to send witness, max retry')
relaytransported_total=$(cat $get_console_log | egrep -w '@libp2p_transport_relay:connect_to:' | grep -c ':')
challenger_notfound=$(cat $get_console_log | egrep 'not_found' | grep -c 'failed to dial challenger')
challenger_timeout=$(cat $get_console_log | egrep 'timeout' | grep -c 'failed to dial challenger')
challenger_refused=$(cat $get_console_log | egrep 'econnrefused' | grep -c 'failed to dial challenger')
challenger_unreachable=$(cat $get_console_log | egrep 'ehostunreach' | grep -c 'failed to dial challenger')
challenger_nolistenaddr=$(cat $get_console_log | egrep 'no_listen_addr' | grep -c 'failed to dial challenger') 
#
#
echo "****************************************************************************"
echo "Performing actions on Node:"
echo "          VM: $miner_vm_name"
echo " Animal Name: $miner_animal_name"
echo "Log Location: $get_console_log"
echo "****************************************************************************"
echo " "
#
if [ $1 == "p2p-status" ]; then
	balena exec $miner_vm_name miner info p2p_status
elif [ $1 == "gossip-peers" ]; then
	balena exec $miner_vm_name miner peer gossip_peers
elif [ $1 == "peer-refresh" ]; then
	balena exec $miner_vm_name miner peer refresh
elif [ $1 == "relay-reset"  ]; then
	balena exec $miner_vm_name miner peer relay_reset
elif [ $1 == "deamon-restart" ]; then
	balena exec $miner_vm_name miner restart
	echo 'wait 30 seconds before running any further HMC commands to allow the miner deamon services to finish loading'
elif [ $1 == "vm-restart" ]; then
	balena exec $miner_vm_name miner reboot
	echo 'wait 1 minute before running any further HMC commands to allow the miner VM to boot up again and deamon services to finish loading'
elif [ $1 == "log-analyzer" ]; then
	echo '******************************************************************'
	echo 'Total Witnessed:                                    = '$total_witnesses
	echo '               |-- Sending:                         = '$(($sending_witnesses-$resending_witnesses))
	echo '               |-- Resending:                       = '$resending_witnesses
	echo 'Successful:                                         = '$successful_witnesses ' (' $(($successful_witnesses*100/$total_witnesses))'%)'
	echo 'Unreachable:                                        = '$failedtodial_witnesses ' (' $(($failedtodial_witnesses*100/$total_witnesses))'%)'
	echo 'Send or Re-send Failed:                             = '$failedtosendresend_witnesses ' (' $((failedtosendresend_witnesses*100/$total_witnesses))'%)'
	echo 'Other (Witness Failures):                           = '$(($total_witnesses-($successful_witnesses+$failedtodial_witnesses+$failedtosendresend_witnesses))) ' (' $(( ($total_witnesses-($successful_witnesses+$failedtodial_witnesses+$failedtosendresend_witnesses))*100/$total_witnesses))'%)'
	echo 'Challenger Issues:'
	echo '               |-- Challenger Not Found:            = '$challenger_notfound
	echo '               |-- Challenger Timed Out:            = '$challenger_timeout
	echo '               |-- Challenger Refused Connection:   = '$challenger_refused
	echo '               |-- Challenger Unreachable:          = '$challenger_unreachable
	echo '               |-- Challenger No Listening Address: = '$challenger_nolistenaddr
	echo '******************************************************************'
else
	echo "Bye"
fi

source: https://raw.githubusercontent.com/saad-akhtar/helium_miner_checker/main/hmc.sh

@inigoflores

When running the tool with ./processlogs.php -l both RSSI and SNR are outputted, but not noise. This is a simple calculation (RSSI - SNR) source but a bit cumbersome to effectively do yourself in the console.

I want to see this to better understand if I would benefit from a SAW filter or not.

PS. If you don't have time to do it, but want it added, let me know and I'll do it. Was quite some time since I dabbled in php though!

Thanks!

Hello, I installed on sensecap and it worked. But i can only display a few record, I mean, yesterday i could see about 120 but today only about 70. Also i can't list yesterday's records. Is this normal or am I using the wrong syntax?

Please implement the same for this processlogs.php as for the LoRa logs one (keep the raw log entries in UTC, display datetimes in local tz)

Excellent stuff by the way! thanx a lot for these scripts :)

I have a system that automatically collects logs from all my Sensecap's onto a management server. For this to be a bit more tidy I store my logs as .../HOSTNAME/PREFIX_YYYY-MM-DD.log.gz (example: .../helium-01/console_2022-03-27.log.gz).

Here are two commits that allows this. First commit just adds an asterix to the filename glob so that logs with the date in them can be found. Second commit allows to open, and decompress, gzipped files. Also added a third commit which unsets a temporary variable to save some increased memory usage I caused.

I have intentionally used substr to detect if the filename is .gz instead of using str_ends_with from PHP 8.x, to keep backwards compatibility. None of the patches provide should give and negative impact to any existing use cases.

Please consider accepting this pull request to keep the patches in the original branch. Thank you.

Tested the processlogs.php against /var/log/miner/ on a Controllino v1 running miner-amd64_2022.03.07.0_GA and it works fine.

References:

https://engineering.helium.com/2022/03/07/blockchain-improvements-sync-bugfix.html https://quay.io/repository/team-helium/miner?tab=tags

Awesome little script, would it be possible to add a feature to it? How many witnesses have been sent per hour. I usually get an idea of the state of the network by analyzing this and how many hotspots are sending beacons. Usually it's a constant number but sometimes it just goes way down.

hello... i can not open it i have this error maybe i went to wrong way can you tell me how to open it? lind@Mehdis-iMac helium-miner-log-analyzer % ./processlogs.php zsh: ./processlogs.php: bad interpreter: /usr/bin/php: no such file or directory

Great product thanks! I am using the rak v2 with a modified firmware https://github.com/HeliumDIY/helium_ansible and the logs are located in /home/pi/miner_data/log I ended up overriding $logsFolder manually, I think that it would help if you added to the list.

Thanks!

Tested the processlogs.php against /var/log/miner/ on a Controllino v1 running miner-amd64_2022.02.22.0_GA and it works fine.

References:

1. https://engineering.helium.com/2022/02/23/blockchain-release-sync-improvements.html
2. https://quay.io/repository/team-helium/miner?tab=tags

A new miner image (2022.12.13.0 GA) was released, activating off-chain PoC witnesses and beacons for testing. Before switching to SOL, it could be interesting to include these as an additional feature in the script already.

Some example log entries:

Beaconing [ gateway-rs ] transmitting beacon, beacon: xxxxx ==, module: beacon

Witnessing [ gateway-rs ] poc witness report submitted, beacon: xxxxx ==, module: beacon

I am unsure though if those are off-chain PoC witnesses or whether they may possibly be other LoRa packets.

Running a grep command with the same search phrases produces different results from your php code. The php total for "failed" is the same, but grep shows 220 sent witnesses, while the php shows only 213. No witnesses were sent between runs.

/var/log/miner # date Thu Jun 9 03:08:18 UTC 2022 /var/log/miner # /opt/miner/processlogs.php

Using logs in folder /var/log/miner/

General Witnesses Overview

Total witnesses = 213 (9.65/hour) Succesfully delivered = 209 (98.12%) Failed = 4 (1.88%) /var/log/miner # date Thu Jun 9 03:09:09 UTC 2022 /var/log/miner # grep "miner_onion_server_light:decrypt:" console.log* | grep "sending witness at RSSI"|wc -l 220 /var/log/miner # grep "@miner_poc_grpc_client_statem:send_report" console.log* | grep "failed to submit report"|wc -l 4 /var/log/miner # grep "miner_onion_server_light:decrypt:" console.log|grep "sending witness at RSSI"|sort|tail 2022-06-09 01:03:58.969 153 [info] <0.2102.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -110, Frequency: 904.7, SNR: -1.8 2022-06-09 01:09:52.240 153 [info] <0.2102.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -88, Frequency: 904.1, SNR: 10.8 2022-06-09 01:16:29.374 201 [info] <0.2096.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -103, Frequency: 904.5, SNR: 6.5 2022-06-09 01:22:46.038 201 [info] <0.2096.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -97, Frequency: 904.3, SNR: 5.0 2022-06-09 01:23:22.905 201 [info] <0.2096.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -118, Frequency: 903.9, SNR: -7.5 2022-06-09 02:17:19.792 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -116, Frequency: 904.9, SNR: -7.5 2022-06-09 02:22:54.526 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -115, Frequency: 903.9, SNR: -6.0 2022-06-09 02:24:42.024 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -110, Frequency: 904.9, SNR: -4.0 2022-06-09 02:50:24.910 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -115, Frequency: 903.9, SNR: -3.8 2022-06-09 03:00:33.104 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -119, Frequency: 904.7, SNR: -11.8 /var/log/miner # date Thu Jun 9 03:11:25 UTC 2022

New feature request (not very important since you can't do much with this info): Add your current connection info to the validator (gRPC over http/2 on outbound tcp/8080). Include: IP Address + Port and P2P address (67.211.223.5:8080, 112vvCWKx3s3JrHqxan4fAqNAoezsNXAGyFvZchZmV6mCfwmMcvA) from example below:

Also: not much info you can extract from the api on https://api.helium.io/v1/validators/112vvCWKx3s3JrHqxan4fAqNAoezsNXAGyFvZchZmV6mCfwmMcvA https://explorer.helium.com/validators/112vvCWKx3s3JrHqxan4fAqNAoezsNXAGyFvZchZmV6mCfwmMcvA/activity

Example log contents to be processed:

2022-05-16 05:18:22.653 7 [info] <0.1822.0>@miner_poc_grpc_client_statem:init:{174,5} starting miner_poc_grpc_client_statem
2022-05-16 05:18:29.518 7 [info] <0.1822.0>@miner_poc_grpc_client_statem:setup:{219,13} *** connecting to validator with ip: "67.211.223.5", port: 8080, addr: "/p2p/112vvCWKx3s3JrHqxan4fAqNAoezsNXAGyFvZchZmV6mCfwmMcvA"
2022-05-16 05:18:29.766 7 [info] <0.1822.0>@miner_poc_grpc_client_statem:connect_validator:{514,17} successfully connected to validator via connection #{client => http2_client,host => <<"67.211.223.5">>,http_connection => <0.1854.0>,scheme => <<"http">>}

The challenge will be to detect the current validator connection. In theory, it may change over time (after initial boot up of the miner)

Hello, maybe go in contact with https://glamos.eu/walkapp/ to import the miner data into the app. I look like a big benefit to get an overview about his miner over this combination. Thx in advance.