Simple Magento 2 Prometheus Exporter.

run_as_root GmbH

Last update: Nov 29, 2022

Related tags

Miscellaneous metrics magento prometheus magento2 prometheus-exporter magento-extension magento2-extension magento2-module magento-module prometheus-metrics

Overview

Magento 2 Prometheus Exporter

This Magento 2 Module exposes a new route under /metrics with Magento 2 specific metrics in the format of prometheus. The different metrics are grouped into modules and can be enabled/disabled via the Magento Backend.

Installation

Install the Module via composer by running:

composer require run-as-root/magento2-prometheus-exporter
php bin/magento setup:upgrade

Module Configuration

The modules system configuration is located under Stores -> Configuration -> Prometheus -> Metric Configuration. You can enable or disable specific metrics by using the multiselect.

Prometheus Configuration

After installing the Magento Module, your Prometheus needs to get pointed to your Magento Metrics endpoint. To do so, add the following lines to your prometheus.yml under scrape_configs:

- job_name: 'Magento 2 Exporter'
  scrape_interval: 5m
  scrape_timeout: 60s
  metrics_path: /metrics
  static_configs:
  - targets: 
    - your-magento-url

Module functionality

The module registers a cron job that runs every minute. The cronjob is responsible for aggregating the metric data. The aggregated data is stored in the table run_as_root_prometheus_metrics. The added controller collects the data stored in the table and renders the correct response for prometheus.

Metrics

The following metrics will be collected:

Metric	Labels	TYPE	Help
magento_orders_count_total	status, store_code	gauge	All Magento Orders
magento_orders_amount_total	status, store_code	gauge	Total amount of all Magento Orders
magento_order_items_count_total	status, store_code	gauge	Total count of orderitems
magento_cms_block_count_total	store_code	gauge	Total count of available cms blocks
magento_cms_page_count_total	store_code	gauge	Total count of available cms pages
magento_customer_count_total	store_code	gauge	Total count of available customer
magento_cronjob_broken_count_total		gauge	Broken CronJobs occur when when status is pending but execution_time is set.
magento_cronjob_count_total	status, job_code	gauge	Total count of available CronJob Count.
magento_indexer_backlog_count_total	isValid, title, status	gauge	Total count of backlog item in indexer.
magento_shipments_count_total	source, store_code	counter	Count of Shipments created by store and source.
magento_catalog_category_count_total	status, menu_status, store_code	gauge	Count of Categories by store, status and menu status.
magento_store_count_total	status	gauge	Total count of Stores by status.
magento_website_count_total		gauge	Total count websites.

Add you own Metric

To add a new metric, you need to implement the \RunAsRoot\PrometheusExporter\Api\MetricAggregatorInterface. The metric aggregator object is responsible for collecting the necessary information for the specific metric from magento and then add a new metric record. New records can be easily added via \RunAsRoot\PrometheusExporter\Service\UpdateMetricService.

In addition to the implementation of the MetricAggregatorInterface, you have to add your specific Aggregator to the MetricAggregatorPool defined in the di.xml. For example:

<type name="RunAsRoot\PrometheusExporter\Metric\MetricAggregatorPool">
        <arguments>
            <argument name="items" xsi:type="array">
                <item name="OrderAmountAggregator" xsi:type="object">RunAsRoot\PrometheusExporter\Aggregator\Order\OrderAmountAggregator</item>
                <item name="OrderCountAggregator" xsi:type="object">RunAsRoot\PrometheusExporter\Aggregator\Order\OrderCountAggregator</item>
                <item name="OrderItemAmountAggregator" xsi:type="object">RunAsRoot\PrometheusExporter\Aggregator\Order\OrderItemAmountAggregator</item>
                <item name="OrderItemCountAggregator" xsi:type="object">RunAsRoot\PrometheusExporter\Aggregator\Order\OrderItemCountAggregator</item>
            </argument>
        </arguments>
    </type>

Contribution

If you have something to contribute, weither it's a feature, a feature request, an issue or something else, feel free to. There are no contribution guidelines.

Comments

Prometheus Client PHP Library

Parsing the metrics into a readable format for Prometheus shouldn't be the responsibility of the Magento module.

Prometheus has a link to a PHP library that handles the parsing of metrics into a readable format for Prometheus to interpret.

Prometheus Client Libraries "Prometheus Client PHP" by Jimbo

Moving this responsibility to a separate library reduces the overall code complexity of the module itself, and most likely also reduces bugs because of the Prometheus Client PHP library being well tested by many developers.

opened by Serializator 6
Metrics endpoint broken when "metric_configuration/metric/metric_status" is null

When the "metric_configuration/metric/metric_status" path in the "core_config_data" table is NULL or simply isn't present, the "/metrics" endpoint will throw an exception.

Fatal error: Uncaught TypeError: explode() expects parameter 2 to be string, null given in /magento2-prometheus-exporter/src/Data/Config.php:29

When the module is installed, you have to save the config under "Stores > Configuration > Prometheus" before the "/metrics" endpoint works.

opened by Serializator 3
Issue 24
Related ticket: https://github.com/run-as-root/magento2-prometheus-exporter/issues/24

Metrics added:

magento_shipments_count_total (source, store_code)

magento_catalog_category_count_total (status, menu_status, store_code)

magento_website_count_total

Metrics changed:

magento_store_count_total -> magento_store_count_total (status)

Adjusted unit tests coverage. Updated README.md

Points to discuss:

magento_catalog_category_count_total - currently collects 4 metric for each store:

category enabled, added to the menu

category enabled, removed to the menu

category disabled, added to the menu

category disabled, removed to the menu

Possible suggestions:

remove menu_status label and collect only category status

do not collect menu_status if category is disabled (= leave only 3 metrics, if category disabled - consider that it won't be showed in the menu).

magento_catalog_category_count_total - default store (admin) is not included to the statistic. It will require separated SQL query to fetch data for admin store (e.g. all available categories). Should we collect this data too, or only leave data collection per actual store?
opened by swnsma 2
Multiline comment breaks Prometheus format
If activating `magento_cronjob_broken_count', Prometheus cannot parse these lines:

# HELP magento_cronjob_broken_count_total Magento 2 CronJob Broken Count. Broken CronJobs occur when when status is pending but execution_time is set.

For multiline help texts, we should probably add # before each line
opened by schmengler 2
Metric Indexer Backlog collects wrong information

Currently, the indexer backlog metric counts the number of items in the backlog based on weird version id ranges. This can lead to wrong data and should be changed to counting the specific items in the _cl tables.
bug

opened by DavidLambauer 1
Reset order count and order amount metrics

If you have one order which is in state processing and the state changes after the metric aggregation to complete, than the metrics still count one order as processing and +1 for complete. You have to reset the metric values before the aggregation, otherwise you get incorrect data.

Please create a new git tag after merging this PR 😉

opened by marvincaspar 1
Commented out help lines breaking parsing

I had to do this because exporter output was unparsable with the uncommented lines with Prometheus trying to parse "CronJob" as a float and failing

opened by vedit 1
run-as-root/magento2-prometheus-exporter#30:
Change Indexer Backlog Count Metrics to get date directly from DB.

Add Indexer Changelog Count Metric to count size of changelog tables.

Add Indexer Last Call Metric to get count of seconds since last index update.
opened by swnsma 0
PRODUCTS IN CART LIMIT

Magento recommends having only 100 products in the cart. A metric would be nice to see if that is the case

Reference: https://support.magento.com/hc/en-us/articles/360048550332

opened by DavidLambauer 0
Metric for Attribute Options above recommended level

For Attribute options, Magento recommends keeping the number of options below 100 for performance reasons. Having more than 100 options per attribute is considered to be a bad catalog structure. Having the need for more options should better be solved by restructuring the product catalog, for example by using complex product types, instead of adding more attribute options.

To keep track of this, we should create a metric with the name

opened by DavidLambauer 0
Create Metric for Product Variations greater than recommended

Magento recommends having products with less than 50 variations to prevent possible performance issues. For example, a configurable product with more than 50 children is considered not recommended and should be avoided.

To keep track of those complex product variations, we should create a new metric named magento_complex_product_variations_above_recommended_level.

opened by DavidLambauer 5
Create a new set of metrics
For now, we're having a bunch of different metrics but still have more that we want to provide.

To enhance the current functionality of the module, we'd like to add the following metrics:

magento_shipments_count_total

magento_catalog_category_count_total

magento_website_count_total

magento_store_count_total

The following labels shall always be included in each metric:

store_code

We need to prioritize performance in our data collectors as this module is used by Magento Stores that are quite large.

Make sure to test and document the changes.
opened by DavidLambauer 0

Releases(3.0.1)

3.0.1(Dec 6, 2021)
This release contains the following changes:

Fixed composer Version Number

Full Changelog: https://github.com/run-as-root/magento2-prometheus-exporter/compare/3.0.0...3.0.1
Source code(tar.gz)
Source code(zip)
2.0.2(Jun 14, 2021)
This release contains the following changes:

Reset order count when the status of an order changed

Thanks for @marvincaspar for the contribution! 🥳
Source code(tar.gz)
Source code(zip)
2.0.1(Jun 1, 2021)
This release contains the following changes:

Fix order count reset by @marvincaspar

Thanks to all contributors.
Source code(tar.gz)
Source code(zip)
2.0.0(Jun 1, 2021)
This release contains the following changes:

Changed CLI namespace to run_as_root.

Added new CLI Command: run_as_root:metrics:collect.

Removed Licenses from PHP Files.

Made all Test Cases final.

Introduced custom cron group run_as_root_prometheus_metrics_aggregator.

Restructured module directories.

Added some Interfaces instead of hard file links.

Updated Readme.

Added CronJobCountAggregator to collect metric magento_cronjob_count_total.

Added BrokenCronJobCountAggregator to collect metric magento_cronjob_broken_count_total.

Added IndexerBacklogCountAggregator to collect metric magento_indexer_backlog_count_total.

Source code(tar.gz)
Source code(zip)