This tool can write the monolog standard log directly to clickhouse in real time via the tcp protocol

Related tags

Miscellaneous log2ck
Overview

log2ck

This tool can write the monolog standard log directly to clickhouse in real time via the tcp protocol. If you can write regular rules, other standardized log can also support it.

中文readme

Feature

  • Minimalist code
  • High performance (compared with online services, the cpu usage is only 1/20 of filebeat)
  • No dependence on third-party services (such as queues, etc.)
  • Configurationalization
  • Customization (custom regularization, line processing callback functions)
  • Support reading log divided by day
  • Support breakpoint resume collection

Usage specification

  1. If you use the default regularity, the log file that needs to be read must be the standard default monolog log format file, and the monolog name and group name cannot contain spaces.
  2. The log to be read must be one line at a time. For example, monolog needs to set the formatter to: 'allowInlineLineBreaks'= > false

How to use

run(); # php manager.php">
composer require "hisune/log2ck"
# vim manager.php
use Hisune\Log2Ck\Manager;
require_once 'vendor/autoload.php';
(new Manager(__DIR__ . DIRECTORY_SEPARATOR . 'config.php'))->run();
# php manager.php

config.php Configuration example

return [
    'env' => [ // System environment variables
//        'bin' => [
//            'php' => '/usr/bin/php', // Optional configuration, the path to which the php bin file belongs
//        ],
        'clickhouse' => [ // Required configuration
            'server' => [
                'host' => '192.168.37.205',
                'port' => '8123',
                'username' => 'default',
                'password' => '',
            ],
            'database' => 'logs', // Database name
            'table' => 'repo', // Table name
        ],
//        'worker' => [
//            'cache_path' => '/dev/shm/', // Optional configuration, worker cache directory
//        ],
//        'logger' => [
//            'enable' => true, // Optional configuration, whether to record logs
//            'path' => __DIR__ . DIRECTORY_SEPARATOR . 'logs' . DIRECTORY_SEPARATOR, // Specify the directory where the logs are logged, optional configuration, and need to end with /
//        ],
    ],
    'tails' => [
        'access' => [ // Key is the log name, which corresponds to the name of clickhouse
            'repo' => 'api2', // The name of the project to which the log belongs
            'path' => '/mnt/c/access.log', // Log path, fixed file name log
//            'path' => '/mnt/c/access-{date}.log', // Log path, a daily log with a file name, currently only one macro variable {date} is supported. For example, the date format: 2022-02-22
//            'pattern' => '/\[(?P
   
   
    
    .*)\] (?P
    
    
     
     \w+).(?P
     
     
      
      \w+): (?P
      
      
       
       .*[^ ]+) (?P
       
       
         [^ ]+) (?P 
        
          [^ ]+)/', // Optional configuration, if regular processing is not required, set to false 
         
       
      
      
     
     
    
    
   
   
//            'callback' => function($data) { // Optional configuration, this line of data is processed according to a custom callback method, and the content of the method can implement any logic for cleaning this stream by itself.
//                $data['message'] = 'xxoo'; // For example, customize the processing of this data
//                return $data;
//            }
        ],
    ],
];

supervisord

It is recommended to use supervisor to manage your manager process.

[program:log2ck]
directory=/data/log2ck
command=php manager.php
user=root
autostart=true
autorestart=true
startretries=10
stderr_logfile=/data/logs/err.log
stdout_logfile=/data/logs/out.log

clickhouse Log table structure

If you use monolog and use the default regular rules, you can directly use the following table structure. If you have a custom regular, you can customize your own clickhouse table structure based on your own regular matching results.

create table repo
(
    repo       LowCardinality(String) comment 'Project name',
    name       LowCardinality(String) comment 'Log name',
    host       LowCardinality(String) comment 'The machine where the log is generated',
    created_at DateTime,
    logger     LowCardinality(String),
    level      LowCardinality(String),
    message    String,
    context    String,
    extra      String
) engine = MergeTree()
      PARTITION BY toDate(created_at)
      ORDER BY (created_at, repo, host)
      TTL created_at + INTERVAL 10 DAY;

If the content of your message or context is json, you can refer to clickhouse's json query function:https://clickhouse.com/docs/en/sql-reference/functions/json-functions/

TODO

  • Further improve write performance: single insert is changed to batch insert
You might also like...
Sends log messages to the Logentries log management service

Phalcon Logentries Phalcon library to connect and make log entries using Logentries. You can adapt it to your own needs or improve it if you want. Ple

This Magento extension provides a Real Full Page Caching for Magento powered by Varnish with support of Session-Based information caching (Cart, Customer Accounts, ...) via ESI includes

This Magento extension provides a Real Full Page Caching (FPC) for Magento powered by Varnish with support of Session-Based information caching (Cart, Customer Accounts, ...) via ESI includes

Quickly and easily preview and test your Magento 2 order confirmation page, without hacks or spending time placing new order each time
Quickly and easily preview and test your Magento 2 order confirmation page, without hacks or spending time placing new order each time

Preview Order Confirmation Page for Magento 2 For Magento 2.0.x, 2.1.x, 2.2.x and 2.3.x Styling and testing Magento's order confirmation page can be a

Class for manage directly leds on raspberry pi in PHP

led_raspberry Class for manage directly leds on raspberry pi in PHP We can access GPIO ports directly with fwrite() for power on|off leds, without use

KodExplorer is a file manager for web. It is also a web code editor, which allows you to develop websites directly within the web browser.
KodExplorer is a file manager for web. It is also a web code editor, which allows you to develop websites directly within the web browser.

KodExplorer is a file manager for web. It is also a web code editor, which allows you to develop websites directly within the web browser.

Simple Plugin To Get EXP Directly To Inventory.

Simple Plugin To Get EXP Directly To Inventory.

The swiss army knife for Magento developers, sysadmins and devops. The tool provides a huge set of well tested command line commands which save hours of work time. All commands are extendable by a module API.

netz98 magerun CLI tools for Magento 2 The n98 magerun cli tools provides some handy tools to work with Magento from command line. Build Status Latest

Time registration tool build with Phalcon

PhalconTime Application PhalconTime is a timekeeping tool that helps you track hours spend on clients and projects. Please write me if you have any fe

Releases(1.1.5)
Owner
Hisune
Work Worker
Hisune
YCOM Impersonate. Login as selected YCOM user 🧙‍♂️in frontend.

YCOM Impersonate Login as selected YCOM user in frontend. Features: Backend users with admin rights or YCOM[] rights, can be automatically logged in v

Friends Of REDAXO 17 Sep 12, 2022
Coding-standard - Magento PHP CodeSniffer Coding Standard

ECG Magento Code Sniffer Coding Standard ECG Magento Code Sniffer Coding Standard is a set of rules and sniffs for PHP_CodeSniffer tool. It allows aut

Magento ECG 309 Jan 3, 2023
This tool can help you to see the real IP behind CloudFlare protected websites.

CrimeFlare Bypass Hostname Alat untuk melihat IP asli dibalik website yang telah dilindungi CloudFlare. Introduction Alat ini berfungsi untuk melakuka

zidan rahmandani 126 Oct 20, 2021
You have just downloaded "Messenger-app" [A lightweight, minimalistic real-time chat application]

MESSENGER-APP You have just downloaded "Messenger-app" [A lightweight, minimalistic real-time chat application] Setup To get it working, follow these

Chr1st0ph3r SAB 1 Oct 29, 2021
⌚ Watch your RAM in real time with this script ⌚

PHP-MemWatch Watch your RAM in real time, with this script! * Substitute of watch -n 1 free -m How to use: 1° Download this project 2° Extract to your

Érik Freitas 4 Dec 5, 2021
A fully-managed real-time messaging service that allows you to send and receive messages between independent applications.

A fully-managed real-time messaging service that allows you to send and receive messages between independent applications.

Google APIs 58 Dec 23, 2022
Run a script in real time inside your pocketmine server!

Geral PlayScript is a plugin that allows you to run PHP code directly from the server by command. How to use Move your PHP code (must have a .php exte

LADINO 3 Aug 9, 2022
A real-time chatting website using HTML, CSS, JavaScript, PHP

ChatApp A real-time chatting website using HTML, CSS, JavaScript, PHP #Features Signup Login Signup & Login Validation Encrypted Password Realtime Mes

Suraj Sahu 6 Oct 18, 2022
> Create e-wallet, send money, withdraw and check balance all via USSD protocol

Mobile Money USSD solution Create e-wallet, send money, withdraw and check balance all via USSD protocol Create e-wallet Step 1 Step 2 Step 3 Step 4 S

Emmanuel HAKORIMANA 1 Nov 3, 2021
Tars is a high-performance RPC framework based on name service and Tars protocol, also integrated administration platform, and implemented hosting-service via flexible schedule.

TARS - A Linux Foundation Project TARS Foundation Official Website TARS Project Official Website WeChat Group: TARS01 WeChat Offical Account: TarsClou

THE TARS FOUNDATION PROJECTS 9.6k Jan 1, 2023