Provide CSV, JSON, XML and YAML files as an Import Source for the Icinga Director and optionally ship hand-crafted additional Icinga2 config files

Overview

Icinga Web 2 Fileshipper module

The main purpose of this module is to extend Icinga Director using some of it's exported hooks. Based on them it offers an Import Source able to deal with CSV, JSON, YAML and XML files. It also offers the possibility to deploy hand-crafted Icinga 2 config files through the Icinga Director.

Icinga Web 2 Fileshipper

For getting started please read our Installation instructions, and then you should be ready to dive into Import Source definitions, supported file formats or and hand-crafted Config File Shipping.

Changes

v1.2.0

  • FEATURE: PHP 8 support
  • FEATURE: Give guidance on potential misconfiguration (#34)
  • FEATURE: do not fail on malformed config file (#35)

v1.1.0

  • FEATURE: Added XLSX file support

v1.0.1

  • FEATURE: CSV files should give NULL for columns with empty strings (#6)
  • FIX: Small documentation fix
Comments
  • Comments starting with '#' are deprecated in ...

    Comments starting with '#' are deprecated in ...

    I receive the following messages when I open in Director the Import Source / Modify Menu which uses the Fileshipper module (fd2c797) - though it also happens with older versions:

    Deprecated: Comments starting with '#' are deprecated in Unknown on line 1 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    
    Deprecated: Comments starting with '#' are deprecated in Unknown on line 2 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    
    Deprecated: Comments starting with '#' are deprecated in Unknown on line 3 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    
    Deprecated: Comments starting with '#' are deprecated in Unknown on line 4 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    
    Deprecated: Comments starting with '#' are deprecated in Unknown on line 5 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    
    Deprecated: Comments starting with '#' are deprecated in Unknown on line 6 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    
    Deprecated: Comments starting with '#' are deprecated in Unknown on line 7 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265
    

    Context

    The problem is only annoying and not serious, or at least for me the module seems to work fine, it just presents these messages, it would be fine to get rid of them. I am also not 100% percent sure, that this is related to the fileshipper module, but since it just pops up when I use it, I report it here.

    Your Environment

    • icingaweb2-2.5.1-1.el7.icinga.noarch
    • icinga2-2.8.4-1.el7.icinga.x86_64
    • Director Version (fc58e50)
    • RHEL 7.5
    • PHP 5.4.16 (/bin/php) and PHP 7.1.8 (/opt/rh/rh-php71/root/usr/bin/php)
    opened by swizzly 2
  • Multiple values in CSV import

    Multiple values in CSV import

    How to achieve importing host to multiple groups (Group membership)?

    Example1: id, Host, Group1, Group2, Group3 1,"Alpha","g1","g2","g3" +assignments: ${Group1} -> groups ${Group2} -> groups ${Group3} -> groups This works for import (g1, g2, g3 detected as different groups and fields, but Sync job performs only host assignment to latest Group3 [g3]) .

    Example2: id, Host, Group, 1,"Alpha","g1 g2 g3" +assignment: ${Group} -> groups
    This works for import (g1, g2, g3 detected as different groups, but Sync job performs only host assignment to g3).

    Space is valid csv multi-value field separator, but isn't valid for Sync job.

    opened by ily4 2
  • Step-wise Import Source preview

    Step-wise Import Source preview

    It would be nice to be able to have more customization options in the Import Source preview. For example selectively enabling only parts of the configured Import Source modifiers would sometimes help beginners to better understand what's going on.

    enhancement 
    opened by Thomas-Gelf 1
  • File name * (all files) results in No key column

    File name * (all files) results in No key column "Key_column_name" in row 1

    Choosing File name: * (all files) always results in a No key column "Key_column_name" in row 1 error.

    Expected Behavior

    All files should correctly be processed

    Current Behavior

    Error occurs even when there is a 1 valid file in the directory. I've duplicated with CSV and YAML.

    image

    Possible Solution

    Steps to Reproduce (for bugs)

    $ cat /etc/icingaweb2/modules/fileshipper/imports.ini
    [csv]
    basedir = /var/cache/icinga2/import/csv
    
    $ ls -la /var/cache/icinga2/import/csv
    total 4
    drwxr-sr-x 2 root   icingaweb2 21 Oct  5 15:07 .
    drwxr-sr-x 7 icinga icingaweb2 63 Oct  5 14:58 ..
    -rw-r--r-- 1 root   icingaweb2 13 Oct  5 15:07 list.csv
    
    $ cat -A /var/cache/icinga2/import/csv/list.csv
    host$
    my_host$
    

    image

    Using the same settings but selecting the single file instead of * (all files) works as expected.

    Context

    Trying to do an import of multiple files.

    Your Environment

    • Module version (System - About): master (f5fff02f3622221d9c43a906743de4ae1fdce368)
    • Icinga Web 2 version and modules (System - About): 2.4.2 (7cec28a31fdff0feb470ec001838bc1dec9c4b69)
    • Icinga 2 version (icinga2 --version):
    icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.1-1)
    
    Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
    License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Application information:
      Installation root: /usr
      Sysconf directory: /etc
      Run directory: /run
      Local state directory: /var
      Package data directory: /usr/share/icinga2
      State path: /var/lib/icinga2/icinga2.state
      Modified attributes path: /var/lib/icinga2/modified-attributes.conf
      Objects path: /var/cache/icinga2/icinga2.debug
      Vars path: /var/cache/icinga2/icinga2.vars
      PID path: /run/icinga2/icinga2.pid
    
    System information:
      Platform: CentOS Linux
      Platform version: 7 (Core)
      Kernel: Linux
      Kernel version: 3.10.0-327.10.1.el7.x86_64
      Architecture: x86_64
    
    Build information:
      Compiler: GNU 4.8.5
      Build host: unknown
    
    • Webserver, PHP versions:
    $ php --version
    PHP 5.4.16 (cli) (built: Nov  6 2016 00:29:02)
    Copyright (c) 1997-2013 The PHP Group
    Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
    
    $ nginx -V
    nginx version: nginx/1.12.1
    built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
    built with OpenSSL 1.0.1e-fips 11 Feb 2013
    TLS SNI support enabled
    configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
    
    $ php-fpm --version
    PHP 5.4.16 (fpm-fcgi) (built: Nov  6 2016 00:30:57)
    Copyright (c) 1997-2013 The PHP Group
    Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
    
    duplicate 
    opened by druchoo 1
  • Memory size is exhausted while changes are deployed

    Memory size is exhausted while changes are deployed

    When trying to import and sync a simple host file (just containing the host block and ip-address) the following error is shown, when deploying the changes:

    Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObjectImports.php on line 221
    

    It is not possible to deploy the changes from activity log nor revoke them. They result in the need of restoring the VM, since re-deploying an older configuration does not reset the changes waiting for deployment.

    If logs or anything else is needed, just tell me what you need :)

    PS: Not sure if this is fileshipper or director related, feel free to send me over to the other repo ^^

    opened by Shadowghost 1
  • CSV Import use Contains in Assign_filter

    CSV Import use Contains in Assign_filter

    Hi,

    is it possible to use the contains string while importing Services from csv via the assign_filter attribute? My CSV: 'apply_name','service_name','assign_where' 'port-channel1','Service-NWC-Interface-Health-extra-Port-Channel','host.vars.monitoring_interfaces contains port-channel1' I tried "==" as well as "in", but it throws errors.

    The normal assignment is like: assign where "port-channel1" in host.vars.monitoring_interfaces

    Best regards

    opened by Simkimdm 1
  • cannot import/ sync host templates

    cannot import/ sync host templates

    I don't know if its by design, but cannot seem to be able to sync object_type template. I'm getting errors

    This Sync Rule failed when last checked at 2016-11-30 14:44:52: Exception while syncing Icinga\Module\Director\Objects\IcingaHost zone01_template: Sync is not allowed to modify template "zone01_template"

    If i remove the sync properties for "object_type", I can create from the file.

    opened by plarivee 1
  • Unable to add a new host group - Error

    Unable to add a new host group - Error

    Hi,

    when I try to add a new host group with icinga2-director, following error occurs: (I have installed the latest version from github (master).)

    #0 /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObject.php(1133): Icinga\Module\Director\Objects\IcingaObject->assertCustomVarsSupport()
    #1 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/IcingaObjectFieldLoader.php(51): Icinga\Module\Director\Objects\IcingaObject->vars()
    #2 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/DirectorObjectForm.php(263): Icinga\Module\Director\Web\Form\IcingaObjectFieldLoader->setValues(Array, 'var_')
    #3 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/DirectorObjectForm.php(566): Icinga\Module\Director\Web\Form\DirectorObjectForm->handleCustomVars(Object(Icinga\Module\Director\Objects\IcingaHostGroup), Array)
    #4 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/QuickForm.php(430): Icinga\Module\Director\Web\Form\DirectorObjectForm->onRequest()
    #5 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/QuickForm.php(437): Icinga\Module\Director\Web\Form\QuickForm->setRequest(Object(Icinga\Web\Request))
    #6 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/QuickForm.php(304): Icinga\Module\Director\Web\Form\QuickForm->getRequest()
    #7 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(205): Icinga\Module\Director\Web\Form\QuickForm->handleRequest()
    #8 /usr/share/php/Zend/Controller/Action.php(516): Icinga\Module\Director\Web\Controller\ObjectController->addAction()
    #9 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch('addAction')
    #10 /usr/share/php/Zend/Controller/Front.php(954): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
    #11 /usr/share/php/Icinga/Application/Web.php(384): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
    #12 /usr/share/php/Icinga/Application/webrouter.php(109): Icinga\Application\Web->dispatch()
    #13 /usr/share/icingaweb2/public/index.php(4): require_once('/usr/share/php/...')
    #14 {main}
    

    For any other information about my setup, please tell me what information I have to give to you!

    Thank you! Klaus.

    opened by tachtler 1
  • When imports.ini is misformed the director form fails

    When imports.ini is misformed the director form fails

    Expected Behavior

    It should gracefully display an error inside the form, alarming the user the config file should be corrected.

    Current Behavior

    Error in UI:

    array_map(): Expected parameter 2 to be an array, string given
    

    Steps to Reproduce (for bugs)

    Broken /etc/icingaweb2/modules/fileshipper/imports.ini:

    xamples]
    basedir = "/opt/import"
    

    Go to Director any create or modify any fileshipper import.

    Your Environment

    • Module version (System - About): 1.7.2
    • Icinga Web 2 version and modules (System - About): 2.8.2
    • Icinga 2 version (icinga2 --version): 2.12.1
    • Operating System and version: CentOS 7
    • Webserver, PHP versions: PHP 7.3
    opened by lazyfrosch 0
  • Bugfix/fix gui labels

    Bugfix/fix gui labels

    Current Behavior

    There is written the word "directoy" in modules/fileshipper/library/Fileshipper/ProvidedHook/Director/ImportSource.php and ' defaults to double quote: "' in the same file, which is not true as the field requires a character

    screenshot from 2019-02-07 16-58-21

    How to reproduce

    • Director
    • Import data sources
    • Add
    • under "source type" select Fileshipper
    • look at the additional options that pop up

    My Environment

    • Director Version: 1.5.1
    • Icinga 2 version: 2.10.1-1
    • Operating System and version: Centos 7
    • PHP versions: rh-php71-php-fpm-7.1.8

    Comment

    This pull request refers to the pull request (https://github.com/Icinga/icingaweb2-module-fileshipper/pull/28) but including some suggestions from @Thomas-Gelf

    opened by angelorosacewp 0
  • Fix English errors in GUI labels and tooltips.

    Fix English errors in GUI labels and tooltips.

    Current Behavior

    There is written the word "directoy" in modules/fileshipper/library/Fileshipper/ProvidedHook/Director/ImportSource.php and ' defaults to double quote: "' in the same file, which is not true as the field requires a character

    screenshot from 2019-02-07 16-58-21

    How to reproduce

    • Director
    • Import data sources
    • Add
    • under "source type" select Fileshipper
    • look at the additional options that pop up

    My Environment

    • Director Version: 1.5.1
    • Icinga 2 version: 2.10.1-1
    • Operating System and version: Centos 7
    • PHP versions: rh-php71-php-fpm-7.1.8
    opened by angelorosacewp 0
  • Huge database tables

    Huge database tables

    Hi, we have fileshipper running for an automated Netbox -> Icinga2 export for a multitude of objects like switches, switch interfaces, PDUs, Firewalls and stuff.

    Importing the files took longer and longer which is astonishing as the amount of objects didnt change. Today i had a look at the database tables and i was baffled:

    i2director=# select count(*) from imported_row;
      count  
    ---------
     1179641
    (1 row)
    
    i2director=# select count(*) from imported_row_property;
      count  
    ---------
     6823277
    (1 row)
    

    I import around 150-180 devices with a total of 1000 objects, so my impression is that there is no cleanup/expire of old objects happening.

    opened by flohoff 0
  • JSON Import: Command: Unable to set

    JSON Import: Command: Unable to set "plugin-check-command"

    I am trying to import a command via JSON. However, there is no possibility to define the command type (plugin-check-comand) in the properties. There is only the possibility to define an import, but you can't set "plugin-check-command" there either, because otherwise errors occur. If you look at the imported command, the value for Command Type is not defined, but it can still be rendered. Saving the imported command in the director is not possible, because this is a forced field.

    It should be noted that the entire documentation for Fileshipper is more than poor. It could be documented, for example, what format the individual properties must have. So it's just guessing and dull trying until it might work sometime.

    Expected Behavior

    Current Behavior

    Possible Solution

    Steps to Reproduce (for bugs)

    [
        {
            "arguments": {
                "-H": {
                    "description": "fs_http_host_argument",
                    "value": "$fs_http_host$"
                }
            },
            "command": "fs_http",
            "object_name": "fs_http"
        }
    ]
    

    source sync_modify sync_properties command

    Context

    Your Environment

    • Module version (System - About): 1.2.0
    • Icinga Web 2 version and modules (System - About): 2.8.2
    • Icinga 2 version (icinga2 --version): r2.12.3
    • Operating System and version: Debian 10
    • Webserver, PHP versions: Apache 2, PHP 7.3.27
    opened by virtualdreams 0
  • Fix english in GUI labels

    Fix english in GUI labels

    Current Behavior

    There is written the word "directoy" in modules/fileshipper/library/Fileshipper/ProvidedHook/Director/ImportSource.php and ' defaults to double quote: "' in the same file, which is not true as the field requires a character

    screenshot from 2019-02-07 16-58-21

    How to reproduce

    • Director
    • Import data sources
    • Add
    • under "source type" select Fileshipper
    • look at the additional options that pop up

    My Environment

    • Director Version: 1.5.1
    • Icinga 2 version: 2.10.1-1
    • Operating System and version: Centos 7
    • PHP versions: rh-php71-php-fpm-7.1.8

    Comment

    This pull request refers to the pull request (https://github.com/Icinga/icingaweb2-module-fileshipper/pull/28) but with suggestions from @Thomas-Gelf

    opened by angelorosacewp 0
  • Allow Import from external URL

    Allow Import from external URL

    Expected Behavior

    Import from an external url like http://example.com/hosts.json

    Current Behavior

    You can only import from the local filesystem

    Context

    We have a System where all Devices are already configured and that way we could directly import them from there

    opened by fionera 0
Releases(v1.2.0)
Owner
Icinga
Icinga is a resilient, open source monitoring and metric solution system.
Icinga
Improve default Magento 2 Import / Export features - cron jobs, CSV , XML , JSON , Excel

Improve default Magento 2 Import / Export features - cron jobs, CSV , XML , JSON , Excel , mapping of any format, Google Sheet, data and price modification, improved speed and a lot more!

Firebear Studio 173 Dec 17, 2022
World countries - available in multiple languages, in CSV, JSON, PHP, SQL and XML formats

Constantly updated lists of world countries and their associated alpha-2, alpha-3 and numeric country codes as defined by the ISO 3166 standard, available in CSV, JSON , PHP, SQL and XML formats, in multiple languages and with national flags included; also available are the ISO 3166-2 codes of provinces/ states associated with the countries

Stefan Gabos 1k Dec 29, 2022
phalcon config loader for yaml

Phalcon Config Loarder for Yaml Loads all the yml in the directory of the app/config. Version PHP: 7.0.x, 7.1.x, 7.2.x Phalcon: 3.x Composer { "r

Toshiyuki Ienaga 2 Oct 7, 2022
A wrapper around symplify/config-transformer used to update recipes and using easy coding standard for generating readable config files.

Symfony Recipes Yaml to PHP Converter This is a wrapper around the symplify/config-transformer used to convert Symfony core recipes which uses .yaml c

Alexander Schranz 3 Nov 24, 2022
JSONFinder - a library that can find json values in a mixed text or html documents, can filter and search the json tree, and converts php objects to json without 'ext-json' extension.

JSONFinder - a library that can find json values in a mixed text or html documents, can filter and search the json tree, and converts php objects to json without 'ext-json' extension.

Eboubaker Eboubaker 2 Jul 31, 2022
Magento commands to find translations that are present in one CSV file but not in another, and to translate CSV dicts with DeepL

Hyvä Themes - Magento translation CSV comparison command hyva-themes/magento2-i18n-csv-diff This module adds the bin/magento i18n:diff-csv and i18n:tr

Hyvä 6 Oct 26, 2022
Nagios/Icinga plugin for checking SATEON field network device status

check_sateon Nagios/Icinga plugin for checking SATEON field network device status Max.Fischer [email protected] Tested on CentOS GNU/Linux 6.5 with Ici

null 1 May 27, 2022
The main website source code based on php , html/css/js and an independent db system using xml/json.

jsm33t.com Well umm, a neat website LIVE SITE » View Demo · Report Bug · Request a feature About The Project Desc.. Built Using Php UI Frameworks Boot

Jasmeet Singh 5 Nov 23, 2022
This app is to measure the hand and eye co-ordination speed based on the score generated taken from Database

CoOrdinationSpeedTest Website link: https://skyward-punctures.000webhostapp.com/ Try this only when you are a psychiatrist ?? ?? This app runs as php

MANOJKUMAAR GOWDA 1 Jan 12, 2022
The Ravioli WooCommerce plugin helps you ship your order with Ravioli.

=== Ravioli for WooCommerce === Contributors: canolcer Tags: ravioli, ecommerce, shipping Requires at least: 5.0 Tested up to: 6.0.1 Stable tag: trunk

Ravioli 2 Nov 7, 2022
Raspberry Pi wifi hotspot with an offline-first community portal. Optionally shares internet access over Tor.

Raspberry Pi wifi hotspot with an offline-first community portal. Optionally shares internet access over Tor.

Martti Malmi 17 Dec 15, 2022
Sync Wordpress Pages and Posts (even custom post types + fields) from static Markdown + YAML files

Sync Markdown Files to WordPress Posts and Pages Static site generators let you use a revision-controlled tree of markdown files to make a site, but d

null 26 Sep 26, 2022
Silverstripe-populate - Populate your database through YAML files

Populate Module This module provides a way to populate a database from YAML fixtures and custom classes. For instance, when a building a web applicati

Silverstripe CMS 22 Oct 3, 2022
Learning about - Basic HTML & CSS, JSON, XML, Session & Cookies, CRUD Operations in Php using MySQL and Create MVC from scratch

This Project is based on course CSC 3215. Learning about - Basic HTML & CSS, JSON, XML, Session & Cookies, CRUD Operations in Php using MySQL and Create MVC (Model–View–Controller) from scratch. Just learning about web technologies, Not focusing on UI (Bootstrap or other 3rd-Party UI libraries or frameworks).

Alvi Hasan 5 Sep 21, 2022
A small CLI tool to check missing dependency declarations in the composer.json and module.xml

Integrity checker Package allows to run static analysis on Magento 2 Module Packages to provide an integrity check of package. Supported tools: Compos

run_as_root GmbH 13 Dec 19, 2022
The easiest way to match data structures like JSON/PlainText/XML against readable patterns. Sandbox:

PHP Matcher Library created for testing all kinds of JSON/XML/TXT/Scalar values against patterns. API: PHPMatcher::match($value = '{"foo": "bar"}', $p

Coduo 774 Dec 31, 2022
Sistema disema con aplicación de consultas en XML y JSON

disema-XML-JSON Sistema web para empresa de diseño "Disema", con operaciones básicas CRUD y uso de html, JQ, JS, php y css. Incluye aplicación de cons

null 1 Jan 12, 2022
laminas-xml2json provides functionality for converting XML structures to JSON

laminas-xml2json This package is considered feature-complete, and is now in security-only maintenance mode, following a decision by the Technical Stee

Laminas Project 13 Dec 28, 2022
Allows generate class files parse from json and map json to php object, including multi-level and complex objects;

nixihz/php-object Allows generate class files parse from json and map json to php object, including multi-level and complex objects; Installation You

zhixin 2 Sep 9, 2022