Magento n98-magerun module for importing and exporting configuration data

Overview

HarrisStreet ImpEx for Magento

Import and Export of Magentos configuration stored in core_config_data table. Handling of different environments with inheritance.

There is an awesome blogpost from @cmuench about this module!

Supports multiple formats like:

At the moment the Yaml import format is the most supported. CSV and JSON works also well.

Export

$ ./n98-magerun.phar hs:ccd:export --help
Usage:
 hs:ccd:export [-m|--format[="..."]] [-a|--hierarchical[="..."]] [-f|--filename[="..."]] [-i|--include[="..."]] [-x|--exclude[="..."]] [-s|--filePerNameSpace[="..."]] [-c|--exclude-default[="..."]]

Options:
 --format (-m)           Format: yaml, json, csv, xml, limeSodaXml (default: "yaml")
 --hierarchical (-a)     Create a hierarchical or a flat structure (not all export format supports that). Enable with: y (default: "n")
 --filename (-f)         File name into which should the export be written. Defaults into var directory.
 --include (-i)          Path prefix, multiple values can be comma separated; exports only those paths
 --includeScope          Scope name, multiple values can be comma separated; exports only those scopes
 --exclude (-x)          Path prefix, multiple values can be comma separated; exports everything except ...
 --filePerNameSpace (-s) Export each namespace into its own file. Enable with: y (default: "n")
 --exclude-default (-c)  Excludes default values (@todo)
 --help (-h)             Display this help message.

The three switches --include, --includeScope and --exclude are added to the SQL query via AND.

Examples

Export hierarchical xml for the general and tax namespace:

$ ./n98-magerun.phar hs:ccd:export --format=xml --include=general,tax --hierarchical=y

Export all configuration settings each namespace into its own file starting with the prefix test:

$ ./n98-magerun.phar hs:ccd:export --filePerNameSpace=y --filename=test
Wrote: 12 settings to file test_admin.yaml
Wrote: 118 settings to file test_advanced.yaml
Wrote: 18 settings to file test_bestsellerproductslider.yaml
Wrote: 6 settings to file test_bss.yaml
Wrote: 294 settings to file test_carriers.yaml
Wrote: 61 settings to file test_catalog.yaml
Wrote: 12 settings to file test_cataloginventory.yaml
Wrote: 16 settings to file test_checkout.yaml
...

Import

To import a configuration you need a specific setup of folders in the filesystem.

$ ./n98-magerun.phar hs:ccd:import --help
Usage:
 hs:ccd:import [-m|--format[="..."]] [-a|--hierarchical[="..."]] [--base[="..."]] folder environment

Arguments:
 folder                Import folder name
 env                   Environment name. SubEnvs separated by slash e.g.: development/osx/developer01

Options:
 --format (-m)         Format: yaml, json, csv, xml, limeSodaXml (default: "yaml")
 --hierarchical (-a)   Create a hierarchical or a flat structure (not all export format supports that). Enable with: y (default: "n")
 --base                Base folder name (default: "base")
 --help (-h)           Display this help message.
 --quiet (-q)          Do not output any message.
 --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
 --version (-V)        Display this application version.
 --ansi                Force ANSI output.
 --no-ansi             Disable ANSI output.
 --no-interaction (-n) Do not ask any interactive question.
 --root-dir            Force magento root dir. No auto detection
 --skip-config         Do not load any custom config.

Yaml File Format

"Example with store-view code" web/unsecure/base_url: stores: my_store_code: 'http://example.com/another-base-url/' # Delete entry web/unsecure/base_url: stores: my_store_code: !!delete # Run n98-magerun commands commands/run: - sys:store:list - db:query "select * from sales_flat_order" ">
# Default scope
web/unsecure/base_url:
  default:
    0: 'http://example.com/my-base-url/'

# Store view scope -> "Example Store-ID 1"
web/unsecure/base_url:    
  stores:
    1: 'http://example.com/my-base-url/'  

# Store view scope -> "Example with store-view code"
web/unsecure/base_url:    
  stores:
    my_store_code: 'http://example.com/another-base-url/'  

# Delete entry    
web/unsecure/base_url:    
  stores:
    my_store_code: !!delete

# Run n98-magerun commands
commands/run:
  - sys:store:list
  - db:query "select * from sales_flat_order"

Folder setup

Folder Structure

An example import for the development environment of cyrill would like:

$ ./n98-magerun.phar hs:ccd:import ./configuration/newCoreConfigData development/cyrill

How does the import work?

Focusing in the printscreen above on the folder: configuration/newCoreConfigData and our example import command.

The importer expects always a base folder (also configurable via CLI option) where all default configuration options are store in n-files. It reads all those files and loads their settings into Magento.

In the next step the importer loads all files from the development folder but non-resursive and loads that content into Magento. After that it jumps into the folder cyrill and loads there the files. Finished!

The output looks like:

Processed: ./configuration/newCoreConfigData/base/contacts.yaml with 4 values.
Processed: ./configuration/newCoreConfigData/base/crontab.yaml with 6 values.
Processed: ./configuration/newCoreConfigData/base/currency.yaml with 11 values.
Processed: ./configuration/newCoreConfigData/base/customer.yaml with 33 values.
Processed: ./configuration/newCoreConfigData/base/design.yaml with 40 values.
Processed: ./configuration/newCoreConfigData/base/dev.yaml with 14 values.
Processed: ./configuration/newCoreConfigData/base/general.yaml with 15 values.
Processed: ./configuration/newCoreConfigData/development/test_web.yaml with 36 values.
Processed: ./configuration/newCoreConfigData/development/cyrill/test_web.yaml with 4 values.

You are totally free of naming the folders and files.

The file format during import will be detected with its extension. .yaml works were as .yml won't.

Convert

Is really useful to create a .magerun file from your configuration to process later that file on a server.

stdout">
$ ./n98-magerun.phar help hs:ccd:convert
Usage:
 hs:ccd:convert [-m|--format[="..."]] [-a|--hierarchical[="..."]] [--base[="..."]] [--export-file[="..."]] folder env

Arguments:
 folder                Import folder name
 env                   Environment name. SubEnvs separated by slash e.g.: development/osx/developer01

Options:
 --base                Base folder name (default: "base")
 --export-file         File name in which the n98 commands shoud be written. If empty -> stdout

Installation

Require this installer in your composer.json file:

"require": {
	…
    "zookal/harris-street-impex": "dev-master",
    …
}

Or via modman:

$ modman clone [email protected]:Zookal/HarrisStreet-ImpEx.git

License

Open Software License (OSL 3.0)

Copyright

Copyright (c) Zookal Pty Ltd, Sydney Australia

Author

Cyrill at Schumacher dot fm or cyrill at zookal dot com

My pgp public key

@SchumacherFM

Comments
  • Stop switch parsing before path and value in config:set

    Stop switch parsing before path and value in config:set

    We had an issue with strings as config values containing words (white-space separated) that started with two dashes as those were interpreted as additional options (which then were undefined).

    This proposed change did levitate the problem on our end.

    opened by tkn98 3
  • Value beginning with '-' causes error

    Value beginning with '-' causes error

    Example YAML:

    some_settings/setting/config_value:
      default:
        0: '-15px'
    

    Trying to import the above results in the following error:

    [RuntimeException]               
      The "-1" option does not exist.  
    

    Seems to be the same for any value starting with a minus sign.

    opened by bluec 3
  • Validate Scope and ScopeId before updating core_config_data

    Validate Scope and ScopeId before updating core_config_data

    to prevent the creation of incorrect database entries we should validate whether a scope-scopeId combination is valid. Otherwise you may have strange side-effects and database entries.

    opened by mwr 3
  • Unable to use composer to update n98-magerun, package cannot be found.

    Unable to use composer to update n98-magerun, package cannot be found.

    When we added this to the require section of composer.json, we get the following error:

    The requested package zookal/harris-street-impex could not be found in any version, there may be a typo in the package name.
    

    It appears the package isn't actually available for composer?

    opened by jonathanvaughn 3
  • Fix issue with escaped character during export

    Fix issue with escaped character during export

    If a configuration value has a single quote, it will be escaped. e.g.

    'Our price is lower than the manufacturer\'s "minimum advertised price."  As a result, we cannot show you the price in catalog or the product page. <br /><br /> You have no obligation to purchase the product once you know the price. You can simply remove the item from your cart.'
    

    The import parser for YAML provide an error and doesn't import the content.

    By changing from single quote to double quote for text delimitation and escaping those double quote the import has no issue.

    "Our price is lower than the manufacturer's \"minimum advertised price.\"  As a result, we cannot show you the price in catalog or the product page. <br /><br /> You have no obligation to purchase the product once you know the price. You can simply remove the item from your cart."
    
    opened by sylvainraye 2
  • hs:ccd:export filename (-f) gets

    hs:ccd:export filename (-f) gets ".yaml" appended

    This is counter-intuitive if you want to write to stdout.

    For example the following command does not work:

    $ n98-magerun.phar hs:ccd:export --include shipping/origin -fphp://stdout
    

    because the following filename becomes created:

    Failed to write: php://stdout.yaml
    

    Automatically adding ".yaml" destroys the portability of the switch. I suggest to remove that feature for -f / --filename and add a new switch that brings it back: --prepend-filename-with-dot-yaml-extension.

    I'd like to get some feedback on this suggestion before I consider to provide a PR. Thanks.

    opened by tkn98 1
  • [TASK] Experimental execution of n98-magerun commands out of configuration

    [TASK] Experimental execution of n98-magerun commands out of configuration

    Hi,

    i created an experimental feature to run n98-magerun commands out of a config. I tested it only in yaml files.

    The configuration lools like this:

    commands/run:
       - cache:flush
       - config:get web/*
    

    What do you think?

    opened by cmuench 1
  • Removed symfony/yaml dependency

    Removed symfony/yaml dependency

    symfony/yaml is allready a dependency of n98/magerun package and already included in the phar file. I have some trouble because "*" will try to install the latest dev-master package of symfony/yaml which needs at least PHP 5.5.3. Not all servers have PHP 5.5 installed.

    opened by cmuench 0
  • YAML export incorrectly escapes single quotes

    YAML export incorrectly escapes single quotes

    The YAML export escapes single quotes with a \ which is incorrect and causes an error to occur on the import of any such strings.

    In YAML, a double quote '' (i.e. 2 singles) is how a single ' quote is escaped.

    Suggested resolution in src/HarrisStreet/CoreConfigData/Exporter/Yaml.php change:

    return '\'' . addcslashes($value, '\'') . '\'';
    

    to

    return sprintf("'%s'", str_replace('\'', '\'\'', $value));
    
    opened by bluec 0
  • YAML export should treat numeric values as strings and enclose in quotes

    YAML export should treat numeric values as strings and enclose in quotes

    Using YAML export a string like '000000' is exported without quotes. This means upon import it is then treated as a numeric value and is imported as 0. You can see how this can cause problems.

    Magento core_config_data treats everything like a string so it makes sense if the export / import does also.

    Possible solution: in src/HarrisStreet/CoreConfigData/Exporter/Yaml.php remove the is_numeric test and treat everything like a string.

    opened by bluec 0
  • Stabilize Import by Using Store/Storeview Codes next to IDs

    Stabilize Import by Using Store/Storeview Codes next to IDs

    Like the DNS system uses names instead of numbers to make the system more robust in a distributed environment, I see great use in using store and store-view codes (which are names) instead of IDs (which are numbers) when importing data (same can be applied to exporting, too).

    This change would be backwards compatible because it can be clearly differentiated between numeric ID and alpha-numeric store code (if you follow the rule to not start it with a number or you don't only use numbers).

    I plan to file a PR as I got positive feedback for this feature already from my peers at work. I'm pretty sure it is much appreciated.

    opened by tkn98 0
  • hs:ccd:export filename (-f) gets

    hs:ccd:export filename (-f) gets ".yaml" appended

    This is counter-intuitive if you want to write to stdout.

    For example the following command does not work:

    $ n98-magerun.phar hs:ccd:export --include shipping/origin -fphp://stdout
    

    because the following filename becomes created:

    Failed to write: php://stdout.yaml
    

    Automatically adding ".yaml" destroys the portability of the switch. I suggest to remove that feature for -f / --filename and add a new switch that brings it back: --prepend-filename-with-dot-yaml-extension.

    opened by tkn98 2
Releases(v1.2.1)
Owner
Zookal
Zookal
netz98 magerun CLI tools for Magento 1

netz98 magerun CLI tools for Magento 1 The n98 magerun cli tools provides some handy tools to work with Magento from command line. Build Status Latest

netz98 1.4k Dec 31, 2022
The VarExporter component allows exporting any serializable PHP data structure to plain PHP code.

The VarExporter component allows exporting any serializable PHP data structure to plain PHP code. While doing so, it preserves all the semantics associated with the serialization mechanism of PHP (__wakeup, __sleep, Serializable).

Symfony 1.8k Jan 1, 2023
Perch Dashboard app for exporting content to (Kirby) text files and Kirby Blueprint files

toKirby Perch Dashboard app for exporting content to (Kirby) text files and Kirby Blueprint files. You can easily install and test it in a few steps.

R. Banus 4 Jan 15, 2022
Magento 2 module to automatically flush the cache whenever you save something in the System Configuration

Yireo AutoFlushCache Magento 2 module to automatically flush the cache whenever you save something in the System Configuration. Do NOT use this in pro

Yireo 21 May 14, 2022
Magento 2 module for displaying additional information in configuration

AvS_ScopeHint for Magento 2 Displays a hint when a configuration value is overwritten on a lower scope (website or store view). Facts version: 1.0.0-b

Andreas von Studnitz 131 Dec 14, 2022
Import/Export configuration data in Magento 2 via CLI.

ConfigImportExport This module provides new CLI commands for Magento 2 to import/export data in/from core_config_data. This module is inspired by the

semaio 135 Dec 9, 2022
Import/Export configuration data in Magento 2 via CLI.

ConfigImportExport This module provides new CLI commands for Magento 2 to import/export data in/from core_config_data. This module is inspired by the

semaio 117 Mar 23, 2022
A tool that allows to quickly export data from Magento 1 and Magento 2 store and import it back into Magento 2

Simple Import / Export tool A tool that allows to quickly export data from Magento 1 and Magento 2 store and import it back into Magento 2. Table data

EcomDev B.V. 51 Dec 5, 2022
Magento 2 Module Experius Page Not Found 404. This module saves all 404 url to a database table

Magento 2 Module Experius Page Not Found 404 This module saves all 404 urls to a database table. Adds an admin grid with 404s It includes a count so y

Experius 28 Dec 9, 2022
Automatically load the next page of products in Magento. Easy to install and configure, this module works 100% out of the box with vanilla Magento 1.9.x and earlier.

Automatically load the next page of products in Magento. Easy to install and configure, this module works 100% out of the box with vanilla Magento 1.9.x and earlier.

Strategery 123 Nov 20, 2021
Magento 2 module which can find potential url related problems in your catalog data

Url data integrity checker module for Magento 2 Purpose The purpose of this module is to give store owners of a Magento 2 shop insight into what url-r

Baldwin 218 Jan 1, 2023
Magento 2 Extension to cleanup admin menu and Store > Configuration area by arranging third party extension items.

Clean Admin Menu - Magento 2 Extension It will merge all 3rd party extension's menu items in backend's primary menu to a common menu item named "Exten

RedChamps 109 Jan 3, 2023
Faker-driven, configuration-based, platform-agnostic, locale-compatible data faker tool

Masquerade Faker-driven, platform-agnostic, locale-compatible data faker tool Point Masquerade to a database, give it a rule-set defined in YAML and M

elgentos ecommerce solutions 219 Dec 13, 2022
provides a nested object property based user interface for accessing this configuration data within application code

laminas-config This package is considered feature-complete, and is now in security-only maintenance mode, following a decision by the Technical Steeri

Laminas Project 43 Dec 26, 2022
Mark Shust's Docker Configuration for Magento

markshust/docker-magento Mark Shust's Docker Configuration for Magento Table of contents Docker Hub Free Course Usage Prerequisites Setup Updates Cust

Mark Shust 1.8k Jan 8, 2023
Melek Berita Backend is a service for crawling data from various websites and processing the data to be used for news data needs.

About Laravel Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable and creative experie

Chacha Nurholis 2 Oct 9, 2022
❄️ Magento 2 Snowflake module allow you to add snow and even more on your site and make winter fun.

❄️ Magento 2 Snowflake module allow you to add snow and even more on your site and make winter fun.

OpenGento 6 Apr 30, 2022