Silverstripe-populate - Populate your database through YAML files

Overview

Populate Module

Build Status

This module provides a way to populate a database from YAML fixtures and custom classes. For instance, when a building a web application the pages and default objects can be defined in YAML and shared around developers. This extends the requireDefaultRecords concept in SilverStripe's DataModel.

Requirements

Installation Instructions

This module must only ever be used in your development environment, and should never be used on production. While there is code to prevent it from being run in production, it is not fool-proof and therefore you must never run this module in production. Install it as a dev dependency in composer like so:

composer require --dev dnadesign/silverstripe-populate

Setup

First create a new yml config file in your config directory app/_config/populate.yml (or add it to an existing config.yml file if you prefer).

DNADesign\Populate\Populate:
  include_yaml_fixtures:
    - 'app/fixtures/populate.yml'

If you're sharing test setup with populate, you can specify any number of paths to load fixtures from.

An example app/fixtures/populate.yml might look like the following:

Page:
  home:
    Title: "Home"
    Content: "My Home Page"
    ParentID: 0
SilverStripe\Security\Member:
  admin:
    ID: 1
    Email: "[email protected]"
    PopulateMergeMatch:
      - 'ID'
      - 'Email'

Out of the box, the records will be created on when you run the PopulateTask through /dev/tasks/PopulateTask/. To make it completely transparent to developers during the application build, you can also include this to hook in on requireDefaultRecords as part of dev/build by including the following in one of your application models requireDefaultRecords methods:

use DNADesign\Populate\Populate;

class Page extends SiteTree
{
    public function requireDefaultRecords()
    {
        parent::requireDefaultRecords();
        Populate::requireRecords();
    }
}

Configuration options

include_yaml_fixtures

An array of YAML files to parse.

mysite/_config/app.yml

DNADesign\Populate\Populate:
  include_yaml_fixtures:
    - 'app/fixtures/populate.yml'

truncate_objects

An array of ClassName's whose instances are to be removed from the database prior to importing. Useful to prevent multiple copies of populated content from being imported. It's recommended to truncate any objects you create, to ensure you can re-run PopulateTask as often as you want during development and get a consistent database state. This supports Versioned objects (like SiteTree) and Fluent (if the module is installed).

DNADesign\Populate\Populate:
  truncate_objects:
    - Page
    - SilverStripe\Assets\Image

truncate_tables

An array of tables to be truncated. Useful when there's no relation between your populated classes and the table you want truncated

DNADesign\Populate\Populate:
  truncate_tables:
    - Image_Special_Table

See Updating Records if you wish to merge new and old records rather than clearing all of them.

YAML Format

Populate uses the same FixtureFactory setup as SilverStripe's unit testing framework. The basic structure of which is:

ClassName:
  somereference:
    FieldName: "Value"

Relations are handled by referring to them by their reference value:

SilverStripe\Security\Member:
    admin:
      Email: "[email protected]"

Page:
  homepage:
    AuthorID: =>SilverStripe\Security\Member.admin

See SilverStripe's fixture documentation for more advanced examples, including $many_many and $many_many_extraFields.

Any object which implements the Versioned extension will be automatically published.

Basic PHP operations can also be included in the YAML file. Any line that is wrapped in a ` character and ends with a semi colon will be evaled in the current scope of the importer.

Page:
  mythankyoupage:
    ThankYouText: "`Page::config()->thank_you_text`;"
    LinkedPage: "`sprintf(\"[Page](%s)\", App\\Page\\HelpPage::get()->first()->Link())`;"

Updating Records

If you do not truncate the entire table, the module will attempt to first look up an existing record and update that existing record. For this to happen the YAML must declare the fields to match in the look up. You can use several options for this.

PopulateMergeWhen

Contains a WHERE clause to match e.g "URLSegment = 'home' AND ParentID = 0".

Mysite\PageTypes\HomePage:
  home:
    Title: "My awesome homepage"
    PopulateMergeWhen: "URLSegment = 'home' AND ParentID = 0"

PopulateMergeMatch

Takes a list of fields defined in the YAML and matches them based on the database to avoid repeating content

Mysite\PageTypes\HomePage:
  home:
    Title: "My awesome homepage"
    URLSegment: 'home'
    ParentID: 0
    PopulateMergeMatch:
      - URLSegment
      - ParentID

PopulateMergeAny

Takes the first record in the database and merges with that. This option is suitable for things like SiteConfig where you normally only contain a single record.

SilverStripe\SiteConfig\SiteConfig:
  mysiteconfig:
    Tagline: "SilverStripe is awesome"
    PopulateMergeAny: true

If the criteria meets more than 1 instance, all instances bar the first are removed from the database so ensure you criteria is specific enough to get the unique field value.

Default Assets

The script also handles creating default File and image records through the PopulateFileFrom flag. This copies the file from another path (say mysite) and puts the file inside your assets folder.

SilverStripe\Assets\Image:
  lgoptimusl3ii:
    Filename: assets/shop/lgoptimusl3ii.png
    PopulateFileFrom: app/images/demo/large.png

Mysite\PageTypes\Product:
  lgoptimus:
    ProductImage: =>SilverStripe\Assets\Image.lgoptimusl3ii

Extensions

The module also provides extensions that can be opted into depending on your project needs

PopulateMySQLExport

This extension outputs the result of the Populate::requireDefaultRecords() as a SQL Dump on your local machine. This speeds up the process if using Populate as part of a test suite or some other CI service as instead of manually calling the task (which will use the ORM) your test case can be fed raw MySQL to import and hopefully speed up execution times.

To apply the extension add it to Populate, configure the path, flush, then run dev/tasks/PopulateTask

DNADesign\Populate\PopulateMySQLExportExtension:
  export_db_path: ~/path.sql

DNADesign\Populate\Populate:
  extensions
    - DNADesign\Populate\PopulateMySQLExportExtension

Publish configuration

By default the module uses publishSingle() to publish records. If, for whatever reason, you would prefer to that the module uses publishRecursive(), you can enable this by settings the following configuration:

DNADesign\Populate\Populate:
  enable_publish_recursive: true

Allow Populate to run on "live" environments

DANGER ZONE: Please understand that you are about to provide admins with the ability to run Populate on your production environment. Before setting this configuration you should understand and accept the risks related to the loss of production data.

DNADesign\Populate\Populate:
  allow_build_on_live: true

Credits

silverstripe-populate was originally created by wilr and DNA Design.

Comments
  • Support fluent and clean up

    Support fluent and clean up

    This allows you to now support using the module with fluent It also splits out objects into classes which we can find the tables for and then tables which are just truncated Task is now run from populate-task and documented correctly for running it

    opened by adrhumphreys 11
  • Update module to support SilverStripe 4

    Update module to support SilverStripe 4

    • Namespace code
    • Update tests to work with SS4, try and setup travis to see if it works
    • Update Travis to tests against 5.6, 7.0 and 7.1
    • Update README with new instructions
    • Move module to sit inside vendor/ instead of in webroot
    opened by madmatt 6
  • Feature: Allow Populate to build on live environments via config

    Feature: Allow Populate to build on live environments via config

    Closes #28

    By default, builds are allowed on dev and test, but disallowed on live. This PR does not change that default behaviour.

    Added a new configuration for allow_build_on_live which allows developers to specify that builds are also allowed on live.

    While I won't be using this configuration myself, I couldn't really think of a valid reason for the module to make this determination of behalf of all developers. If a developer wishes to allow builds on live, then (imo) we should trust that they have understood the risks, and allow them to do it.

    opened by chrispenny 3
  • Remove use of deprecated publish method. Plus linting fixes

    Remove use of deprecated publish method. Plus linting fixes

    The functional change is in PopulateFactory.php at line 186. Previously we were using the deprecated publish() method. This has been updated to use the publishSingle() method by default (keeping backwards compatibility), but I've also added a config enable_publish_recursive which devs can opt in to if they prefer.

    No functional changes to Populate.php or PopulateFactoryTest.php. These changes are purely to align them with the project's .editorconfig.

    opened by chrispenny 3
  • Allow Populate::requireRecords() to run in live mode

    Allow Populate::requireRecords() to run in live mode

    It would be really helpful if there was a config setting or parameter that could be passed into the Populate::requireRecords() method to allow this to be run in live mode :)


    Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

    opened by timkung 2
  • Subsites compatibility

    Subsites compatibility

    Hey,

    I've run into an issue using populate with subsites.

    Each subsite is able to have pages with the same URL segment, for example:

    mainsite.com/home mysubsite.com/home

    Populate handles URL segment collisions like the CMS does, by prefixing the page id, so you get home-2 on the subsite URL segment.

    Here's a example populate.yml

    Subsite:
      MySubsite:
        Title: 'My Subsite'
        Theme: 'mysubsite'
    
    SubsiteDomain:
      MySubsiteDomain:
        Domain: 'mysubsite.com'
        Protocol: 'automatic'
        Subsite: =>Subsite.MySubsite
    
    Page:
      Home:
        ParentID: 0
        ShowInMenus: false
        Title: 'Home'
        URLSegment: 'home'
      SubsiteHome:
        ParentID: 0
        ShowInMenus: false
        Title: 'Home'
        URLSegment: 'home'
       Subsite: =>Subsite.MySubsite
    

    I'm currently working around this by creating an DataExtension on Populate and updating the records manually.

    class PopulateExtension extends DataExtension
    {
        public function onAfterPopulateRecords()
        {
            // Do stuff
        }
    }
    

    Is there a way I can populate pages across subsites without having to use the onAfterPopulateRecords hook?

    Thanks


    Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

    opened by flashbackzoo 2
  • fixing an issue with PopulateMergeMatch giving errors on db rebuild

    fixing an issue with PopulateMergeMatch giving errors on db rebuild

    I was experiencing som issues using the PopulateMergeMatch - it basically didn't let me use it, and once I fixed that (see line 20), it started deleting my entries, which I don't believe should be the purpose, hence i introduced $mode in line 35. I furthermore had to unset PopulateMergeMatch, as the object couldn't be saved with an array (line 56).

    I don't know if you can reproduce the issue? I'll use this branch for now as it works for me like this.

    This is the yml I'm using:

    VermarktungsartPrototype:
      KAUF:
        Title: "Kauf"
        Identifier: "KAUF"
        PopulateMergeMatch:
          - Identifier
      MIETE_PACHT:
        Title: "Miete/Pacht"
        Identifier: "MIETE_PACHT"
        PopulateMergeMatch:
          - Identifier
    
    opened by anselmdk 2
  • MNT: PHP 8, switch from Tavis to Github Actions

    MNT: PHP 8, switch from Tavis to Github Actions

    • Remove Travis - switch to Github Actions
    • Minimum PHP requirement is now 7.4
    • PHP ^8 support added
      • Really just had to add some null coalescing operators in some spots
    • Upgrade to PHP Unit ^9
      • Some very minor test contract changes (no functional changes)
    • pgsql tests were failing due to PopulateFactory::createObject().
    opened by chrispenny 1
  • Unable to create PDF file

    Unable to create PDF file

    Unable to create a file (not image) with populate because this line uses Image class.

    https://github.com/silverstripe/silverstripe-populate/blob/718b2a5d4df8015e7c2e0ef481d3353fb58dbf11/code/PopulateFactory.php#L254

    opened by satrun77 1
  • Bugfix: Update PHP, fix composer requirement, remove use of deprecated publish() method

    Bugfix: Update PHP, fix composer requirement, remove use of deprecated publish() method

    Bugfix: Remove use of deprecated publish method

    This was originally covered in #30 but that also included Linting fixes, which maybe expanded the scope to far.

    Feature, publish recursive: Added a basic config that devs can set if they would like to use publishRecursive() instead.

    Update PHP Version: Only PHP 7.1+ is supported by framework.

    Fix composer requirement: Incorrect name used for phpunit/phpunit. Composer now just breaks if you have capitalisation.

    opened by chrispenny 1
  • Update Readme for PHP operations

    Update Readme for PHP operations

    The backtick approach in yml does not work unless wrapped in double quotes. Also added an example of a namespaced class as its useful to note the backslashes should be escaped.

    opened by scott1702 1
  • Feature Request: truncate_objects should also delete Fluent's Localised table

    Feature Request: truncate_objects should also delete Fluent's Localised table

    If a DataObject has Fluent extension applied, it'd be nice, if the table defined by fluent could be truncated automatically.

    E.g.

    SiteTree_Localised SiteTree_Localised_Versions etc...

    Would be a huge timesaver, so I don't have to define all the tables manually.

    opened by wernerkrauss 0
  • fix: correct $segment for PopulateTask per docs

    fix: correct $segment for PopulateTask per docs

    @GuySartorelli as per https://github.com/silverstripe/silverstripe-staticpublishqueue/commit/2f05ec1ec3ca0837b258564dc30034676f2d67d1 it looks like this class was upgraded without one defined but the documentation refers to the old version. To review if we should mark this as a API change before merge. I don't think this would be as drastic as the above one since it's unlikely anyones running it in production (only perhaps for demo sites)

    opened by wilr 2
  • MNT: Move to Github Actions dynamic matrix

    MNT: Move to Github Actions dynamic matrix

    Blocker at the moment is that PostgreSQL tests are failing: https://github.com/silverstripe/silverstripe-populate/actions/runs/3146840126/jobs/5115740952

    opened by chrispenny 1
  • Allows PopulateMergeMatch to run with Images

    Allows PopulateMergeMatch to run with Images

    Fixes #45

    I don't know this library well enough to conclusively say that no other piece of code is expecting a boolean instead of an object, but this makes sense to me and it has fixed the issue I was having.

    opened by davejtoews 0
  • Images cause error when repeatedly running populate task.

    Images cause error when repeatedly running populate task.

    Version 2.1.0

    Relevant portion of my yml

    SilverStripe\Assets\Image:
      imageOne:
        Created: "1970-01-02 03:04:05"
        Filename: assets/TestContent/imageBlockOne.jpg
        PopulateFileFrom: app/fixtures/assets/546-900x600.jpg
        PopulateMergeMatch:
          - Created
    
    App\Elements\ImageElement:
      imageBlockOne:
        Title: Image Element
        ShowTitle: true
        Created: "1970-01-02 03:04:05"
        ImageID: =>SilverStripe\Assets\Image.imageOne
        ParentID: =>DNADesign\Elemental\Models\ElementalArea.imageArea
        PopulateMergeMatch:
          - Created
    

    This runs fine the first time, but on the second pass creating the image results in the following:

    [Notice] Trying to get property 'ID' of non-object Line 72 in /var/www/html/vendor/dnadesign/silverstripe-populate/code/PopulateFactory.php

    And then attempting to reference the image results in:

    [Emergency] Uncaught InvalidArgumentException: No fixture definitions found for "=>SilverStripe\Assets\Image.imageOne" Line 332 in /var/www/html/vendor/silverstripe/framework/src/Dev/FixtureBlueprint.php

    opened by davejtoews 0
  • Bugfix - Prevents error when property is not defined

    Bugfix - Prevents error when property is not defined

    This property name was changed in a previous commit w/o changing the references to it in other sections of the code.

    The mismatch results in an error if the $truncate_objects value is not set in a config yml.

    Whether or not the name change makes sense, this is a breaking change that would require a major version bump so I think it should be reverted to the original.

    Fixes #42

    opened by davejtoews 0
Releases(2.2.0)
  • 2.2.0(Oct 3, 2022)

    What's Changed

    • Added more keywords to composer.json by @Zauberfisch in https://github.com/silverstripe/silverstripe-populate/pull/39
    • Fix creating file instance using fixture file extension by @satrun77 in https://github.com/silverstripe/silverstripe-populate/pull/41
    • MNT: PHP 8, switch from Tavis to Github Actions by @chrispenny in https://github.com/silverstripe/silverstripe-populate/pull/47

    New Contributors

    • @satrun77 made their first contribution in https://github.com/silverstripe/silverstripe-populate/pull/41

    Full Changelog: https://github.com/silverstripe/silverstripe-populate/compare/2.1.0...2.2.0

    Source code(tar.gz)
    Source code(zip)
  • 2.1.0(Oct 21, 2021)

    What's Changed

    • Feature: Allow circular dependency resolution by @madmatt in https://github.com/silverstripe/silverstripe-populate/pull/26
    • Improvement: Add PHP Codesniffer. Update PHPUnit to support 7.4 by @chrispenny in https://github.com/silverstripe/silverstripe-populate/pull/36
    • Feature: Allow Populate to build on live environments via config by @chrispenny in https://github.com/silverstripe/silverstripe-populate/pull/37

    Full Changelog: https://github.com/silverstripe/silverstripe-populate/compare/2.0.1...2.1.0

    Source code(tar.gz)
    Source code(zip)
  • 2.0.1(Oct 20, 2021)

    What's Changed

    • This module should be a dev requirement by @wernerkrauss in https://github.com/silverstripe/silverstripe-populate/pull/31
    • Update Readme for PHP operations by @scott1702 in https://github.com/silverstripe/silverstripe-populate/pull/29
    • Add credits to README now that module is in silverstripe org by @madmatt in https://github.com/silverstripe/silverstripe-populate/pull/34
    • Bugfix: Update PHP, fix composer requirement, remove use of deprecated publish() method by @chrispenny in https://github.com/silverstripe/silverstripe-populate/pull/33
    • Support fluent and clean up by @adrhumphreys in https://github.com/silverstripe/silverstripe-populate/pull/32

    New Contributors

    • @chrispenny made their first contribution in https://github.com/silverstripe/silverstripe-populate/pull/33
    • @adrhumphreys made their first contribution in https://github.com/silverstripe/silverstripe-populate/pull/32

    Full Changelog: https://github.com/silverstripe/silverstripe-populate/compare/2.0.0...2.0.1

    Source code(tar.gz)
    Source code(zip)
Owner
Silverstripe CMS
Silverstripe CMS is the intuitive content management system and flexible framework loved by editors and developers alike.
Silverstripe CMS
Silverstripe-masquerade - SilverStripe module to allow users to "masquerade" as other users

SilverStripe Masquerade Module About This module is designed to allow an Administrator to "login" as another "Member" without changing their password

Daniel Hensby 14 Apr 14, 2022
Silverstripe-debugbar/ - SilverStripe DebugBar module

SilverStripe DebugBar module Requirements SilverStripe ^4.0 maximebf/php-debugbar jdorn/sql-formatter Installation You can install the debug bar with

Thomas Portelange 52 Dec 21, 2022
Silverstripe-fulltextsearch - Adds external full text search engine support to SilverStripe

FullTextSearch module Adds support for fulltext search engines like Sphinx and Solr to SilverStripe CMS. Compatible with PHP 7.2 Important notes when

Silverstripe CMS 42 Dec 30, 2022
Silverstripe-searchable - Adds to the default Silverstripe search by adding a custom results controller and allowing properly adding custom data objects and custom fields for searching

SilverStripe Searchable Module UPDATE - Full Text Search This module now uses Full Text Support for MySQL/MariaDB databases in version 3.* Adds more c

ilateral 13 Apr 14, 2022
A SilverStripe module for conveniently injecting JSON-LD metadata into the header of each rendered page in SilverStripe

A SilverStripe module for conveniently injecting JSON-LD metadata into the header of each rendered page in Silver

null 4 Apr 20, 2022
Sync Wordpress Pages and Posts (even custom post types + fields) from static Markdown + YAML files

Sync Markdown Files to WordPress Posts and Pages Static site generators let you use a revision-controlled tree of markdown files to make a site, but d

null 26 Sep 26, 2022
More options when uploading files such as name changes, resizing or compression through TinyPNG.

Kirby Upload Extended More options when uploading files like name changes, resizing via Kirby or compression and optional resizing via TinyPNG. Thanks

Oli 24 Nov 12, 2022
phalcon config loader for yaml

Phalcon Config Loarder for Yaml Loads all the yml in the directory of the app/config. Version PHP: 7.0.x, 7.1.x, 7.2.x Phalcon: 3.x Composer { "r

Toshiyuki Ienaga 2 Oct 7, 2022
SilverStripe Garbage Collection Module

SilverStripe Module for defining and processing Garbage Collection on SilverStripe Applications.

Brett Tasker 8 Aug 12, 2022
Silverstripe-sspy - Python based SSPAK export with higher reliability and cross-platform compatibility

SSPY - Python Stand-alone SSPAK solution © Simon Firesphere Erkelens; Moss Mossman Cantwell Usage: sspy [create|load|extract] (db|assets) --file=my.

Simon Erkelens 1 Jun 29, 2021
Sspak - Tool for managing bundles of db/assets from SilverStripe environments

SSPak SSPak is a SilverStripe tool for managing database and assets content, for back-up, restoration, or transfer between environments. The file form

Silverstripe CMS 45 Dec 14, 2022
Markdownfield - Markdown field for SilverStripe

MarkdownField This module introduces a new DB field type Markdown & Markdown Editor. It uses github style Markdown style. And uses the simple markdown

SilverStripers 10 Jul 10, 2022
Automatically delete old SiteTree page versions from Silverstripe

Version truncator for Silverstripe An extension for Silverstripe to automatically delete old versioned DataObject records from your database when a re

Ralph Slooten 35 Dec 7, 2022
Silverstripe-ideannotator - Generate docblocks for DataObjects, Page, PageControllers and (Data)Extensions

silverstripe-ideannotator This module generates @property, @method and @mixin tags for DataObjects, PageControllers and (Data)Extensions, so ide's lik

SilverLeague 44 Dec 21, 2022
This module integrates Silverstripe CMS with Google Translate API and then allows content editors to use automatic translation for every translatable field.

Autotranslate This module integrates Silverstripe CMS with Google Translate API and then allows content editors to use automatic translation for every

null 4 Jan 3, 2022
Alerts users in the SilverStripe CMS when multiple people are editing the same page.

Multi-User Editing Alert Alerts users in the SilverStripe CMS when multiple people are editing the same page. Maintainer Contact Julian Seidenberg <ju

Silverstripe CMS 15 Dec 17, 2021
Silverstripe-tinytidy - Control which styles are available in TinyMCE's style dropdown menu and what elements they can be applied to

TinyTidy for SilverStripe This module mainly serves as an example of how to customise the 'styles' dropdown menu in the TinyMCE editor to control whic

Jono Menz 30 Jul 30, 2020
Silverstripe module allowing editors to create newsletters using elemental blocks and export them to a sendy instance

Silverstripe Sendy Silverstripe module allowing editors to create newsletters using elemental blocks and export them to a sendy instance. Introduction

Syntro Opensource 4 Apr 20, 2022