Sspak - Tool for managing bundles of db/assets from SilverStripe environments

Related tags

Miscellaneous sspak
Overview

SSPak

Build Status SilverStripe supported module Code Quality

SSPak is a SilverStripe tool for managing database and assets content, for back-up, restoration, or transfer between environments.

The file format

An sspak file is either a Phar (executable) file or a Tar (non-executable) file, containing the following files at the top level:

  • database.sql.gz: A gzipped SQL file that will re-create the entire database, including all content. It will contain the 'drop' statements necessary to replace any existing content as needed.

  • assets.tar.gz: A gzipped tar file containing all assets. The root directory within the tar file must be called "assets".

  • git-remote: A text file of the following form:

    remote = (url) branch = (name) sha = (sha-hash)

By convention, the file should have the extension .sspak for non-executable versions, and .sspak.phar for executable versions.

Installation

You can run the installation script one of three ways.

Composer

You can install this package globally with Composer (ensure your composer bin is in your system path):

$> composer global require silverstripe/sspak:dev-master

cURL

If you have cURL, run this command (everything except for the $> part):

$> curl -sS https://silverstripe.github.io/sspak/install | php -- /usr/local/bin

The final argument is the directory that the script will be loaded into. If omitted, the script will be installed into the current directory. If you don't have permission to write to the directory, "sudo" will be used to escalate permissions.

For example, this would also work:

$> cd /usr/local/bin
$> curl -sS https://silverstripe.github.io/sspak/install | sudo php

Manually

If you prefer not to use the installer, you can download the script and copy it to your executable path as follows:

$> wget https://silverstripe.github.io/sspak/sspak.phar
$> chmod +x sspak.phar
$> sudo mv sspak.phar /usr/local/bin/sspak

Common Issues

Creating archive disabled by the php.ini setting phar.readonly

Set your phar.readonly setting to false in your php.ini (and php-cli.ini) files.

Use

All sspak commands take the following general form.

$> sspak (command) (from) (to)

Create an sspak file and save to /tmp:

$> sspak save /var/www /tmp/site.sspak

Create an sspak file based on a remote site:

$> sspak save me@prodserver:/var/www prod-site.sspak

Create an sspak file based on a remote site using a specific private key to connect:

$> sspak save --identity=prodserver.key me@prodserver:/var/www prod-site.sspak

Create an executable sspak file by adding a phar extension:

$> sspak save me@prodserver:/var/www prod-site.sspak.phar

Create an sspak from existing files:

$> sspak saveexisting --db=/path/to/database.sql --assets=/path/to/assets /tmp/site.sspak

Extract files from an existing sspak into the specified directory:

$> sspak extract /tmp/site.sspak /destination/path

Load an sspak file into a local instance:

$> sspak load prod-site.sspak ~/Sites/devsite

Load an sspak file into a local instance, dropping the existing DB first (mysql only):

$> sspak load prod-site.sspak ~/Sites/devsite --drop-db

Load an sspak file into a remote instance using a specific private key to connect:

$> sspak save --identity=backupserver.key prod-site.sspak me@backupserver:/var/www

Transfer in one step: (not implemented yet)

$> sspak transfer me@prodserver:/var/www ~/Sites/devsite

Sudo as www-data to perform the actions

$> sspak save --sudo=www-data me@prodserver:/var/www prod-site.sspak
$> sspak load --sudo=www1 prod-site.sspak ~/Sites/devsite
$> sspak transfer --from-sudo=www-data --to-sudo=www1 me@prodserver:/var/www ~/Sites/devsite

Save only the database:

$> sspak save --db me@prodserver:/var/www dev.sspak

Load only the assets:

$> sspak load --assets dev.sspak ~/Sites/devsite

Install a new site from an sspak (needs to contain a git-remote):

$> sspak install newsite.sspak ~/Sites/newsite

Save all while using a custom TMP folder (make sure the folder exists and is writable):

$> TMPDIR="/tmp/my_custom_tmp" sspak save /var/www /tmp/site.sspak

Caveats

If you don't have PKI passwordless log-in into remote servers, you will be asked for your log-in a few times.

How it works

sspak relies on the SilverStripe executable code to determine database credentials. It does this by using a small script, sspak-sniffer.php, which it uploads to the /tmp folder of any remote servers.

This script returns database credentials and the location of the assets path. Once it has that, it will remotely execute mysql, mysqldump and tar commands to archive or restore the content.

It expects the following commands to be available on any remote servers:

  • php
  • mysql
  • mysqldump
  • tar
  • gzip
  • sudo

It will also use the /tmp folder on the machine that you are running from, and it will need to have enough free space on there to create temporary copies of the individual files within the .sspak file, if you are using the non-executable version. .sspak.phar files can be populated without needing a tmp file in between.

Comments
  • sspak doesn't run on local SS instances with framework >=4.6

    sspak doesn't run on local SS instances with framework >=4.6

    Local like a local (webroot) - the dot in the middle ./vendor/bin/sspak save . ./test.sspak The sniffer script started to fail with: https://github.com/silverstripe/silverstripe-framework/commit/e08bf1cdd96ea2930b2a41162eaad384404c1672

    if DatabaseAdapterRegistry::autoconfigure($databaseConfig); is commented out, it runs trough https://github.com/silverstripe/silverstripe-framework/blob/4/src/Core/CoreKernel.php#L372 otherwise it has a hard time getting the cache https://github.com/silverstripe/silverstripe-framework/blob/4/src/Dev/Install/DatabaseAdapterRegistry.php#L184

    @emteknetnz - I have difficulties understanding how it fails. May you can shed some light on it?

    ACs

    • We preserve the fix that was put in earlier to cache the glob
    • You can run sspak /dev/build without flush on a brand new site without error
    • Confirm or deny if the $_GET['flush] on the code on earlier fix (see comment below) is required, and if CLI dev/build (with and without flush) stuff flushes the cache. Either update code or write a comment explaining usage of $_GET['flush] (possible that it's there so that things flush in a particuar order)
    • Confirm or deny if there's a wider issues where there's a difference between /dev/build and /dev/build?flush, and if so, raise a new issue
    type/bug affects/v4 complexity/medium impact/high 
    opened by lerni 17
  • Continuous builds

    Continuous builds

    Make travis publish our releases on successful merge.

    Things that need doing:

    1. Turning on travis (make sure you only test pushes and not PRs)
    2. Adding an encrypted environment variable GH_TOKEN to allow travis to push to the gh-pages branch
    opened by dhensby 12
  • Error - Can't process database type 'MySQLPDODatabase'

    Error - Can't process database type 'MySQLPDODatabase'

    When I run the following command

    $> sspak save --db /var/www/html/mysite local-test2.sspak

    then return the following Can't process database type 'MySQLPDODatabase'

    opened by robbyahn 9
  • FIX Always import to the correct DB

    FIX Always import to the correct DB

    This fixes an issue where a sql dump has a USE or CREATE statement.

    sspak won't handle that well and will end up creating a new DB instead of importing into the target DB.

    This change strips out lines starting with USE or CREATE to ensure that the intended DB is used.

    opened by dhensby 9
  • Fix install

    Fix install

    The install script needs a few tweaks:

    • It should run composer install if checked out from git.
    • It should check out the specific sha if the sha is different from the branch.
    • It should run dev/build, using the sudo user.

    The install script in self-extracting archives should also assume that the first argument is the current script, and limit actions to load and install.

    type/bug 
    opened by sminnee 6
  • sspak load destroys assets symlink

    sspak load destroys assets symlink

    I deploy a capistrano style with deployer php, which symlinks assets in each release.

    Using sspak load my.sspak /path/to/current destroys the assets symlink in current and generates an assets directory with the new content - which is lost in the next release.

    See https://github.com/silverstripe/sspak/blob/d697d767758d713952a4e0b21c0cd3b1088462e2/src/Webroot.php#L138-L157 where that happens.

    opened by wernerkrauss 5
  • ENHANCEMENT Support global composer installation

    ENHANCEMENT Support global composer installation

    • Adds a new bin location for composer so we can use sspak still as the bin filename
    • Rejigged readme to use # for titles, and added installation example with composer
    • Tweaked a couple of constant values to ensure composer-compatible sspak works with B/C over original

    Resolves #21

    opened by robbieaverill 5
  • Notice: Undefined index: REQUEST_URI

    Notice: Undefined index: REQUEST_URI

    Trying to save from a remote server with sspak save USER@SERVER:/home/[username]/public_html site.sspak, but getting this:

    Could not parse sspak-sniffer content:
    
    Notice: Undefined index: REQUEST_URI in /home/[username]/public_html/framework/control/Director.php on line 892
    <p>Redirecting to <a href="" title="Click this link if your browser does not redirect you">...</a></p>
    <meta http-equiv="refresh" content="1; url=" />
    <script type="text/javascript">setTimeout(function(){
        window.location.href = "";
    }, 50);</script>";
    
    opened by purplespider 5
  • Add --symlink option to install

    Add --symlink option to install

    For example:

    sspak install my.sspak /sites/mysite/releases/123 --symlink=/sites/mysite/www
    

    This would install to /sites/mysite/releases/123, run the post-installation script, and then create /sites/mysite/www as a symlink pointing to /sites/mysite/releases/123.

    type/enhancement 
    opened by sminnee 5
  • Unknown table 'COLUMN_STATISTICS' in information_schema (1109) error

    Unknown table 'COLUMN_STATISTICS' in information_schema (1109) error

    Getting an error when running sspak with MySQL 5.7.

    Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')
    FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'xxxxxxx' AND
    TABLE_NAME = 'Answer';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
    

    Might be a specific db setting, but passing in --column-statistics=0 to mysqldump fixes the issue. Will provide a PR.

    type/bug affects/v4 complexity/low impact/medium 
    opened by michalkleiner 4
  • Reduce number of opened SSH connections

    Reduce number of opened SSH connections

    Performing a simple sspak save --db user@remote:/home/site database.sspak command seems to open and close a total of 5 SSH connections.

    I work with a host who limit the maximum number of SSH connections from the same IP within 1 minute to just 4, so the sspak save command causes this limit to be hit and a short block applied.

    Not sure exactly why it requires 5 connections to download the database, is this an area that could be optimised?

    type/enhancement affects/v4 complexity/high impact/low 
    opened by purplespider 4
  • PHP Deprecated notice:  strtolower(): Passing null

    PHP Deprecated notice: strtolower(): Passing null

    Simply running sspak you'll see this at the top if you're on php8.1<

    PHP Deprecated:  strtolower(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///usr/local/bin/sspak/bin/sspak on line 37
    
    Deprecated: strtolower(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///usr/local/bin/sspak/bin/sspak on line 37
    
    type/bug complexity/low impact/low 
    opened by n8-dev 1
  • Audit multiple installation methods

    Audit multiple installation methods

    There are supposed to be multiple ways to install sspak: a) composer.json b) composer global require silverstripe/sspak c) installation script - curl -sS http://silverstripe.github.io/sspak/install | php -- /usr/local/bin d) direct download - sudo mv sspak.phar /usr/local/bin/sspak

    a) works to the best of my knowledge, though this should be confirmed

    b) requires suffixing :dev-master i.e. composer global require silverstripe/sspak:dev-master. You also need to have ~/.config/composer/vendor/bin in your $PATH for it work seamlessly e.g. sspak <commands>

    Some time ago, the binary in bin/sspak stopped being updated, presumably because a github token was revoked. That binary is now out of date, therefore c) and d) do not function as expected. This is very bad, since it means people may be installing an out of date version.

    My personal recommendation is that we should remove options c) and d) and force the use of composer, which is the standard way to install all things php. This is my view simply because I do not want this silently breaking again. Others may have a different view though.

    Acceptance criteria

    • [ ] Audit the 4 installation methods to see if they function properly
    • [ ] Release 1.0.0 so that the :dev-master suffix isn't required for b)
    • [ ] Investigate if Silverstripe Ltd platform uses sspak at all for extracting sspak files, and if so if it relies on either c) or d)
    • [ ] Investigate the amount of work required to get the sspak to regularly update via github actions on the tag github event (OR simply decide that we wish to only use composer) - if it's non-trivial then strongly recommended we just drop functionality
    • [ ] Remove the github pages - all documentation should be in the README.md instead
    • [ ] Update documentation to reflect the recommended ways to install sspak
    opened by emteknetnz 0
  • MNT Standardise modules

    MNT Standardise modules

    Issue https://github.com/silverstripeltd/product-issues/issues/570

    CI for this PR is run on the creative-commoners org: https://github.com/creative-commoners/sspak/actions

    opened by emteknetnz 3
  • Add option to not backup assets

    Add option to not backup assets

    I have a number of docker containers running with a volume bound to the public/assets directory, when using sspak load it tries to move the assets to assets-old but since the directory is a volume it fails. I've had success by commenting out

    https://github.com/silverstripe/sspak/blob/1d1ffff9af48b4ebd6d133830c325244e201695d/src/Webroot.php#L159

    but it's a manual step I have to do every time as sspak is baked into the image and the containers are frequently rebuilt.

    type/enhancement impact/low 
    opened by OldStarchy 2
  • Latest version failing to create temp build dir

    Latest version failing to create temp build dir

    After reinstalling sspak on my iMac, it would fail with an error about being unable to create the temp folder:

    $ sspak save --db user@host db.sspak
    
    Command: ssh -t 'user@host' 'mkdir '\''/var/folders/_m/9n7ryp613nz33y5bs9mqs9vc0000gn/T/sspak-680110'\'''
    Execution failed: returned 1.
    Output:
    mkdir: cannot create directory ‘/var/folders/_m/9n7ryp613nz33y5bs9mqs9vc0000gn/T/sspak-680110’: No such file or directory
    

    After trying many things to resolve it (such as testing that PHP could actually create a dir in the above location - it could), I copied the version of sspak from my laptop (into /usr/local/bin) and it works fine. So it seems the issue has been introduced into the current release at some point.

    Maybe this PR is related: https://github.com/silverstripe/sspak/pull/75 However I did try installing the latest version and adding TMPDIR="/tmp" to the command to see if it was the change in default temp directory that was causing it, but I still received the error.

    type/bug affects/v4 complexity/medium impact/medium 
    opened by purplespider 1
Owner
Silverstripe CMS
Silverstripe CMS is the intuitive content management system and flexible framework loved by editors and developers alike.
Silverstripe CMS
Pimcore Localized Assets - localize your assets with no duplicating files

Localized assets in Pimcore Pimcore Bundle to localize your assets with same file. Installation composer require lemonmind/pimcore-localized-assets bi

LemonMind.com 7 Aug 31, 2022
Silverstripe-masquerade - SilverStripe module to allow users to "masquerade" as other users

SilverStripe Masquerade Module About This module is designed to allow an Administrator to "login" as another "Member" without changing their password

Daniel Hensby 14 Apr 14, 2022
Silverstripe-debugbar/ - SilverStripe DebugBar module

SilverStripe DebugBar module Requirements SilverStripe ^4.0 maximebf/php-debugbar jdorn/sql-formatter Installation You can install the debug bar with

Thomas Portelange 52 Dec 21, 2022
Silverstripe-fulltextsearch - Adds external full text search engine support to SilverStripe

FullTextSearch module Adds support for fulltext search engines like Sphinx and Solr to SilverStripe CMS. Compatible with PHP 7.2 Important notes when

Silverstripe CMS 42 Dec 30, 2022
Silverstripe-searchable - Adds to the default Silverstripe search by adding a custom results controller and allowing properly adding custom data objects and custom fields for searching

SilverStripe Searchable Module UPDATE - Full Text Search This module now uses Full Text Support for MySQL/MariaDB databases in version 3.* Adds more c

ilateral 13 Apr 14, 2022
A SilverStripe module for conveniently injecting JSON-LD metadata into the header of each rendered page in SilverStripe

A SilverStripe module for conveniently injecting JSON-LD metadata into the header of each rendered page in Silver

null 4 Apr 20, 2022
Component bundles with symfony (php)

Component bundles with symfony (php)

Chr1s # 3 Jun 23, 2022
Envbar allows you to differentiate between environments by adding a custom colored bar above the top navigation.

Envbar Envbar allows you to differentiate between environments by adding a custom colored bar above the top navigation. This should help backend users

Magenizr 6 Oct 7, 2022
Warden is a CLI utility for orchestrating Docker based developer environments

Warden Warden is a CLI utility for orchestrating Docker based developer environments, and enables multiple local environments to run simultaneously wi

David Alger 314 Dec 2, 2022
SlimJim was born out of a need for a simple auto update script which would update multiple development/test environments every time someone

SlimJim WHY? SlimJim was born out of a need for a simple auto update script which would update multiple development/test environments every time someo

Jesal Gadhia 100 Apr 22, 2022
Create and update progress bars in different environments

Create and update progress bars in different environments

Laminas Project 8 Jul 28, 2022
A tool for managing SSH key access to any number of servers.

Revons - SSH Key Authority Features Easily manage SSH key access for all accounts on your servers. Manage user access and server-to-server access rule

Revons Community 1 Mar 14, 2022
Plugin for Kirby that allows you to load assets generated by Vite.

Kirby Vite Plugin Plugin for Kirby that allows you to load assets generated by Vite. In development mode, assets are loaded from Vite's development se

Oblik Studio 10 Nov 20, 2022
A Magento 1.x module which facilitates automatic purging of static assets from HTTP caches such as browser cache, CDN, Varnish, etc using best practices outlined within the HTML5 boilerplate community.

Magento Cachebuster Cachebuster is a Magento module which facilitates automatic purging of static assets from HTTP caches such as browser cache, CDN,

Gordon Knoppe 129 Apr 1, 2022
SilverStripe Garbage Collection Module

SilverStripe Module for defining and processing Garbage Collection on SilverStripe Applications.

Brett Tasker 8 Aug 12, 2022
Markdownfield - Markdown field for SilverStripe

MarkdownField This module introduces a new DB field type Markdown & Markdown Editor. It uses github style Markdown style. And uses the simple markdown

SilverStripers 10 Jul 10, 2022
Automatically delete old SiteTree page versions from Silverstripe

Version truncator for Silverstripe An extension for Silverstripe to automatically delete old versioned DataObject records from your database when a re

Ralph Slooten 35 Dec 7, 2022
Silverstripe-populate - Populate your database through YAML files

Populate Module This module provides a way to populate a database from YAML fixtures and custom classes. For instance, when a building a web applicati

Silverstripe CMS 22 Oct 3, 2022
Silverstripe-ideannotator - Generate docblocks for DataObjects, Page, PageControllers and (Data)Extensions

silverstripe-ideannotator This module generates @property, @method and @mixin tags for DataObjects, PageControllers and (Data)Extensions, so ide's lik

SilverLeague 44 Dec 21, 2022