Flysystem storage with local metadata storage for speed and manageability.

Overview

Laravel Filer

This project was started to scratch my itch on our growing Laravel site:

  • Metadata for all files is stored in a local repository - Supported backing systems are json, database, and memory (for testing). This is designed to speed up certain operations which normally call out to the remote filesystems.
  • Handles fallback to the original disk - If you provide an "original disk" list, this adapter will attempt to import data from those original disks into the metadata repository.
  • Pluggable Strategies - While the current version ships with a single strategy, you can replace the Basic implementation which allows for 1 + async, mirror, or other interactions with backing storage adapters.
  • Manage data + files - Coming soon: query and manage the metadata to do things like:
    • Find files stored on a single service and mirror them
    • Migrate files between stores while still maintaining continuity
  • Abstract data from metadata - Planning at some point on allow things like deduplication and copy-on-write to make copies, renames, and deletions work better.

Getting Started

To get started, require the project:

composer require nvahalik/laravel-filer

Once that's done, you'll need to edit the filer config file and then update your filesystem configuration.

Config File

By default, the metadata is stored in a JSON file. You can edit config/filer.php to change the default storage mechanism from json to database or memory. Note that memory is really a null adapter. The JSON adapter wraps memory and just serializes and saves it after each operation.

Filesystem Configuration

The configuration is very similar to other disks:

'example' => [
    'driver' => 'filer',
    'original_disks' => [
        'test:s3-original',
    ],
    'id' => 'test',
    'disk_strategy' => 'basic',
    'backing_disks' => [
        'test:s3-new',
        'test:s3-original',
    ],
    'visibility' => 'private',
],

The original_disks is an option if you are migrating from an existing disk or disks to the filer system. Effectively, this is a fallback so that files which are not found in the local metadata store will be searched for in original_disks. If they are found, their metadata will be imported. If not, the file will be treated as missing. We'll cover doing mass importing of metadata later on.

Note: that this will slow the filesystem down until the cache is filled. Once the cache is loaded, you can remove these original_disks and those extra operations looking for files will be eliminated.

Note 2: files which are truly missing do not get cached. Therefore, if a file is missing, and you repeatedly attempt to assert its existence, it will search over and over again. This could be improved by caching the results or likewise having some sort of missing-files cache.

id is just an internal ID for the metadata store. File duplications are not allowed within the metadata of a single id, for example, but would be allowed for different ids.

disk_strategy has only a single option currently: 'basic' but will be pluggable to allow for different strategies to be added and used. The basic strategy simply writes to the first available disk from the list of provided backing_disks.

backing_disks allows you to define multiple flysystem disks to use. Want to use multiple S3-compatible adapters? You can. Note that for the basic adapter, the order of the disks determines the order in which they are tried.

A couple of examples

Given the configuration above, if the following code is run:

Storage::disk('example')->has('does/not/exist.txt');
  1. The existing metadata repo will be searched.
  2. Then, the single 'original_disks' will be searched.
  3. Finally, the operation will fail.
Storage::disk('example')->put('does/not/exist.txt', 'content');
  1. The existing metadata repo will be searched.
  2. Then, the single 'original_disks' will be searched.
  3. Then, a write will be attempted on test:s3-new.
  4. If that fails, then a write will be attempted on test:s3-original.
  5. If any of the writes succeeds, that adapter's backing information will be returned and the entries metadata updated.
  6. If any of them fails, then false will be returned and the operation will have failed.

Importing Metadata

If you already have a ton of files on S3, you can use the filer:import-s3-metadata command to import that data into your metadata repository:

# Grab the existing contents.
s3cmd ls s3://bucket-name -rl > s3output.txt

# Import that data into the "example" storageId.
php artisan filer:import-s3-metadata example s3output.txt

The importer uses File::lines() to load its data, and therefore should not consume a lot of memory. Additionally, it will look at the bucket name in the URL which is present in the output and attempt to find that within your existing filesystems config.

Visibility

By default, it will grab this from the filesystem configuration. If none is found nor provided with --visibility, it will default to private.

Filename stripping

You can strip a string from the filenames by specifying the --strip option.

Disk

If you need to specify the disk directly or want to otherwise override it, just pass it in with --disk. This is not checked, so don't mess it up.

Example

php artisan filer:import-s3-metadata example s3output.txt --disk=some-disk --visibility=public --strip=prefix-dir/ 

The above command would strip prefix-dir/ from the imported URLs, set their visibility to public, and mark their default backing-disk to some-disk.

You might also like...
Google Cloud Storage filesystem driver for Laravel
Google Cloud Storage filesystem driver for Laravel

Google Cloud Storage filesystem driver for Laravel Google Cloud Storage filesystem driver for Laravel. This started as a fork from Superbalist/laravel

This package provides an integration with FFmpeg for Laravel. Laravel's Filesystem handles the storage of the files.

Laravel FFMpeg This package provides an integration with FFmpeg for Laravel 6.0 and higher. Laravel's Filesystem handles the storage of the files. Lau

Laravel Package - Files to S3 Like Cloud Storage

This is a Laravel package that handles and make easy the upload, overwrite, delete, cdn purge of files and directories on AWS S3 like cloud storages. It supports a form uploaded file (UploadedFile) or a base64 file string

A Laravel 8 and Livewire 2 demo showing how to search and filter by tags, showing article and video counts for each tag (Polymorphic relationship)
A Laravel 8 and Livewire 2 demo showing how to search and filter by tags, showing article and video counts for each tag (Polymorphic relationship)

Advanced search and filter with Laravel and Livewire A demo app using Laravel 8 and Livewire 2 showing how to implement a list of articles and tags, v

Laravel package to generate and to validate a UUID according to the RFC 4122 standard. Only support for version 1, 3, 4 and 5 UUID are built-in.

Laravel Uuid Laravel package to generate and to validate a universally unique identifier (UUID) according to the RFC 4122 standard. Support for versio

Boilerplate code for protecting a form with proof of work. Uses javascript in the browser to generate the hashcash and PHP on the server to generate the puzzle and validate the proof of work.

Boilerplate code for protecting a form with proof of work. Uses javascript in the browser to generate the hashcash and PHP on the server to generate the puzzle and validate the proof of work.

List of 77 languages for Laravel Framework 4, 5, 6, 7 and 8, Laravel Jetstream , Laravel Fortify, Laravel Breeze, Laravel Cashier, Laravel Nova and Laravel Spark.

Laravel Lang In this repository, you can find the lang files for the Laravel Framework 4/5/6/7/8, Laravel Jetstream , Laravel Fortify, Laravel Cashier

Stop duplicating your Eloquent query scopes and constraints in PHP. This package lets you re-use your query scopes and constraints by adding them as a subquery.

Laravel Eloquent Scope as Select Stop duplicating your Eloquent query scopes and constraints in PHP. This package lets you re-use your query scopes an

symfony workflow component for laravel7 and 8 ,php 7 and 8

Laravel workflow Use the Symfony Workflow component in Laravel8,PHP7,PHP8 This repository based on @brexis,his project since 2019-09 No longer maintai

Comments
  • Support Flysystem 2/Laravel 9

    Support Flysystem 2/Laravel 9

    Support for Flysystem 2 and Laravel 9 will require additional changes:

    https://flysystem.thephpleague.com/docs/upgrade-from-1.x/

    We touch the Flysystem API in the following spots:

    • [ ] src/AdapterStrategy/AdapterStrategy.php
    • [ ] src/AdapterStrategy/Basic.php
    • [ ] src/Console/Commands/ImportMetadata.php
    • [ ] src/FilerServiceProvider.php
    • [ ] src/Flysystem/FlysystemAdapter.php
    • [ ] src/Metadata.php
    opened by nvahalik 0
  • Enable file migration

    Enable file migration

    Here's the use-case:

    I've been using a particular S3 provider for some time, but I'm worried about their stability. It would be great if their data could be migrated from one provider to another provider over time.

    Here are 2 possible ways to handle this:

    1. Batch system -> migration jobs
    2. Manual migration

    Batch System -> migration jobs

    1. Create a command or some ability to provide a list of metadata records (e.g. using the existing Metadata repository as a source).
    2. Create a job which loads the data and writes or streams that data from one location to another.
    3. Create a batch filled with jobs which handles the migration of the data and the updating of the backing store information.
    4. Implement some means to see/view the migration process.

    Slower, but could be integrated into the tooling of the module directly.

    Manual migration

    1. Using a tool such as s3cmd/s5cmd/s3p, copy the files to/from the source/destination buckets.
    2. Create a new manifest using s3cmd/s5cmd (see existing metadata import).
    3. Update using CLI tools the files within the manifests to point to the new backing store.
    4. Then, once tested, remove the backing store information from the metadata repository.

    Faster, but requires careful managing of the data.

    opened by nvahalik 0
  • Support copying files

    Support copying files

    While copying files isn't part of my initial use case, it would be good to support copying files. The biggest question is how?

    1. Support something like copy-on-write by just creating a new metadata entry, setting some metadata indicating that this is a shared copy, and then updating adapter manager accordingly to write to a new file or copy it before any write takes place.
    2. Just copy the file on the adapters, honoring the rules and supporting the ability to copy to the same or to specify a specific backing adapter.

    Currently, the rename functionality doesn't actually touch the backing adapters. It just changes the path.

    opened by nvahalik 0
Owner
Nick Vahalik
Nick Vahalik
Aliyun oss filesystem storage adapter for laravel 5. You can use Aliyun OSS just like laravel Storage as usual

Aliyun oss filesystem storage adapter for laravel 5. You can use Aliyun OSS just like laravel Storage as usual

jacob 517 Dec 29, 2022
Flow package to synchronize metadata and binary data of imported Neos.Media assets

Wwwision.AssetSync Flow package to synchronize metadata and resources of imported Neos.Media assets Installation Install this package via: composer re

Bastian Waidelich 5 Feb 7, 2022
Load head metadata from a manifest file which can be shared with a SPA project

Laravel Head Manifest Installation Step 1: Add Laravel Head Manifest to your laravel project composer require critiq/laravel-head-manifest Step 2: Add

Critiq 1 Nov 17, 2021
Retrieve the EC2 Metadata using Laravel's eloquent syntax.

Laravel EC2 Metadata Retrieve the EC2 Metadata using Laravel's eloquent syntax. ?? Supporting If you are using one or more Renoki Co. open-source pack

Renoki Co. 4 Dec 27, 2021
Rapidly speed up your Laravel workflow with generators

Fast Workflow in Laravel With Custom Generators This Laravel package provides a variety of generators to speed up your development process. These gene

Jeffrey Way 22 Oct 28, 2022
Laracademy Generators - is a tool set that helps speed up the development process of a Laravel application.

Laracademy Generators Laracademy Generators - is a tool set that helps speed up the development process of a Laravel application. Author(s): Laracadem

Laracademy 320 Dec 24, 2022
A Laravel package to speed up deployment by skipping asset compilation whenever possible.

Airdrop for Laravel Read the full docs at hammerstone.dev/airdrop/docs. Hammerstone Airdrop for Laravel is a package that speeds up your deploys by sk

Hammerstone 160 Nov 24, 2022
Livewire trait (throttling). Limiting request processing speed

Livewire Throttling Installation You can install the package via composer: composer require f1uder/livewire-throttling Usage Livewire component <?php

Fluder 5 Dec 7, 2022
Intranet Home Page is a highly-configurable self-hosted browser homepage with integrations for public and local data feeds.

Intranet-Home-Page Created in response to personal "dashboards" that are little more than pages with a list of frequently accessed links, Intranet Hom

null 56 Aug 27, 2022
A premade, easy to use local development setup to be used for authoring Laravel applications

Laravel Drydock This project is a premade, easy to use local development setup to be used for authoring Laravel applications. The deliverables of this

Alexander Trauzzi 19 Nov 11, 2022