Seed your database with CSV files

Overview

CSV Seeder

Latest Version on Packagist Software License Build Status Quality Score Total Downloads

Seed your database with CSV files

This package allows CSV based seeds.

Installation

Require this package in your composer.json and run composer update (or run composer require flynsarmy/csv-seeder:2.* directly):

For PHP 7.4+

"flynsarmy/csv-seeder": "2.0.*"

For older PHP versions

"flynsarmy/csv-seeder": "1.*"

Usage

Your CSV's header row should match the DB columns you wish to import. IE to import id and name columns, your CSV should look like:

id,name
1,Foo
2,Bar

Seed classes must extend Flynsarmy\CsvSeeder\CsvSeeder, they must define the destination database table and CSV file path, and finally they must call parent::run() like so:

use Flynsarmy\CsvSeeder\CsvSeeder;

class StopsTableSeeder extends CsvSeeder {

	public function __construct()
	{
		$this->table = 'your_table';
		$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	}

	public function run()
	{
		// Recommended when importing larger CSVs
		DB::disableQueryLog();

		// Uncomment the below to wipe the table clean before populating
		DB::table($this->table)->truncate();

		parent::run();
	}
}

Drop your CSV into /database/seeds/csvs/your_csv.csv or whatever path you specify in your constructor above.

Configuration

In addition to setting the database table and CSV filename, the following configuration options are available. They can be set in your class constructor:

  • connection (string '') Connection to use for inserts. Leave empty for default connection.
  • insert_chunk_size (int 500) An SQL insert statement will trigger every insert_chunk_size number of rows while reading the CSV
  • csv_delimiter (string ,) The CSV field delimiter.
  • hashable (array [password]) List of fields to be hashed before import, useful if you are importing users and need their passwords hashed. Uses Hash::make(). Note: This is EXTREMELY SLOW. If you have a lot of rows in your CSV your import will take quite a long time.
  • offset_rows (int 0) How many rows at the start of the CSV to ignore. Warning: If used, you probably want to set a mapping as your header row in the CSV will be skipped.
  • mapping (array []) Associative array of csvCol => dbCol. See examples section for details. If not specified, the first row (after offset) of the CSV will be used as the mapping.
  • should_trim (bool false) Whether to trim the data in each cell of the CSV during import.
  • timestamps (bool false) Whether or not to add created_at and updated_at columns on import.
    • created_at (string current time in ISO 8601 format) Only used if timestamps is true
    • updated_at (string current time in ISO 8601 format) Only used if timestamps is true

Examples

CSV with pipe delimited values:

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
}

Specifying which CSV columns to import:

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	$this->mapping = [
	    0 => 'first_name',
	    1 => 'last_name',
	    5 => 'age',
	];
}

Trimming the whitespace from the imported data:

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	$this->mapping = [
	    0 => 'first_name',
	    1 => 'last_name',
	    5 => 'age',
	];
	$this->should_trim = true;
}

Skipping the CSV header row (Note: A mapping is required if this is done):

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	$this->offset_rows = 1;
	$this->mapping = [
	    0 => 'first_name',
	    1 => 'last_name',
	    2 => 'password',
	];
	$this->should_trim = true;
}

Specifying the DB connection to use:

public function __construct()
{
	$this->table = 'users';
	$this->connection = 'my_connection';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
}

Migration Guide

2.0

  • $seeder->hashable is now an array of columns rather than a single column name. Wrap your old string value in [].

License

CsvSeeder is open-sourced software licensed under the MIT license

Comments
  • FatalThrowableError  : syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST)

    FatalThrowableError : syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST)

    I am using Php 7.2, Laravel 6.18 and i chacked against 2.0.0->2.0.2 versions and I am always getting this error even if I have an almost empty CsvSeeder

    Symfony\Component\Debug\Exception\FatalThrowableError  : syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST)
    
      at /var/www/vendor/flynsarmy/csv-seeder/src/Flynsarmy/CsvSeeder/CsvSeeder.php:22
        18|      * DB table name
        19|      *
        20|      * @var string
        21|      */
        22|     public string $table;
        23|
        24|     /**
        25|      * CSV filename
        26|      *
    
    
      Exception trace:
    
      1   Composer\Autoload\includeFile("/var/www/vendor/composer/../flynsarmy/csv-seeder/src/Flynsarmy/CsvSeeder/CsvSeeder.php")
          /var/www/vendor/composer/ClassLoader.php:322
      2   Composer\Autoload\ClassLoader::loadClass("Flynsarmy\CsvSeeder\CsvSeeder")
          [internal]:0
    ...
    

    This is the CsvSeeder:

    <?php
    use Flynsarmy\CsvSeeder\CsvSeeder;
    class Comuni201901Seeder extends CsvSeeder
    {
        public function __construct()
        {
            $this->table = 'comuni201901s';
            //$this->csv_delimiter = ',';
            $this->filename = base_path().'/database/seeds/csv/comuni_2019_01.csv';
            //$this->should_trim = true;
        }
    
        public function run()
        {
            // Recommended when importing larger CSVs
            DB::disableQueryLog();
            // Uncomment the below to wipe the table clean before populating
            //DB::table($this->table)->truncate();
            //parent::run();
        }
    }
    
    opened by mcfoi 6
  • insert_chunk_size issue

    insert_chunk_size issue

    I have this issue where I have a csv with 7740 rows. If I set the insert_chuck_size = 1 all rows are inserted into the db table. But as soon as I start to increase the insert_chuck_size it starts to drop off some of the records at the end of the csv file and don't insert them.

    I suspect there is an counter/indexing problem on the chunking mechanism, as I increase the chunk size the more records it leaves out from import from the bottom of the csv.

    opened by slakbal 5
  • comments

    comments

    Hello,

    What about comments in csv files?
    eg I have this csv file:

    St. James the Greater;2016-07-25;true;false;true

    An comment

    Constitution Day;2016-12-06;true;false;true National Holiday;2016-03-13;true;false;true #Vatikan

    What I mean is that all lines after # should be ignored.

    opened by reinhardp 4
  • Allow for extra columns in the CSV that are not in the DB

    Allow for extra columns in the CSV that are not in the DB

    My CSV seeder data files sometimes have extra columns, which caused errors during import.

    I added code to cross reference the columns being imported to the database structure. Extra input columns are now ignored and do not throw a SQL insert error.

    This means that laravel-csv-seeder is now more robust:

    • Column order is unimportant
    • missing columns in the CSV that exist in the DB are inserted with NULL
    • Extra columns in the CSV that do not exist in the DB are ignored (new functionality added by this pull request)
    opened by jackfruh 4
  • fix issue where data did not seed on non-default connection

    fix issue where data did not seed on non-default connection

    This fixes an issue where the data wouldn't seed on a non-default connection because the query for the column mappings was using the default connection.

    opened by DanAtFh 3
  • Not all entries make it into the database.

    Not all entries make it into the database.

    Hello,

    I am having a bit of an issue importing a csv full of email addresses.

    As per your example here is the code I am using:

    public function __construct()
    {
        $this->table = 'users';
        $this->csv_delimiter = '|';
        $this->filename = base_path().'/database/seeds/csvs/list.csv';
        $this->mapping = [
            0=> 'email',
            1 => 'language'
        ];
    }
    
    public function run()
    {
        // Recommended when importing larger CSVs
        DB::disableQueryLog();
    
        // Uncomment the below to wipe the table clean before populating
        DB::table($this->table)->truncate();
    
        parent::run();
    }
    

    My CSV looks something like this:

    [email protected]|nl [email protected]|en [email protected]|fr ...

    I originally have 11033 entries in the csv, but after the import only 8883 make it, and if I use the the insert_chunk_size option I only get a little over 5000 entries.

    I have noticed that there seems to be rows that don't make it or get deleted since the end id of the last row does match the number of entries, i.e. row 8883 in mysql has ID 11033

    For example in the table the ids jump from 7550 -> 7701 from one row to the next. i.e

    | id | email | language | | --- | --- | --- | | 7550 | [email protected] | en | | 7701 | [email protected] | fr |

    Thanks for the plugin!

    opened by Shobs 3
  • Multiple Inserts being broken

    Multiple Inserts being broken

    While using this I think I found a bug. When data in the csv file sometimes has certain fields, https://github.com/Flynsarmy/laravel-csv-seeder/blob/master/src/Flynsarmy/CsvSeeder/CsvSeeder.php#L107 seems to break the inserts at least on laravel 4.2. I get many errors unless I make the insert_chunk_size = 1.

    opened by ragingdave 3
  • New release?

    New release?

    Any chance you would want to do a new release? I am currently stuck as I am in need of this fix :)

    https://github.com/Flynsarmy/laravel-csv-seeder/commit/4f50c9d36ffc550f3eda94bd74a78d1a70823c79

    I will gladly help anyway I can.

    opened by sinnbeck 2
  • array_pull() helper method deprecated

    array_pull() helper method deprecated

    Overview

    Issue when using laravel-csv-seeder in Laravel v7.

    Expected Behavior

    Expect to seed database using csv seeder.

    Actual Behavior

    Receive the following error when trying to seed with artisan migrate --seed:

     Call to undefined function Flynsarmy\CsvSeeder\array_pull()
    
      at vendor/flynsarmy/csv-seeder/src/Flynsarmy/CsvSeeder/CsvSeeder.php:168
        164|
        165|                 // skip csv columns that don't exist in the database
        166|                 foreach($mapping  as $index => $fieldname){
        167|                     if (!DB::getSchemaBuilder()->hasColumn($this->table, $fieldname)){
      > 168|                        array_pull($mapping, $index);
        169|                     }
        170|                 }
        171|             }
        172|             else
    

    Resolution

    It turns out the array_pull() method was deprecated in Laravel v5.8. These methods are now preferred as static methods: Arr::pull()

    The deprecated methods have been separated into a laravel/helpers package. Including this package resolved the issue:

    composer require laravel/helpers
    

    You may want to update the library to use the static method, make laravel/helpers a peer dependency, or simply update the README with the requirements. I'd be happy to make a PR if you're interested.

    Thanks for the library, by the way. It's very handy!

    opened by Splode 2
  • Added auto_detect_line_endings

    Added auto_detect_line_endings

    Added auto_detect_line_endings. This fixed a compatibility problem with CSV files generated in Excel Mac (excel mac used x0D for line endings instead of x0A) Note that set_ini is temporary for that request. it does not change the users php.ini http://php.net/manual/en/filesystem.configuration.php#ini.auto-detect-line-endings http://php.net/manual/en/function.ini-set.php

    opened by jackfruh 2
  • skip CSV columns that don't exist in the database

    skip CSV columns that don't exist in the database

    If no mapping is found, first row is used, but csv columns not found in the DB are stripped out to prevent insert errors. This is done only once, up at the top, and is only done if no mapping is specified.

    opened by jackfruh 2
  • Fixing issue: Invalid datetime format, causing by using MySQL in stri…

    Fixing issue: Invalid datetime format, causing by using MySQL in stri…

    Faced with the issue with the use of MySQL with strict mode enabled. During the import, the following error appears: Invalid datetime format: 1292 Incorrect datetime value

    One of the possible fixes is changing $created_at, $updated_at type from string to the Cabon type, which will force Illuminate to do the black magic, with the date formatting.

    opened by hicaliber-nikita-alekseev 0
  • Does not show error message when database throws an error

    Does not show error message when database throws an error

    I got the "Seeded:" message but the csv file was not seeded because of a database error(forgotten to make a column nullable). I wasted lot of time to figure out the error. It would be wonderful it it can throw if there is any database error.

    PS: Thanks for providing this library for free. It was really helpful.

    opened by glupeksha 8
Releases(v2.0.6)
Owner
null
Read and write CSV files with PHP.

Read and write CSV files with PHP. This package provides a minimalistic wrapper around the excellent league/csv package. The API is heavily inspired b

Ryan Chandler 6 Nov 16, 2022
Small script for importing the KvK (Dutch Chamber of Commerce) Open Data Set (CSV file) to a MySQL database.

KvK-CSV-2-SQL Small script for importing the KvK (Dutch Chamber of Commerce) Open Data Set (CSV file) to a MySQL database. Table of content KvK-CSV-2-

BASTIAAN 3 Aug 5, 2022
This project is about import CSV to Laravel with Laravel Excel library.

About Laravel Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable and creative experie

Abraham Víctor Zaragoza Rodríguez 1 Nov 20, 2021
Pure PHP NoSQL database with no dependency. Flat file, JSON based document database.

Please give it a Star if you like the project ?? ❤️ SleekDB - A NoSQL Database made using PHP Full documentation: https://sleekdb.github.io/ SleekDB i

Kazi Mehedi Hasan 745 Jan 7, 2023
SleekwareDB is a NoSQL database storage service. A database storage service that can be used for various platforms and is easy to integrate.

SleekwareDB is a NoSQL database storage service. A database storage service that can be used for various platforms and is easy to integrate. NoSQL API

SleekwareDB 12 Dec 11, 2022
Laravel Migrations Generator: Automatically generate your migrations from an existing database schema.

Laravel Migrations Generator Generate Laravel Migrations from an existing database, including indexes and foreign keys! Upgrading to Laravel 5.4 Pleas

Bernhard Breytenbach 3.3k Dec 30, 2022
Adminer database management tool for your Laravel application.

Laravel Adminer Adminer database management tool for your Laravel application. Table of Contents Introduction Features Installation CSRF token middlew

Khalid Moharrum 45 Jul 25, 2022
Clean up your Magento database by removing orphaned, unused and wrongly added attribute, attribute values and settings (for M2).

Magento 2 EAV Cleaner Console Command Purpose of this project is to check for different flaws that can occur due to EAV and provide cleanup functions.

FireGento e. V. - Hackathons 41 Dec 14, 2022
The lightweight PHP database framework to accelerate development

The lightweight PHP database framework to accelerate development Features Lightweight - Less than 100 KB, portable with only one file Easy - Extremely

Angel Lai 4.6k Dec 28, 2022
[READ ONLY] Subtree split of the Illuminate Database component (see laravel/framework)

Illuminate Database The Illuminate Database component is a full database toolkit for PHP, providing an expressive query builder, ActiveRecord style OR

The Laravel Components 2.5k Dec 27, 2022
ORM layer that creates models, config and database on the fly

RedBeanPHP 5 RedBeanPHP is an easy to use ORM tool for PHP. Automatically creates tables and columns as you go No configuration, just fire and forget

Gabor de Mooij 2.2k Jan 9, 2023
Spot v2.x DataMapper built on top of Doctrine's Database Abstraction Layer

Spot DataMapper ORM v2.0 Spot v2.x is built on the Doctrine DBAL, and targets PHP 5.4+. The aim of Spot is to be a lightweight DataMapper alternative

Spot ORM 602 Dec 27, 2022
SQL database access through PDO.

Aura.Sql Provides an extension to the native PDO along with a profiler and connection locator. Because ExtendedPdo is an extension of the native PDO,

Aura for PHP 533 Dec 30, 2022
PHP Database Migrations for Everyone

Phinx: Simple PHP Database Migrations Intro Phinx makes it ridiculously easy to manage the database migrations for your PHP app. In less than 5 minute

CakePHP 4.3k Jan 7, 2023
Database management in a single PHP file

Adminer - Database management in a single PHP file Adminer Editor - Data manipulation for end-users https://www.adminer.org/ Supports: MySQL, MariaDB

Jakub Vrána 5.5k Jan 1, 2023
Doctrine Database Abstraction Layer

Doctrine DBAL 4.0-dev 3.0 2.13 N/A N/A Powerful database abstraction layer with many features for database schema introspection, schema management and

Doctrine 8.9k Dec 28, 2022
Adjacency List’ed Closure Table database design pattern implementation for the Laravel framework.

ClosureTable This is a database manipulation package for the Laravel 5.4+ framework. You may want to use it when you need to store and operate hierarc

Yan Ivanov 441 Dec 11, 2022
A drop-in library for certain database functionality in Laravel, that allows for extra features that may never make it into the main project.

Eloquence Eloquence is a package to extend Laravel's base Eloquent models and functionality. It provides a number of utilities and classes to work wit

Kirk Bushell 470 Dec 8, 2022
[Package] Multi-tenant Database Schema Manager for Laravel

Multi-tenant Database Schema Manager for Laravel Tenanti allow you to manage multi-tenant data schema and migration manager for your Laravel applicati

Orchestra Platform 580 Dec 5, 2022