Laravel DynamoDB
A DynamoDB based Eloquent model and Query builder for Laravel.
You can find an example implementation in kitar/simplechat.
- Motivation
- Installation
- Sample data
- Model
- Authentication with model
- Query Builder
- Testing
Motivation
- I want to use DynamoDB with Laravel. (e.g., authenticate with custom user provider)
- I want to use a simple API which doesn't need to worry about cumbersome things like manually handling Expression Attributes.
- I want to extend Laravel's code as much as I can to:
- Rely on Laravel's robust codes.
- keep the additional implementation simple and maintainable.
- I don't want to make it fully compatible with Eloquent because DynamoDB is different from relational databases.
- I'm longing for jessengers/laravel-mongodb. What if we have that for DynamoDB?
Installation
Install the package via Composer:
$ composer require kitar/laravel-dynamodb
Laravel (6.x, 7.x, 8.x, 9.x)
Add dynamodb configs to config/database.php:
'connections' => [
'dynamodb' => [
'driver' => 'dynamodb',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'token' => env('AWS_SESSION_TOKEN', null),
'endpoint' => env('DYNAMODB_ENDPOINT', null),
'prefix' => '', // table prefix
],
...
],
Update the DB_CONNECTION variable in your .env file:
DB_CONNECTION=dynamodb
Non-Laravel projects
For usage outside Laravel, you can create the connection manually and start querying with Query Builder.
$connection = new Kitar\Dynamodb\Connection([
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'token' => env('AWS_SESSION_TOKEN', null),
'endpoint' => env('DYNAMODB_ENDPOINT', null),
'prefix' => '', // table prefix
]);
$connection->table('your-table')->...
Sample data
Many of the example codes in this document are querying to DynamoDB's official sample data. If you want to try these codes with actual DynamoDB tables, it's handy to load them to your tables before.
Model
DynamoDB model extends Eloquent model so that we can use familiar features such as mutators, serialization, etc.
The main difference between Eloquent model and DynamoDB model is:
- Eloquent model
- Can handle relations.
- Forward calls to model (Eloquent) query builder. (e.g.,
create,createOrFirstwherewith)
- DynamoDB model
- Cannot handle relations.
- Forward calls to database (DynamoDB) query builder. (e.g.,
getItem,putItem,scan,filter)
Extending the base model
Most of the attributes are the same as the original Eloquent model, but there are few DynamoDB-specific attributes.
| Name | Required | Description |
|---|---|---|
| table | yes | Name of the Table. |
| primaryKey | yes | Name of the Partition Key. |
| sortKey | Name of the Sort Key. | |
| sortKeyDefault | Default value for the Sort Key. |
For example, if our table has only partition key, the model will look like this:
use Kitar\Dynamodb\Model\Model;
class ProductCatalog extends Model
{
protected $table = 'ProductCatalog';
protected $primaryKey = 'Id';
protected $fillable = ['Id', 'Price', 'Title'];
}
If our table also has sort key:
use Kitar\Dynamodb\Model\Model;
class Thread extends Model
{
protected $table = 'Thread';
protected $primaryKey = 'ForumName';
protected $sortKey = 'Subject';
protected $fillable = ['ForumName', 'Subject'];
}
If we set sortKeyDefault, it will be used when we instantiate or call find without sort key.
use Kitar\Dynamodb\Model\Model;
use Illuminate\Auth\Authenticatable;
use Illuminate\Contracts\Auth\Authenticatable as AuthenticatableContract;
class User extends Model implements AuthenticatableContract
{
use Authenticatable;
protected $table = 'User';
protected $primaryKey = 'email';
protected $sortKey = 'type';
protected $sortKeyDefault = 'profile';
protected $fillable = [
'name', 'email', 'password', 'type',
];
}
Note that this model is implementing
Illuminate\Contracts\Auth\Authenticatableand usingIlluminate\Auth\Authenticatable. This is optional, but if we use them, we can use this model with authentication as well. For authentication, please refer to Authentication section) for more details.
Basic Usage
Retrieving all models
$products = ProductCatalog::scan();
or alternatively,
$products = ProductCatalog::all();
You also can override the scan() method to fit your needs, such as filtering models for single table design. For example:
public static function scan($exclusiveStartKey = null, $sort = 'asc', $limit = 50)
{
$products = static::index('GSI1')
->keyCondition('GSI1PK', '=', 'PRODUCT#')
->keyCondition('GSI1SK', 'begins_with', 'PRODUCT#')
->exclusiveStartKey($exclusiveStartKey)
->scanIndexForward($sort == 'desc' ? false : true)
->limit($limit)
->query();
return [
'items' => $products,
'LastEvaluatedKey' => $products->first()->meta()['LastEvaluatedKey'] ?? null,
];
}
DynamoDB can only handle result set up to 1MB per call, so we have to paginate if there are more results. see Paginating the Results for more details.
Retrieving a model
If the model has only partition key:
ProductCatalog::find(101);
If the model also has sort key:
Thread::find([
'ForumName' => 'Amazon DynamoDB', // Partition key
'Subject' => 'DynamoDB Thread 1' // Sort key
]);
If the model has sort key and sortKeyDefault is defined:
User::find('[email protected]'); // Partition key. sortKeyDefault will be used for Sort key.
You also can modify the behavior of the find() method to fit your needs. For example:
public static function find($userId)
{
return parent::find([
'PK' => str_starts_with($userId, 'USER#') ? $userId : 'USER#'.$userId,
'SK' => 'USER#',
]);
}
create()
$user = User::create([
'email' => '[email protected]',
'type' => 'profile' // Sort key. If we don't specify this, sortKeyDefault will be used.
]);
save()
$user = new User([
'email' => '[email protected]',
'type' => 'profile'
]);
$user->save();
$user->name = 'foo';
$user->save();
update()
$user->update([
'name' => 'foobar'
]);
delete()
$user->delete();
increment() / decrement()
When we call increment() and decrement(), the Atomic Counter will be used under the hood.
$user->increment('views', 1);
$user->decrement('views', 1);
We can also pass additional attributes to update.
$user->increment('views', 1, [
'last_viewed_at' => '...',
]);
Advanced Queries
We can use Query Builder functions through model such as query scan filter condition keyCondition etc.
For example:
Thread::keyCondition('ForumName', '=', 'Amazon DynamoDB')
->keyCondition('Subject', 'begins_with', 'DynamoDB')
->filter('Views', '=', 0)
->query();
Please refer to Query Builder for the details.
Authentication with model
We can create a Custom User Provider to authenticate with DynamoDB. For the detail, please refer to Laravel's official document.
To use authentication with the model, the model should implement Illuminate\Contracts\Auth\Authenticatable contract. In this section, we'll use the example User model above.
Register custom user provider
After we prepare authenticatable model, we need to make the custom user provider. We can make it own (it's simple), but we'll use Kitar\Dynamodb\Model\AuthUserProvider in this section.
To register custom user provider, add codes below in App/Providers/AuthServiceProvider.php.
use Kitar\Dynamodb\Model\AuthUserProvider;
...
public function boot()
{
$this->registerPolicies();
Auth::provider('dynamodb', function ($app, array $config) {
return new AuthUserProvider(
$app['hash'],
$config['model'],
$config['api_token_name'] ?? null,
$config['api_token_index'] ?? null
);
});
}
Change auth config
Then specify driver and model name for authentication in config/auth.php.
'providers' => [
// Eloquent
// 'users' => [
// 'driver' => 'eloquent',
// 'model' => App\User::class,
// ],
// DynamoDB
'users' => [
'driver' => 'dynamodb',
'model' => App\User::class,
'api_token_name' => 'api_token',
'api_token_index' => 'api_token-index'
],
],
api_token_name and api_token_index are optional, but we need them if we use api token authentication.
Registration Controller
You might need to modify the registration controller. For example, if we use Laravel Breeze, the modification looks like below.
class RegisteredUserController extends Controller
{
...
public function store(Request $request)
{
$request->validate([
'name' => 'required|string|max:255',
'email' => ['required', 'string', 'email', 'max:255', function ($attribute, $value, $fail) {
if (User::find($value)) {
$fail('The '.$attribute.' has already been taken.');
}
}],
'password' => 'required|string|confirmed|min:8',
]);
$user = new User([
'name' => $request->name,
'email' => $request->email,
'password' => Hash::make($request->password),
]);
$user->save();
Auth::login($user);
event(new Registered($user));
return redirect(RouteServiceProvider::HOME);
}
}
There are two modifications. The first one is adding the closure validator for email instead of unique validator. The second one is using the save() method to create user instead of the create() method.
Query Builder
We can use Query Builder without model.
$result = DB::table('Thread')->scan();
Or even outside Laravel.
$connection = new Kitar\Dynamodb\Connection([
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'token' => env('AWS_SESSION_TOKEN', null),
'endpoint' => env('DYNAMODB_ENDPOINT', null),
'prefix' => '', // table prefix
]);
$result = $connection->table('Thread')->scan();
If we query through the model, we don't need to specify the table name, and the response will be the model instance(s).
$threads = Thread::scan();
Basic Usage
getItem()
$response = DB::table('ProductCatalog')
->getItem(['Id' => 101]);
Instead of marshaling manually, pass a plain array.
Kitar\Dynamodb\Query\Grammarwill automatically marshal them before querying.
putItem()
DB::table('Thread')
->putItem([
'ForumName' => 'Amazon DynamoDB',
'Subject' => 'New discussion thread',
'Message' => 'First post in this thread',
'LastPostedBy' => '[email protected]',
'LastPostedDateTime' => '201603190422'
]);
updateItem()
DB::table('Thread')
->key([
'ForumName' => 'Laravel',
'Subject' => 'Laravel Thread 1'
])->updateItem([
'LastPostedBy' => null, // REMOVE
'Replies' => null, // REMOVE
'Message' => 'Updated' // SET
]);
Currently, we only support simple SET and REMOVE actions. If the attribute has value, it will be passed to SET action. If the value is null, it will be passed to REMOVE action.
deleteItem()
DB::table('Thread')
->deleteItem([
'ForumName' => 'Amazon DynamoDB',
'Subject' => 'New discussion thread'
]);
Projection Expressions
A Projection Expression is a string that identifies the attributes that web want. (It's like select statement for SQL)
select()
We can specify Projection Expressions in the same manner as the original select clause.
$response = DB::table('ProductCatalog')
->select('Price', 'Title')
->getItem(['Id' => 101]);
Condition Expressions
When we manipulate data in Amazon DynamoDB table, we use putItem, updateItem and DeleteItem. We can use Condition Expressions to determine which items should be modified.
condition()
To specify Condition Expression, we use condition clause. This works basically same as the original where clause, but it's for Condition Expressions.
DB::table('ProductCatalog')
->condition('Id', 'attribute_not_exists')
->putItem([
'Id' => 101,
'ProductCategory' => 'Can I overwrite?'
]);
Note that we specify
attribute_not_existsfor the operator of condition. This is DynamoDB-specific operator which calledfunction. See DynamoDB-specific operators for condition() and filter() for more details.
OR statements
DB::table('ProductCatalog')
->condition('Id', 'attribute_not_exists')
->orCondition('Price', 'attribute_not_exists)
->putItem([...]);
AND statements
DB::table('ProductCatalog')
->condition('Id', 'attribute_not_exists')
->condition('Price', 'attribute_not_exists)
->putItem([...]);
conditionIn()
ProductCatalog::key(['Id' => 101])
->conditionIn('ProductCategory', ['Book', 'Bicycle'])
->updateItem([
'Description' => 'updated!'
]);
conditionBetween()
ProductCatalog::key(['Id' => 101])
->conditionBetween('Price', [0, 10])
->updateItem([
'Description' => 'updated!'
]);
Working with Queries
The Query operation in Amazon DynamoDB finds items based on primary key values.
query() and keyCondition()
When we query, we must specify keyCondition as well.
We can use some comparison operators for sort key, but we must use the equality condition for the partition key.
$response = DB::table('Thread')
->keyCondition('ForumName', '=', 'Amazon DynamoDB')
->keyCondition('Subject', 'begins_with', 'DynamoDB')
->query();
keyConditionBetween()
$response = DB::table('Thread')
->keyCondition('ForumName', '=', 'Amazon DynamoDB')
->keyConditionBetween('Subject', ['DynamoDB Thread 1', 'DynamoDB Thread 2'])
->query();
Sort order
query results are always sorted by the sort key value. To reverse the order, set the ScanIndexForward parameter to false.
$response = DB::table('Thread')
->keyCondition('ForumName', '=', 'Amazon DynamoDB')
->scanIndexForward(false)
->query();
Working with Scans
scan()
$response = DB::table('Thread')->scan();
Filtering the Results
When we query or scan, we can filter results with Filter Expressions before it returned.
It can't reduce the amount of read capacity, but it can reduce the size of traffic data.
filter()
$response = DB::table('Thread')
->filter('LastPostedBy', '=', 'User A')
->scan();
OR statement
$response = DB::table('Thread')
->filter('LastPostedBy', '=', 'User A')
->orFilter('LastPostedBy', '=', 'User B')
->scan();
AND statement
$response = DB::table('Thread')
->filter('LastPostedBy', '=', 'User A')
->filter('Subject', 'begins_with', 'DynamoDB')
->scan();
filterIn()
$response = DB::table('Thread')
->filterIn('LastPostedBy', ['User A', 'User B'])
->scan();
filterBetween()
$response = DB::table('ProductCatalog')
->filterBetween('Price', [0, 100])
->scan();
Paginating the Results
A single query or scan only returns a result set that fits within the 1 MB size limit. If there are more results, we need to paginate.
exclusiveStartKey()
If there are more results, the response contains LastEvaluatedKey.
$response = DB::table('ProductCatalog')
->limit(5)
->scan();
$response['LastEvaluatedKey']; // array
We can pass this key to exclusiveStartKey to get next results.
$response = DB::table('ProductCatalog')
->exclusiveStartKey($response['LastEvaluatedKey'])
->limit(5)
->scan();
If you are using Query Builder through model, you can access to exclusiveStartKey by:
$products = ProductCatalog::limit(5)->scan();
$products->first()->meta()['LastEvaluatedKey']; // array
Using Global Secondary Indexes
Some applications might need to perform many kinds of queries, using a variety of different attributes as query criteria. To support these requirements, you can create one or more global secondary indexes and issue query requests against these indexes in Amazon DynamoDB.
index()
Use index clause to specify Global Secondary Index name.
$response = DB::table('Reply')
->index('PostedBy-Message-index')
->keyCondition('PostedBy', '=', 'User A')
->keyCondition('Message', '=', 'DynamoDB Thread 2 Reply 1 text')
->query();
Atomic Counter
DynamoDB supports Atomic Counter. When we call increment() and decrement() through Model or Query Builder, Atomic Counter will be used under the hood.
DB::('Thread')->key([
'ForumName' => 'Laravel',
'Subject' => 'Laravel Thread 1'
])->increment('Replies', 2);
We can also pass additional attributes to update.
DB::('Thread')->key([
'ForumName' => 'Laravel',
'Subject' => 'Laravel Thread 1'
])->increment('Replies', 2, [
'LastPostedBy' => 'User A',
]);
DynamoDB-specific operators for condition() and filter()
For condition and filter clauses, we can use DynamoDB's comparators and functions.
Comparators
= <> < <= > >= can be used in the form of:
filter($key, $comparator, $value);
functions
Available functions are:
filter($key, 'attribute_exists');
filter($key, 'attribute_not_exists');
filter($key, 'attribute_type', $type);
filter($key, 'begins_with', $value);
filter($key, 'contains', $value);
sizefunction is not supported at this time.
Testing
$ ./vendor/bin/phpunit
17 Jan 20, 2022
75 Dec 7, 2022
484 Jan 4, 2023
38 Nov 22, 2022
35 Dec 15, 2022
149 Nov 11, 2022
88 Dec 31, 2022
3 Jan 1, 2022
5 Jan 6, 2023
15 Oct 29, 2022
52 Sep 6, 2022
4 May 12, 2022
87 Nov 26, 2022
13 Aug 17, 2022
4 Mar 1, 2022
2 Jul 10, 2022
138 Nov 28, 2022