このページは、まだ日本語ではご利用いただけません。翻訳中です。
旧バージョンのドキュメントを参照しています。 最新のドキュメントはこちらをご参照ください。
Storing Custom Entities
While not all plugins need it, your plugin might need to store more than its configuration in the database. In that case, Kong provides you with an abstraction on top of its primary data stores which allows you to store custom entities.
As explained in the previous chapter, Kong interacts with the model layer through classes we refer to as “DAOs”, and available on a singleton often referred to as the “DAO Factory”. This chapter will explain how to provide an abstraction for your own entities.
Modules
kong.plugins.<plugin_name>.daos
kong.plugins.<plugin_name>.migrations.init
kong.plugins.<plugin_name>.migrations.000_base_<plugin_name>
kong.plugins.<plugin_name>.migrations.001_<from-version>_to_<to_version>
kong.plugins.<plugin_name>.migrations.002_<from-version>_to_<to_version>
Create the migrations folder
Once you have defined your model, you must create your migration modules which will be executed by Kong to create the table in which your records of your entity will be stored.
If your plugin doesn’t have it already, you should add a <plugin_name>/migrations
folder to it. If there is no init.lua
file inside already, you should create one.
This is where all the migrations for your plugin will be referenced.
The initial version of your migrations/init.lua
file will point to a single migration.
In this case we have called it 000_base_my_plugin
.
-- `migrations/init.lua`
return {
"000_base_my_plugin",
}
This means that there will be a file in <plugin_name>/migrations/000_base_my_plugin.lua
containing the initial migrations. We’ll see how this is done in a minute.
Add a new migration to an existing plugin
Sometimes it is necessary to introduce changes after a version of a plugin has already been released. A new functionality might be needed. A database table row might need changing.
When this happens, you must create a new migrations file. You must not modify the existing migration files once they are published (you can still make them more robust and bulletproof if you want, e.g. always try to write the migrations reentrant).
While there is no strict rule for naming your migration files, there is a convention that the
initial one is prefixed by 000
, the next one by 001
, and so on.
Following with our previous example, if we wanted to release a new version of the plugin with
changes in the database (for example, a table was needed called foo
) we would insert it by
adding a file called <plugin_name>/migrations/001_100_to_110.lua
, and referencing it on the
migrations init file like so (where 100
is the previous version of the plugin 1.0.0
and
110
is the version to which plugin is migrated to 1.1.0
:
-- `<plugin_name>/migrations/init.lua`
return {
"000_base_my_plugin",
"001_100_to_110",
}
Migration file syntax
A migration file is a Lua file which returns a table with the following structure:
-- `<plugin_name>/migrations/000_base_my_plugin.lua`
return {
postgres = {
up = [[
CREATE TABLE IF NOT EXISTS "my_plugin_table" (
"id" UUID PRIMARY KEY,
"created_at" TIMESTAMP WITHOUT TIME ZONE,
"col1" TEXT
);
DO $$
BEGIN
CREATE INDEX IF NOT EXISTS "my_plugin_table_col1"
ON "my_plugin_table" ("col1");
EXCEPTION WHEN UNDEFINED_COLUMN THEN
-- Do nothing, accept existing state
END$$;
]],
}
}
-- `<plugin_name>/migrations/001_100_to_110.lua`
return {
postgres = {
up = [[
DO $$
BEGIN
ALTER TABLE IF EXISTS ONLY "my_plugin_table" ADD "cache_key" TEXT UNIQUE;
EXCEPTION WHEN DUPLICATE_COLUMN THEN
-- Do nothing, accept existing state
END;
$$;
]],
teardown = function(connector, helpers)
assert(connector:connect_migrations())
assert(connector:query([[
DO $$
BEGIN
ALTER TABLE IF EXISTS ONLY "my_plugin_table" DROP "col1";
EXCEPTION WHEN UNDEFINED_COLUMN THEN
-- Do nothing, accept existing state
END$$;
]])
end,
}
}
Each strategy section has two parts, up
and teardown
.
-
up
is an optional string of raw SQL statements. Those statements are executed whenkong migrations up
is executed.We recommend that all the non-destructive operations, such as creation of new tables and addition of new records, are done on the
up
sections. -
teardown
is an optional Lua function, which takes aconnector
parameter. The connector can invoke thequery
method to execute SQL queries. Teardown is triggered bykong migrations finish
.We recommend that destructive operations, such as removal of data, changing row types, and insertion of new data, are done on the
teardown
sections.
All SQL statements should be written so that they are as reentrant as possible.
For example, use DROP TABLE IF EXISTS
instead of DROP TABLE
,
CREATE INDEX IF NOT EXIST
instead of CREATE INDEX
, and so on. If a migration fails for some
reason, it is expected that the first attempt at fixing the problem will be simply
re-running the migrations.
If your
schema
uses aunique
constraint, you must set this constraint in the migrations for PostgreSQL.
To see a real-life example, give a look at the Key-Auth plugin migrations.
Define a schema
The first step to using custom entities in a custom plugin is defining one or more schemas.
A schema is a Lua table which describes entities. There’s structural information like how are the different fields of the entity named and what are their types, which is similar to the fields describing your plugin configuration). Compared to plugin configuration schemas, custom entity schemas require additional metadata (e.g. which field, or fields, constitute the entities’ primary key).
Schemas are to be defined in a module named:
kong.plugins.<plugin_name>.daos
Meaning that there should be a file called <plugin_name>/daos.lua
inside your
plugin folder. The daos.lua
file should return a table containing one or more
schemas. For example:
-- daos.lua
local typedefs = require "kong.db.schema.typedefs"
return {
-- this plugin only results in one custom DAO, named `keyauth_credentials`:
{
name = "keyauth_credentials", -- the actual table in the database
endpoint_key = "key",
primary_key = { "id" },
cache_key = { "key" },
generate_admin_api = true,
admin_api_name = "key-auths",
admin_api_nested_name = "key-auth",
fields = {
{
-- a value to be inserted by the DAO itself
-- (think of serial id and the uniqueness of such required here)
id = typedefs.uuid,
},
{
-- also interted by the DAO itself
created_at = typedefs.auto_timestamp_s,
},
{
-- a foreign key to a consumer's id
consumer = {
type = "foreign",
reference = "consumers",
default = ngx.null,
on_delete = "cascade",
},
},
{
-- a unique API key
key = {
type = "string",
required = false,
unique = true,
auto = true,
},
},
},
},
}
This example daos.lua
file introduces a single schema called keyauth_credentials
.
Here is a description of some top-level properties:
Name | Type | Description |
---|---|---|
name |
string (required) |
It will be used to determine the DAO name (kong.db.[name] ). |
primary_key |
table (required) |
Field names forming the entity's primary key.
Schemas support composite keys, even if most Kong core entities currently use an UUID named
id .
|
endpoint_key |
string (optional) |
The name of the field used as an alternative identifier on the Admin API.
On the example above, key is the endpoint_key. This means that a credential with
id = 123 and key = "foo" could be referenced as both
/keyauth_credentials/123 and /keyauth_credentials/foo .
|
cache_key |
table (optional) |
Contains the name of the fields used for generating the cache_key , a string which must
unequivocally identify the entity inside Kong's cache. A unique field, like key in your example,
is usually good candidate. In other cases a combination of several fields is preferable.
|
generate_admin_api |
boolean (optional) |
Whether to auto-generate admin api for the entity or not. By default the admin api is generated for all
daos, including custom ones. If you want to create a fully customized admin api for the dao or
want to disable auto-generation for the dao altogether, set this option to false .
|
admin_api_name |
boolean (optional) |
When generate_admin_api is enabled the admin api auto-generator uses the name
to derive the collection urls for the auto-generated admin api. Sometimes you may want to name the
collection urls differently from the name . E.g. with DAO keyauth_credentials
we actually wanted the auto-generator to generate endpoints for this dao with alternate and more
url-friendly name key-auths , e.g. http://<KONG_ADMIN>/key-auths instead of
http://<KONG_ADMIN>/keyauth_credentials ).
|
admin_api_nested_name |
boolean (optional) |
Similar to admin_api_name the admin_api_nested_name specifies the name for
a dao that admin api auto-generator creates in nested contexts. You only need to use this parameter
if you are not happy with name or admin_api_name . Kong for legacy reasons
have urls like http://<KONG_ADMIN>/consumers/john/key-auth where key-auth
does not follow plural form of http://<KONG_ADMIN>/key-auths . admin_api_nested_name
enables you to specify different name in those cases.
|
fields |
table |
Each field definition is a table with a single key, which is the field's name. The table value is a sub-table containing the field's attributes, some of which will be explained below. |
Many field attributes encode validation rules. When attempting to insert or update entities using the DAO, these validations will be checked, and an error returned if the provided input doesn’t conform to them.
The typedefs
variable (obtained by requiring kong.db.schema.typedefs
) is a table containing
a lot of useful type definitions and aliases, including typedefs.uuid
, the most usual type for the primary key,
and typedefs.auto_timestamp_s
, for created_at
fields. It is used extensively when defining fields.
Here’s a non-exhaustive explanation of some of the field attributes available:
Attribute name | type | Description |
---|---|---|
type |
string |
Schemas support the following scalar types: "string" , "integer" , "number" and
"boolean" . Compound types like "array" , "record" , or "set" are
also supported.In addition to these values, the type attribute can also take the special "foreign" value,
which denotes a foreign relationship.Each field will need to be backed by database fields of appropriately similar types, created via migrations. type is the only required attribute for all field definitions.
|
default |
any (matching with type attribute) |
Specifies the value the field will have when attempting to insert it, if no value was provided. Default values are always set via Lua, never by the underlying database. It is thus not recommended to set any default values on fields in migrations. |
required |
boolean |
When set to true on a field, an error will be thrown when attempting to insert an entity lacking a value
for said field (unless the field in question has a default value).
|
unique |
boolean |
When set to This attribute must be backed up by declaring fields as |
auto |
boolean |
When attempting to insert an entity without providing a value for this a field where auto is set to true ,
|
reference |
string |
Required for fields of type foreign . The given string must be the name of an existing schema,
to which the foreign key will "point to". This means that if a schema B has a foreign key pointing to schema A,
then A needs to be loaded before B.
|
on_delete |
string |
Optional and exclusive for fields of type foreign . It dictates what must happen
with entities linked by a foreign key when the entity being referenced is deleted. It can have three possible
values:
In PostgreSQL, you must declare the references as ON DELETE CASCADE/NULL/RESTRICT in a migration.
|
To learn more about schemas, see:
- The source code of typedefs.lua to get an idea of what’s provided there by default.
- The Core Schemas to see examples of some other field attributes not discussed here.
-
All the
daos.lua
files for embedded plugins, especially the key-auth one, which was used for this guide as an example.
The custom DAO
The schemas are not used directly to interact with the database. Instead, a DAO
is built for each valid schema. A DAO takes the name of the schema it wraps, and is
accessible through the kong.db
interface.
For the example schema above, the DAO generated would be available for plugins
via kong.db.keyauth_credentials
.
Select an entity
local entity, err, err_t = kong.db.<name>:select(primary_key)
Attempts to find an entity in the database and return it. Three things can happen:
- The entity was found. In this case, it is returned as a regular Lua table.
- An error occurred - for example the connection with the database was lost. In that
case the first returned value will be
nil
, the second one will be a string describing the error, and the last one will be the same error in table form. - An error does not occur but the entity is not found. Then the function will
just return
nil
, with no error.
Example of usage:
local entity, err = kong.db.keyauth_credentials:select({
id = "c77c50d2-5947-4904-9f37-fa36182a71a9"
})
if err then
kong.log.err("Error when inserting keyauth credential: " .. err)
return nil
end
if not entity then
kong.log.err("Could not find credential.")
return nil
end
Iterate over all the entities
for entity, err on kong.db.<name>:each(entities_per_page) do
if err then
...
end
...
end
This method efficiently iterates over all the entities in the database by making paginated
requests. The entities_per_page
parameter, which defaults to 100
, controls how many
entities per page are returned.
On each iteration, a new entity
will be returned or, if there is any error, the err
variable will be filled up with an error. The recommended way to iterate is checking err
first,
and otherwise assume that entity
is present.
Example of usage:
for credential, err on kong.db.keyauth_credentials:each(1000) do
if err then
kong.log.err("Error when iterating over keyauth credentials: " .. err)
return nil
end
kong.log("id: " .. credential.id)
end
This example iterates over the credentials in pages of 1000 items, logging their ids unless an error happens.
Insert an entity
local entity, err, err_t = kong.db.<name>:insert(<values>)
Inserts an entity in the database, and returns a copy of the inserted entity, or
nil
, an error message (a string) and a table describing the error in table form.
When the insert is successful, the returned entity contains the extra values produced by
default
and auto
.
The following example uses the keyauth_credentials
DAO to insert a credential for a given
Consumer, setting its key
to "secret"
. Notice the syntax for referencing foreign keys.
local entity, err = kong.db.keyauth_credentials:insert({
consumer = { id = "c77c50d2-5947-4904-9f37-fa36182a71a9" },
key = "secret",
})
if not entity then
kong.log.err("Error when inserting keyauth credential: " .. err)
return nil
end
The returned entity, assuming no error happened will have auto
-filled fields, like id
and created_at
.
Update an entity
local entity, err, err_t = kong.db.<name>:update(primary_key, <values>)
Updates an existing entity, provided it can be found using the provided primary key and a set of values.
The returned entity will be the entity after the update takes place, or nil
+ an error message + an error table.
The following example modifies the key
field of an existing credential given the credential’s id:
local entity, err = kong.db.keyauth_credentials:update(
{ id = "2b6a2022-770a-49df-874d-11e2bf2634f5" },
{ key = "updated_secret" },
)
if not entity then
kong.log.err("Error when updating keyauth credential: " .. err)
return nil
end
Notice how the syntax for specifying a primary key is similar to the one used to specify a foreign key.
Upsert an entity
local entity, err, err_t = kong.db.<name>:upsert(primary_key, <values>)
upsert
is a mixture of insert
and update
:
- When the provided
primary_key
identifies an existing entity, it works likeupdate
. - When the provided
primary_key
does not identify an existing entity, it works likeinsert
Given this code:
local entity, err = kong.db.keyauth_credentials:upsert(
{ id = "2b6a2022-770a-49df-874d-11e2bf2634f5" },
{ consumer = { id = "a96145fb-d71e-4c88-8a5a-2c8b1947534c" } }
)
if not entity then
kong.log.err("Error when upserting keyauth credential: " .. err)
return nil
end
Two things can happen:
- If a credential with id
2b6a2022-770a-49df-874d-11e2bf2634f5
exists, then this code will attempt to set its Consumer to the provided one. - If the credential does not exist, then this code is attempting to create a new credential, with the given id and Consumer.
Delete an entity
local ok, err, err_t = kong.db.<name>:delete(primary_key)
Attempts to delete the entity identified by primary_key
. It returns true
if the entity doesn’t exist after calling this method, or nil
+ error +
error table if an error is detected.
Notice that calling delete
will succeed if the entity didn’t exist before
calling it. This is for performance reasons - we want to avoid doing a
read-before-delete if we can avoid it. If you want to do this check, you
must do it manually, by checking with select
before invoking delete
.
Example:
local ok, err = kong.db.keyauth_credentials:delete({
id = "2b6a2022-770a-49df-874d-11e2bf2634f5"
})
if not ok then
kong.log.err("Error when deleting keyauth credential: " .. err)
return nil
end
Cache custom entities
Sometimes custom entities are required on every request/response, which in turn triggers a query on the data stores every time. This is very inefficient because querying the data stores adds latency and slows the request/response down, and the resulting increased load on the data stores could affect the data stores performance itself and, in turn, other Kong nodes.
When a custom entity is required on every request/response it is good practice to cache it in-memory by leveraging the in-memory cache API provided by Kong.
The next chapter will focus on caching custom entities, and invalidating them when they change in the data stores: Caching custom entities.