# Bulk importer

/dashboard/bulk-importer

At Dashboard Bulk importer you can mass import images, albums, and users by parsing the contents of a filesystem location. It is intended to be used when you want to import a massive amount of content that otherwise will be troublesome to import by using the API or the web user interface.

💡 Importing takes the content from the importing path and import it into the system. That's database, filesystem and/or external storage. The system doesn't mirror a directory, it imports it and remove the file from the import path. Failed files will be stored at ./importing/failed/.

# How to use it

💡 Check the repository at chevereto/demo-importing (opens new window) for an example on how to organize your files to use the Bulk content importer.

Bulk importer works by scanning the importing/ path, where folders are used to denote a given parsing format for the files contained within.

Placing the content at these directories following the parsing formats conventions will enable to import images to users created as a folder, add categories, etc.

./importing/	Parsing
parse-users/	Top-level folder as username
parse-albums/	Top-level folders as albums
no-parse/	No folder parsing

👉 Go to dashboard/bulk-importer to review importing jobs.

# Charset

Files and .json metadata must be in UTF-8.

# Command

# Cron entry

The importing command can be automatically scheduled by using CRON (opens new window).

# Threads

You can speed up the process by running the importing in multiple threads by passing different env for THREAD_ID.

# File locking

The importing process can be locked by placing an empty lock file at ./importing/.lock.

The process won't be carried until this file gets removed. This comes handy when you want to import a large dataset and you care about the nested folder structure.

# Parsing formats

The bulk importer scans the target importing directory and creates content accordingly parsing rules described below. User assets (avatar, background image) will get uploaded to the system user folder.

💡 The parser creates users and albums only if those doesn't exists. It detects if username exists, and it does the same for albums (based on the album name + its owner).

# Top-level folder as username

Top-level folders within the importing path will be handled as username. Second-level folders will be parsed as album name.

./<path>
└── rodolfo <- username
    ├── weapons <- album name
    │   ├── machine-gun.jpg
    │   └── rocket-launcher.jpg
    └── logo-alt.png

For the tree above, the parser will:

Create rodolfo user (if the user doesn't exists)
Create album weapons under rodolfo user (if the album doesn't exists)
Upload the images contained in ./<path>/rodolfo/weapons to rodolfo's weapons album
Upload logo-alt.png to rodolfo (no album)

# Top-level folders as albums

Top-level folders within the importing path will be handled as album_name.

./<path>
└── weapons <- album name
    ├── machine-gun.jpg
    ├── rocket-launcher.jpg
    └── logo-alt.png

For the tree above, the parser will:

Create album weapons as guest user (if the album doesn't exists)
Upload the images contained in ./<path>/weapons to weapons album

# No parse

No folder is parsed, only images will be imported as guest.

./<path>
├── machine-gun.jpg
├── rocket-launcher.jpg
└── logo-alt.png

For the tree above, the parser will:

Upload the images contained in ./<path> to public, as guest user.

# Statuses

The importing jobs statuses get defined as follow:

Status	Description
Queued	Job is in the processing queue (default status)
Working	Job is being parsed
Paused	Job is paused
Canceled	Job aborted by the user
Completed	Job completed (everything parsed)

💡 Importing may show "completed" when there's nothing else to parse, but internally it will get re-queued automatically.

# Metadata

The bulk importer supports metadata using the JSON format, same as Google Photos. Metadata must be provided per content basis, meaning that you must use one metadata file for each content.

✅ Metadata must be in UTF-8 format. Don't forget to fix your charset.

Content	Type	Metadata file
rodolfo	username	rodolfo/metadata.json
rodolfo/weapons	album	rodolfo/weapons/metadata.json
rodolfo/weapons/machine-gun.jpg	image	rodolfo/weapons/machine-gun.json
rodolfo/weapons/rocket-launcher.jpg	image	rodolfo/weapons/rocket-launcher.json
rodolfo/logo-alt.png	image	rodolfo/logo-alt.json

Tree below shows metadata for the table above.

./<path>
└── rodolfo
    ├── weapons
    │   ├── machine-gun.jpg
    │   ├── machine-gun.json
    │   ├── rocket-launcher.jpg
    │   ├── rocket-launcher.json
    │   └── metadata.json
    ├── logo-alt.png
    ├── logo-alt.json
    └── metadata.json

Note

Metadata properties merges the content being parsed, meaning that you don't need to explicitly indicate all properties, only what you need to describe.

# Image metadata

JSON metadata file below provides a sample metadata for machine-gun.jpg.

{
    "imageData": {
        "title": "Machine gun",
        "description": "Say hello to my little friend!",
        "nsfw": false,
        "category": {
            "name": "Guns",
            "urlKey": "Guns",
            "description": "Guns don't kill people, sugar does."
        }
    }
}

💡 Image categories will be created if needed long as you provide both name and urlKey.

# Album metadata

JSON metadata file bellow provides a sample metadata for guns/ album.

{
    "albumData": {
        "title": "Guns",
        "description": "Guns should be banned.",
        "access": "private",
        "privacy": {
            "type": "password",
            "password": "12345"
        }
    }
}

For album privacy, you can pick from:

Privacy type	Effect
public	Public album (default)
private	Private album for owner
private_but_link	Same as `private` + those with the album link
password	Will set the password for accessing the album

For example, if you need private_but_link the metadata property should look like this:

{
    "privacy": {
        "type": "private_but_link"
    }
}

# User metadata

JSON metadata file bellow provides a sample metadata for rodolfo/ user.

{
    "userData": {
        "role": "user",
        "name": "Rodolfo Berrios",
        "username": "rodolfo",
        "email": "rodolfo@chevereto.com",
        "website": "https://rodolfoberrios.com",
        "bio": "I'm the Master of Puppets @chevereto, the alpha-omega. He/Him/They.",
        "profileImages": {
            "avatar": "rodo-avatar.jpg",
            "background": "rodo-bkg.jpg"
        },
        "networks": {
            "facebook": "no!",
            "twitter": "no!"
        },
        "timezone": "America/Santiago",
        "language": "cl",
        "private": false
    }
}

Profile images have to be located in the .assets/ folder inside the user folder.

./<path>
└── rodolfo
    └── .assets
        ├── rodo-avatar.jpg
        └── rodo-bkg.jpg

User assets

The type of the image assets must be one of the file formats handled by your installation.