Efficiently import files from Supabase in Python

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

To import files from Supabase in Python, download their bytes with the official open-source SDK and save them locally. The script below handles a single object or every visible object beneath a folder, including later listing pages and nested folders. It refuses to overwrite local files and returns a failing exit status if an import stops.

Set up Supabase and Python

You need a Supabase project, a private bucket, and an email/password user who can read its objects. This walkthrough uses a POSIX shell and a local filesystem that supports hard links. It was tested on Linux with Python 3.12.13, supabase 2.13.0, and storage3 0.11.3. Use a maintained Python 3.12 patch release; the exact patch above is the tested version.

1. Install required Python libraries

In a directory you control, with python3.12 and its venv module installed, paste:

(
  command -v python3.12 >/dev/null || exit 1
  if [ -e .venv ] || [ -L .venv ]; then
    printf 'Choose a directory without an existing .venv.\n' >&2
    exit 1
  fi
  python3.12 -m venv .venv &&
    .venv/bin/python -m pip install 'supabase==2.13.0'
)

The subshell leaves your current directory and shell options intact. An installation failure can leave a partial .venv; keep it for diagnosis and try again in a fresh directory. The commands below use this virtual environment explicitly, without activating it.

2. Configure access credentials and permissions in Supabase

Supply these environment variables through your process environment or secret manager:

This recipe targets the pinned client’s legacy JWT key interface. Version 2.13.0 validates project keys as JWTs⁠, so sb_publishable_… keys require a newer client. Use the anon key here, and authenticate the ordinary user; keep service-role keys out of the importer. Supabase plans to deprecate legacy keys at the end of 2026⁠; this recipe applies to projects that still expose them.

For a small example, have your project owner create a private my-bucket bucket and upload image.jpg and nested/photo.jpg beneath <user-uuid>/images. The user’s UUID is their Auth user ID, not their email. The project owner can grant read access to each user’s UUID folder with this policy in the SQL editor, replacing my-bucket if needed:

create policy "Read own UUID folder"
on storage.objects for select to authenticated
using (
  bucket_id = 'my-bucket'
  and (storage.foldername(name))[1] = (select auth.uid())::text
);

Both listing⁠ and downloading⁠ require SELECT permission on storage.objects. The importer needs no bucket-creation or upload permission. Review any existing policies too: permissive policies can grant access beyond this example’s folder restriction. See the Storage access control guide⁠.

3. Establish a connection to your Supabase bucket

Save this complete program as import_supabase.py. It signs in before creating the bucket handle. Keep the destination’s parent directories under your control, and run imports sequentially without another process changing those directories. The script refuses symlink parents and unsafe remote path components.

import argparse
import os
import sys
import tempfile
from pathlib import Path
from urllib.parse import quote

from supabase import create_client

VALID_NAME_CHARACTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-!.*'() &$=@;+,?"


def validate_path(value: str, allow_empty: bool = False) -> None:
    if allow_empty and value == '':
        return
    for name in value.split('/'):
        if (not name or name in {'.', '..'}
                or any(char not in VALID_NAME_CHARACTERS for char in name)):
            raise ValueError('Unsafe remote path component')


def download_file(bucket, key: str, target: Path) -> None:
    validate_path(key)
    print(f'Downloading {key}', flush=True)
    for parent in [target.parent, *target.parent.parents]:
        if parent.is_symlink():
            raise ValueError('Symlink destination parent')
    target.parent.mkdir(parents=True, exist_ok=True)
    data = bucket.download(quote(key, safe='/'))
    with tempfile.TemporaryDirectory(dir=target.parent) as staging:
        temporary = Path(staging) / 'download'
        temporary.write_bytes(data)
        # Linking publishes a complete file and refuses an existing destination.
        os.link(temporary, target)
    print(f'Downloaded {key}')


def import_directory(bucket, prefix: str, root: Path) -> int:
    prefix = prefix.rstrip('/')
    validate_path(prefix, allow_empty=True)
    pending = [(prefix, root)]
    count = 0
    while pending:
        remote, local = pending.pop()
        offset = 0
        while True:
            rows = bucket.list(path=remote, options={
                'limit': 100, 'offset': offset,
                'sortBy': {'column': 'name', 'order': 'asc'},
            })
            for row in rows:
                name = row['name']
                validate_path(name)
                if '/' in name:
                    raise ValueError('Expected an immediate child name')
                key = f'{remote}/{name}' if remote else name
                target = local / name
                if row.get('id') is None:
                    pending.append((key, target))
                    continue
                download_file(bucket, key, target)
                count += 1
            if len(rows) < 100:
                break
            offset += 100
    return count


def main() -> None:
    parser = argparse.ArgumentParser(description='Download Supabase objects without overwriting')
    parser.add_argument('mode', choices=['file', 'directory'])
    parser.add_argument('bucket')
    parser.add_argument('source')
    parser.add_argument('destination')
    args = parser.parse_args()
    client = create_client(os.environ['SUPABASE_URL'], os.environ['SUPABASE_PROJECT_KEY'])
    client.auth.sign_in_with_password({
        'email': os.environ['SUPABASE_USER_EMAIL'],
        'password': os.environ['SUPABASE_USER_PASSWORD'],
    })
    bucket = client.storage.from_(args.bucket)
    destination = Path(args.destination).absolute()
    if args.mode == 'file':
        download_file(bucket, args.source, destination)
        count = 1
    else:
        count = import_directory(bucket, args.source, destination)
    print(f'Imported {count} file(s)')


if __name__ == '__main__':
    try:
        main()
    except Exception as error:
        print(f'Import stopped ({type(error).__name__}). Check credentials, '
              'Storage SELECT policies, and destination paths.', file=sys.stderr)
        sys.exit(1)

download() returns the whole object as bytes. The temporary file is written first; os.link() then creates the destination without replacing a file, directory, or symlink already at that path. Staging is removed when the block exits. This requires hard-link support in the destination filesystem; it does not use streaming downloads.

How to import files from Supabase using Python

Importing a single file

With the environment variables above set, run:

.venv/bin/python import_supabase.py file my-bucket "$SUPABASE_SOURCE_PREFIX/image.jpg" local/image.jpg

The object’s bytes are saved at local/image.jpg. A successful run reports the downloaded key and an imported count of one. An existing destination causes FileExistsError and exit status 1; choose a new destination instead of removing a file you need.

Importing multiple files from a directory

Use the same program to copy the entire visible folder tree:

.venv/bin/python import_supabase.py directory my-bucket "$SUPABASE_SOURCE_PREFIX" downloads

For the two-object setup above, you get downloads/image.jpg and downloads/nested/photo.jpg, with an imported count of two. Listing rows with a null object ID represent folders; the script lists their children rather than downloading those rows. Every folder is paginated independently, and the offset includes folder rows as well as files.

This example accepts ASCII names allowed by Storage⁠, including spaces and question marks. It encodes them for download while keeping their local names unchanged. Empty files are valid. The path checks additionally reject empty components, . and .., backslashes, colons, percent signs, hash signs, and non-ASCII characters. Existing destination files are preserved; a collision stops the import rather than skipping a file and reporting success.

Storage limitations and considerations

Each download holds one complete object in memory, and each listing holds up to 100 immediate children. Choose files that fit in available memory. A large folder tree also accumulates pending folder paths; this example sets no whole-import memory or time limit.

Keep the remote tree stable while importing. Offset pagination is not a snapshot: concurrent uploads, renames, or deletions can cause missing or repeated entries. A failure stops the import with exit status 1 and keeps files already completed; the directory import is not a transaction. After correcting the cause, use a fresh destination so those completed files do not block a rerun.

Diagnose a stopped or empty import

An authentication error calls for checking the project URL, legacy anon key, and confirmed user’s credentials. A Storage error calls for checking the bucket, exact object path, and that user’s SELECT policies. An OSError, including FileExistsError, calls for checking local permissions, existing destinations, free disk space, and hard-link support. The last attempted key appears in the progress output so you can identify a failed object. Provider responses and credentials are not printed by the script.

An imported count of zero means the listing exposed no objects. It does not prove the bucket is empty: RLS can hide objects from that user. Confirm the expected inventory with the project owner before treating a local copy as complete. Bucket MIME restrictions and upload size limits⁠ govern uploads; they do not validate the contents of downloaded files.

Use the imported files

Read the saved files with your application’s usual Python libraries. If you need a separate media processing workflow after download, Transloadit’s Python SDK is further reading; the downloader above does not submit files for processing.