Key takeaways
- Import a bounded object or page of a prefix rather than starting an unbounded bucket-wide rewrite.
- Write derivatives to a distinct prefix so recursive imports cannot process their own outputs.
- Use fit resizing with an explicit maximum size to preserve aspect ratio, and set zoom to false so smaller originals are not enlarged.
An image backfill is not an in-place edit. It is a repeatable migration from a known source object to a versioned derivative whose quality, dimensions, destination, and application reference can be verified independently. Keeping import and export in Google Cloud Storage preserves storage ownership while moving the media work out of application servers.
What matters most
- Prefer separate read and write Template Credentials with the least permissions each Step needs.
- Keep outputs private until the application verifies them and deliberately changes serving references.
- Persist source generation or version identity, workflow version, and destination path for safe replay.
Treat the job as a versioned backfill
Do not overwrite an image library merely because a new format or dimension looks better on one fixture. Define the source population, transformation version, output namespace, acceptance checks, and reference migration first. The original object should remain recoverable until the derivative has been verified and consumers have moved safely.
Use version-scoped output paths so two transformation policies can coexist during rollout and rollback remains a reference change. The sample path is unique per run rather than derived from the source object, so the application must prevent duplicate processing by checking its own source-to-result record before creating another Assembly.
Separate the input and output namespaces
A recursive /google/import Step can enumerate a source prefix. If /google/store writes beneath that same prefix, a later page or rerun can import generated derivatives as though they were originals. Put outputs in a non-overlapping prefix or separate bucket, and make the distinction obvious in configuration and monitoring.
For a large prefix, process a bounded page and preserve next_page_token outside the Assembly for the next call. Bucket contents can change during a migration, so pair pagination with an application manifest or source inventory when exact once-only coverage matters.
Source prefix
Contains only originals selected for the current migration policy.
Output prefix
Contains versioned derivatives and is never traversed by the source import.
Migration manifest
Maps source identity to workflow version, Assembly ID, and destination object.
Build the Google Storage image Template
The import Step selects a file or bounded prefix, /image/resize creates a WebP whose dimensions fit within the chosen box, and /google/store writes the derivative as private. Set zoom to false when small originals should not be enlarged. Review transparency and color behavior before making WebP the contract for every source family.
The sample uses distinct credential names for read and write access. The reader needs access to the source objects; the writer needs object creation in the destination. Add deletion only when an intentional overwrite policy requires it.
{
"allow_steps_override": false,
"steps": {
"source_images": {
"robot": "/google/import",
"credentials": "gcs-source-read",
"path": "originals/catalog/",
"recursive": true,
"files_per_page": 100
},
"web_derivatives": {
"use": "source_images",
"robot": "/image/resize",
"width": 1600,
"height": 1200,
"resize_strategy": "fit",
"zoom": false,
"format": "webp"
},
"gcs_output": {
"use": "web_derivatives",
"robot": "/google/store",
"credentials": "gcs-output-write",
"path": "derived/web-v1/${assembly.id}/${unique_prefix}/${file.url_name}",
"acl": "private",
"result": true
}
}
}Validate image quality and object behavior
Use fixtures that cover EXIF orientation, transparency, embedded color profiles, very large dimensions, small sources, animation, malformed files, and same-named objects in different folders. Compare rendered appearance as well as dimensions, format, and byte size. A smaller output that changes color or removes required animation is not a successful migration.
Set ACL deliberately. The /google/store schema defaults ACL to public-read, so a private review workflow must override it. Add a cache policy only when it matches the eventual delivery and authorization model.
Roll out with checkpoints and rollback
Record every source object and destination result before changing consumer references. Reconcile missing or failed items by source identity instead of rerunning the complete prefix. If a page fails halfway through, retry only the source objects the application has not recorded as complete. The sample path is unique per run, so processing the same source again writes a new object; de-duplication comes from the application’s source-to-result record, checked before the Assembly is created.
Move a small traffic cohort to the new paths, compare bytes and visual metrics, then expand. Keep originals and the previous application references until the rollback window closes. Deleting source objects is a separate retention decision and should never be an implicit final Step of image processing.
Measure the migration as an operational workload
Track imported objects, processed objects, output bytes, failures by class, retries, and unresolved manifest entries. Compare the derivative savings with import, processing, export, storage, and eventual delivery costs. A backfill that reduces bytes but cannot be resumed or audited is not operationally complete.
Alert on stalled pages, repeated source identities, writes outside the destination prefix, and unexpected public ACLs. Keep a bounded concurrency level so processing and Google API activity do not compete with ordinary application traffic or exhaust the destination’s operational limits.
Technical details worth knowing
- /google/import accepts a file path, a directory path ending in a slash, or an array of paths. Recursive directory imports paginate with next_page_token and files_per_page, and the token for the next call is returned in the imported files’ metadata. This differs from /supabase/import, which paginates with page_number.
- /image/resize with resize_strategy set to fit preserves aspect ratio and keeps each side within the requested bounds.
- /image/resize uses the input format when format is null; setting format to webp deliberately creates a WebP derivative.
- /google/store can set the object path, ACL, Cache-Control metadata, and result URL templates. Its documented default ACL is public-read, so private must be set explicitly for a review-first workflow.
- Write-only Google credentials need storage.objects.create; storage.objects.delete is only needed when the workflow intentionally overwrites existing paths. The linked Google credential documentation describes the write role only, so check Google Cloud’s IAM documentation for the permissions the import credential needs.
- The same provider on both ends does not require the same credential, bucket, or key. Separating read and write authority reduces the impact of a compromised Template Credential.
A practical approach
- 1
Choose a representative source prefix and define a non-overlapping output namespace.
- 2
Create scoped read and write credentials and save the import–resize–store Template.
- 3
Process one bounded page and compare dimensions, color, transparency, bytes, and object metadata.
- 4
Roll out with checkpoints, idempotent records, and a reversible reference migration.
When Transloadit is useful
Use /google/import to select one object or a paginated prefix, /image/resize to create the reviewed derivative, and /google/store to write a private output. Separate read and write credentials when possible, and never place outputs beneath a prefix that the next import will ingest again.
Architecture boundary
Google Cloud Storage remains the durable source and destination. Transloadit reads the selected objects, creates image derivatives, and writes new objects, while the application owns inventory, rollout, deletion, versioning, and the decision to replace references.
Frequently asked questions
Can the workflow overwrite the source object?
It can be given delete-and-write authority, but a versioned destination is safer. It supports review, rollback, and coexistence while consumers migrate.
Can import and store use the same Google credentials?
They can, but separate read and write credentials reduce authority and make the source/destination boundary easier to audit.
Why must the output prefix be separate?
A recursive import can otherwise discover prior derivatives and process them again, multiplying objects and degrading quality.
Does fit resizing always create the requested width and height?
No. Fit preserves aspect ratio and keeps each side within the bounds. Exact dimensions require a different strategy and an explicit cropping or padding decision.
Should the derivatives be public?
Keep them private during validation. Choose public access or a delivery layer later according to the application’s authorization and caching model.