Batch process and migrate large media libraries
Import hundreds of gigabytes or terabytes from existing cloud storage, run one automated workflow across many files, and migrate the results back.
Read large media libraries from S3, Google Cloud, Azure, SFTP, HTTP, and other supported sources.
Submit Assembly-sized batches while Transloadit schedules concurrent jobs across a fleet that can scale to 1,500 machines.
Export processed files to the original storage location or move them to a new provider and prefix.
DSC_08452-1200w.jpg420 KB
contract-p1.png180 KB
keynote-720p.mp488 MB
quarterly-report.pdf310 KBFrom hundreds of gigabytes to terabyte-scale migrations
Use Transloadit as a managed file processing API for a one-time cloud storage migration, a large media backfill, or recurring bulk file processing. Import from your existing storage, transform each file, and export the results without moving the entire library through your application servers.
Inventory the source
Choose the buckets, prefixes, or object paths to migrate and estimate the total files and bytes.
Paginate into Assemblies
Paginate large storage prefixes into jobs of roughly 500–1,000 files and submit them at a controlled rate.
Process concurrently
Apply one saved Template across the migration while Batch Job Slots govern parallel processing in each region.
Export and reconcile
Write results to the same or a new storage destination, then verify each Assembly before marking files complete.
Your application
- Inventory
- Paginate
- Submit
- Assemblies submitted
Transloadit
- Import
- Schedule
- Transform
- Export
- Report
- Results returned
Your application
- Record IDs
- Approve
- Publish
You own the ends, we own the middle
Your application keeps the parts that depend on your data: choosing what to migrate, submitting it in Assembly-sized batches at a rate you control, and recording what came back. That is an inventory loop and an idempotent callback handler.
Everything between those two points is ours: importing the bytes, scheduling across the fleet and its queues, running the same Template over every file, exporting the results, and accounting for errors per Assembly. See how queues and concurrent processing work.
Use our fleet instead of building your own batch processors
Run multiple Assemblies concurrently while Batch Job Slots provide a predictable per-region allowance.
Keep bulk imported work in the Batch Queue so live uploads retain priority during a migration.
Import from one provider and export processed results to the same location or a new destination.
Save a reviewed Template once, then apply the same processing graph to every page of the library.
Read existing files from S3, Google Cloud, Azure, SFTP, HTTP, and other supported sources.
Poll Assembly Status JSON or consume signed Notifications without holding an application request open.
Take Batch Processing into production
The guides, Robot reference, and working demos teams reach for when Batch Processing has to run reliably at scale.
“I’m not usually one for fanboy-ism but Transloadit is one service I would recommend without hesitation. Their product and customer support is outstanding.”
Dave Perrett, CTO at Shuttlerock
Guides
- Media automation: from upload to reliable outputAutomate repeatable media intake, transformation, validation, and export while preserving observability and control.
- A complete guide to digital-asset workflowsDesign a digital-asset workflow from intake and processing through review, publication, retention, and deletion.
- Integrating DAM workflow systems with media processingConnect DAM states to media-processing jobs without duplicating ownership or creating callback races.
- File Upload API guide: architecture, security, and provider selectionChoose and implement a File Upload API by comparing architecture, resumability, direct-to-cloud transfer, security, storage boundaries, and providers.
