Import files from Cloudflare R2 in Java with Rclone
Use Java’s ProcessBuilder to run rclone copy, then expose the downloaded directory only after
rclone exits successfully. The example below imports a private Cloudflare R2 prefix, keeps nested
filenames, and returns a failing exit status when the transfer fails or exceeds its deadline.
Prepare the bucket and tools
This walkthrough targets a Linux application using OpenJDK 21 and a UTF-8 locale. It was tested with OpenJDK 21.0.12.1 and rclone 1.75.2. Use a maintained update of your JDK; the tested patch is not an intrinsic minimum. rclone is a separate executable, available from its installation guide. No Java dependencies or build tool are needed.
You need an existing R2 bucket containing the files to download. Keep that prefix unchanged during
the import: copying a changing bucket does not produce a point-in-time snapshot. Use keys that map
to distinct Linux file paths, without empty, . or .. path segments, file/directory collisions,
or names your filesystem cannot represent. Spaces, nested paths, Unicode, and zero-byte files are
supported. Arbitrary object keys are not necessarily valid filesystem paths.
Check each required executable in the shell that will launch Java:
command -v java >/dev/null &&
command -v javac >/dev/null &&
command -v rclone >/dev/null &&
java -version && javac -version && rclone version
The commands below select C.UTF-8; that locale must be installed. Work in a writable directory you
own. The destination’s parent must already exist, support atomic directory renames, and have room
for the complete download. Run one importer at a time with exclusive control of that parent.
Configure a private R2 remote
Obtain the Access Key ID and Secret Access Key for an R2 token scoped to your bucket with Object Read only permission. This permission covers reading and listing objects through the S3-compatible API; the importer does not need to create or delete remote objects. See Cloudflare’s R2 authentication guide.
Save the following as r2-import.conf in your working directory using a text editor. Replace the
three placeholders with your credentials and account ID. This endpoint is for a bucket in the
default jurisdiction; use your bucket’s jurisdiction-specific endpoint when applicable.
[cloudflare_r2]
type = s3
provider = Cloudflare
access_key_id = YOUR_ACCESS_KEY_ID
secret_access_key = YOUR_SECRET_ACCESS_KEY
endpoint = https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com
region = auto
no_check_bucket = true
Restrict access to that file:
chmod 600 ./r2-import.conf
The Java program passes this file explicitly with --config, so it does not depend on rclone’s
usual configuration location. no_check_bucket follows Cloudflare’s
rclone configuration guidance for tokens
with object permissions. Do not commit the file or paste credentials into command arguments.
The child’s inherited RCLONE_* variables are removed so settings such as a global dry run cannot
silently turn this import into a different operation.
Save the Java importer
Save this complete file as RcloneImporter.java. The arguments are the rclone executable, config
file, remote prefix, new destination directory, and whole-transfer timeout in seconds. Supply trusted
application configuration, rather than accepting arbitrary executable paths or remotes from users.
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.time.Duration;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
public class RcloneImporter {
private static final int DIAGNOSTIC_BYTES = 8192;
private static void runRclone(ProcessBuilder builder, Duration timeout)
throws IOException, InterruptedException {
Process process = builder.redirectErrorStream(true).start();
var diagnostics = new ByteArrayOutputStream();
var readFailure = new AtomicReference<IOException>();
Thread drain = Thread.ofVirtual().start(() -> {
try (var input = process.getInputStream()) {
byte[] buffer = new byte[4096];
int count;
while ((count = input.read(buffer)) != -1) {
int kept = Math.min(count, DIAGNOSTIC_BYTES - diagnostics.size());
if (kept > 0) diagnostics.write(buffer, 0, kept);
}
} catch (IOException error) {
readFailure.set(error);
}
});
try {
if (!process.waitFor(timeout.toMillis(), TimeUnit.MILLISECONDS)) {
throw new IOException("rclone timed out after " + timeout);
}
drain.join();
if (readFailure.get() != null) throw readFailure.get();
System.err.print(diagnostics.toString(StandardCharsets.UTF_8));
if (process.exitValue() != 0) {
throw new IOException("rclone exited with code " + process.exitValue());
}
} finally {
// Reap the direct rclone child before deleting anything it might still be writing.
process.destroyForcibly();
boolean interrupted = false;
while (process.isAlive() || drain.isAlive()) {
try {
process.waitFor();
drain.join();
} catch (InterruptedException error) {
interrupted = true;
}
}
if (interrupted) Thread.currentThread().interrupt();
}
}
private static void removeStaging(Path staging) throws IOException {
Files.walkFileTree(staging, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attributes)
throws IOException {
Files.delete(file);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult postVisitDirectory(Path directory, IOException error)
throws IOException {
if (error != null) throw error;
Files.delete(directory);
return FileVisitResult.CONTINUE;
}
});
}
public static void importFiles(Path executable, Path config, String source, Path destination,
Duration timeout) throws IOException, InterruptedException {
if (!Files.isExecutable(executable) || !Files.isRegularFile(config)) {
throw new IOException("Check the rclone executable and configuration file");
}
if (!source.startsWith("cloudflare_r2:") || !source.endsWith("/")) {
throw new IllegalArgumentException("Use cloudflare_r2:bucket/prefix/ as the source");
}
if (timeout.isNegative() || timeout.isZero() || timeout.compareTo(Duration.ofDays(1)) > 0) {
throw new IllegalArgumentException("Timeout must be between 1 and 86400 seconds");
}
Path name = destination.getFileName();
if (name == null || name.toString().equals(".") || name.toString().equals("..")) {
throw new IllegalArgumentException("Choose a new destination directory name");
}
Path target = destination.toAbsolutePath();
if (Files.exists(target, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Destination already exists; choose a new directory");
}
Path staging = Files.createTempDirectory(target.getParent(), ".r2-import-");
boolean published = false;
try {
ProcessBuilder builder = new ProcessBuilder(executable.toAbsolutePath().toString(),
"--config", config.toAbsolutePath().toString(), "copy",
"--", source, staging.toString());
builder.environment().keySet().removeIf(key -> key.startsWith("RCLONE_"));
runRclone(builder, timeout);
if (Thread.currentThread().isInterrupted()) throw new InterruptedException();
Files.move(staging, target, StandardCopyOption.ATOMIC_MOVE);
published = true;
} finally {
if (!published) removeStaging(staging);
}
}
public static void main(String[] args) {
if (args.length != 5) {
System.err.println("Usage: java RcloneImporter RCLONE CONFIG REMOTE DEST SECONDS");
System.exit(2);
}
try {
importFiles(Path.of(args[0]), Path.of(args[1]), args[2], Path.of(args[3]),
Duration.ofSeconds(Long.parseLong(args[4])));
System.out.println("Import complete: " + args[3]);
} catch (InterruptedException error) {
Thread.currentThread().interrupt();
System.err.println("Import interrupted; destination was not published");
System.exit(130);
} catch (IOException | IllegalArgumentException error) {
System.err.println("Import failed: " + error.getMessage());
System.exit(1);
}
}
}
Both child output streams go into one pipe. A separate virtual thread keeps draining that pipe,
retaining only the first 8 KiB for terminal diagnostics. This lets the calling thread enforce the
transfer deadline even when rclone produces more output than a pipe can hold. Java’s
Process documentation
explains the risk of blocked subprocess output.
The staging directory is a sibling of the final destination. After a zero rclone exit, Java requests an atomic move. If the filesystem cannot provide it, the import fails. Atomic moves are not an overwrite lock: another writer can race the existence check, which is why this example requires exclusive access.
Compile and import the prefix
Replace /absolute/path/to/rclone with the executable found earlier and use your bucket and prefix.
Keep the quotes around paths. Run the two commands together so a compilation failure cannot launch
an older class file:
javac -proc:none -encoding UTF-8 RcloneImporter.java &&
LC_ALL=C.UTF-8 java -cp . RcloneImporter /absolute/path/to/rclone ./r2-import.conf \
'cloudflare_r2:my-data-bucket/source-files/' ./downloaded-files 600
On success, the program prints Import complete: ./downloaded-files and exits zero. An object such
as source-files/reports/July résumé.txt becomes downloaded-files/reports/July résumé.txt.
copy copies the prefix’s contents, without adding
source-files as another directory. It does not remove the remote objects.
The 600-second deadline includes rclone’s retries, so even a valid file may exceed it on a slow connection. A timeout kills and waits for the direct rclone child, removes this invocation’s staging directory, and leaves the final destination absent. Choose a longer timeout, up to 86,400 seconds, and retry with the same destination after the failed process has exited. An existing destination, including a symlink, is refused; use a new name for another completed import.
Diagnose an unsuccessful import
- Child exit: Read the capped terminal diagnostics, then check the selected config file, token’s bucket scope, endpoint, and outbound HTTPS access. A failed request is an error, not an empty successful download.
- No files: An empty or nonexistent prefix can copy successfully as an empty directory. Confirm the prefix and inspect the result before using it; zero exit does not prove a particular object existed. Keep an independent expected inventory or publisher checksums when those are required.
- Interruption: An application calling
importFilescan interrupt its waiting Java thread; the method stops the child and cleans staging before propagatingInterruptedException. The caller owns restoring its interrupt flag. This is distinct from killing the JVM. - Abrupt termination: Power loss,
kill -9, and JVM termination can leave.r2-import-*directories or an orphaned child. Stop any remaining child before removing only the staging directory belonging to that invocation. These directories are incomplete work, never the final destination. Cleanup can also fail if filesystem permissions or availability change.
Choose where imports run
This approach fits a Java service that already uses rclone and needs local files before processing them. If the files should enter a hosted Transloadit workflow instead, see the Cloudflare Import Robot and Java SDK.