Effortless file export to OpenStack Swift in .NET C#
Upload files through Swift’s HTTP API with .NET’s built-in HttpClient. This example creates real
segments and publishes a static large-object manifest only after every segment is acknowledged.
Introduction to OpenStack Swift
Swift stores objects in containers. Its static large-object API combines an ordered list of uploaded segments into one downloadable object. The deployment must enable the SLO middleware for this example.
Setting up your environment
Prerequisites
Install a maintained .NET 10 SDK. The example was tested in Bash on Linux with SDK 10.0.111, runtime 10.0.11, and Swift 2.38.2. Obtain a project-scoped token and the object-storage account URL through your deployment’s OpenStack authentication workflow. The token needs access to create containers, upload objects, read segment metadata for SLO validation, and delete failed upload segments.
The sample uses 5 MiB segments and at most 1,000 segments (5,242,880,000 bytes, including that
endpoint). Confirm your deployment’s SLO, object-size, and quota limits allow these values.
The entire upload has a five-minute deadline, so a slower connection can fail below the size limit.
Use ASCII local paths and project directory names. The program enforces this restriction before
opening the file so malformed filename bytes cannot select a different file after argument decoding.
Remote object keys can contain Unicode.
Start in a writable directory with no SwiftExport entry. Enclosing .NET SDK, NuGet, and MSBuild
configuration still applies.
if ! command -v dotnet >/dev/null 2>&1; then
printf '%s\n' 'Install the .NET 10 SDK first.' >&2
false
elif [ -e SwiftExport ] || [ -L SwiftExport ]; then
printf '%s\n' 'SwiftExport already exists; choose another parent directory.' >&2
false
else
dotnet new console --framework net10.0 --name SwiftExport --no-restore &&
dotnet restore SwiftExport/SwiftExport.csproj &&
cd SwiftExport
fi
Successful setup enters the project directory. A failed restore leaves the new project in place
and keeps your shell in the parent directory. Fix the reported SDK or NuGet problem, then run
dotnet restore SwiftExport/SwiftExport.csproj && cd SwiftExport there to finish setup.
No additional NuGet packages are needed.
Secure configuration management
Set SWIFT_STORAGE_URL to the complete HTTPS account endpoint from the service catalog, including
its /v1/AUTH_… path. Set SWIFT_TOKEN to the scoped token. Supply them through your shell or
secret manager. Token acquisition and renewal belong to your deployment’s Keystone workflow.
An expired token causes a nonzero exit; obtain a new scoped token before rerunning.
Implementing Swift storage operations
Replace Program.cs with this complete program. The injected HttpClient belongs to the caller;
the exporter disposes each request and response. Redirects are disabled so the token is not forwarded
to a different destination.
using System.Net.Http.Headers;
using System.Security.Cryptography;
using System.Text.Json;
public static class Program
{
public static async Task<int> Main(string[] args)
{
using var deadline = new CancellationTokenSource(TimeSpan.FromMinutes(5));
Console.CancelKeyPress += (_, e) => { e.Cancel = true; deadline.Cancel(); };
try
{
if (args.Length != 3) throw new ArgumentException("Pass container, object key, and local file.");
if (args[2].Any(character => character > 127) ||
Directory.GetCurrentDirectory().Any(character => character > 127))
throw new ArgumentException("Use ASCII local paths and working directory names.");
var endpoint = new Uri(Required("SWIFT_STORAGE_URL"));
if (endpoint.Scheme != "https" || endpoint.UserInfo.Length != 0 ||
endpoint.Query.Length != 0 || endpoint.Fragment.Length != 0)
throw new ArgumentException("Use the HTTPS account endpoint from the service catalog.");
using var handler = new SocketsHttpHandler { AllowAutoRedirect = false, MaxConnectionsPerServer = 4 };
using var http = new HttpClient(handler) { Timeout = Timeout.InfiniteTimeSpan };
var exporter = new SwiftExporter(http, endpoint, Required("SWIFT_TOKEN"));
await exporter.UploadAsync(args[0], args[1], args[2], deadline.Token);
Console.WriteLine("Uploaded.");
return 0;
}
catch (OperationCanceledException) { Console.Error.WriteLine("Upload canceled or timed out."); }
catch (HttpRequestException error) { Console.Error.WriteLine($"Swift request failed (HTTP {error.StatusCode})."); }
catch (Exception) { Console.Error.WriteLine("Upload failed. Check the file and Swift configuration."); }
return 1;
}
private static string Required(string name) =>
Environment.GetEnvironmentVariable(name) is { Length: > 0 } value
? value : throw new ArgumentException($"Missing {name}.");
}
public sealed class SwiftExporter(HttpClient http, Uri endpoint, string token)
{
private const int SegmentSize = 5 * 1024 * 1024;
private const int MaxSegments = 1000;
private static string Encode(string value)
{
var parts = value.Split('/');
if (parts.Any(part => part.Length == 0 || part == "." || part == ".."))
throw new ArgumentException("Use nonempty path segments other than dot or dot-dot.");
return string.Join("/", parts.Select(Uri.EscapeDataString));
}
private async Task<string?> PutAsync(string path, HttpContent content,
bool container, CancellationToken ct, string? etag = null, string query = "")
{
content.Headers.ContentType ??= new MediaTypeHeaderValue("application/octet-stream");
using var request = new HttpRequestMessage(HttpMethod.Put,
endpoint.AbsoluteUri.TrimEnd('/') + "/" + Encode(path) + query) { Content = content };
request.Headers.Add("X-Auth-Token", token);
if (etag != null) request.Headers.Add("ETag", etag);
using var response = await http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, ct);
var status = (int)response.StatusCode;
if (status != 201 && !(container && status == 202))
throw new HttpRequestException("Unexpected Swift response.", null, response.StatusCode);
// Swift can return an unquoted MD5 that the typed ETag accessor cannot parse.
return response.Headers.TryGetValues("ETag", out var values)
? values.Single().Trim('"') : null;
}
public async Task UploadAsync(string container, string key, string file, CancellationToken ct)
{
if (container.Contains('/')) throw new ArgumentException("A container cannot contain slashes.");
Encode(container);
Encode(key);
await using var input = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read);
long length = input.Length;
if (length > (long)SegmentSize * MaxSegments) throw new IOException("File exceeds the example limit.");
await PutAsync(container, new ByteArrayContent([]), true, ct);
if (length == 0)
{
await PutAsync(container + "/" + key, new ByteArrayContent([]), false, ct);
return;
}
string segments = container + "_segments";
await PutAsync(segments, new ByteArrayContent([]), true, ct);
string version = Guid.NewGuid().ToString("N");
var manifest = new List<object>();
var attempted = new List<string>();
bool publicationStarted = false;
var buffer = new byte[SegmentSize];
long remaining = length;
try
{
for (int index = 0; remaining > 0; index++)
{
int count = (int)Math.Min(remaining, SegmentSize);
await input.ReadExactlyAsync(buffer.AsMemory(0, count), ct);
string hash = Convert.ToHexString(MD5.HashData(buffer.AsSpan(0, count))).ToLowerInvariant();
string name = $"{segments}/{version}/{index:D8}";
attempted.Add(name);
using var content = new ByteArrayContent(buffer, 0, count);
string? returned = await PutAsync(name, content, false, ct, hash);
if (!string.Equals(hash, returned, StringComparison.OrdinalIgnoreCase))
throw new IOException("Swift did not confirm the segment checksum.");
manifest.Add(new { path = "/" + name, etag = hash, size_bytes = count });
remaining -= count;
}
if (input.Length != length || input.ReadByte() != -1) throw new IOException("The input file changed.");
using var manifestContent = new StringContent(JsonSerializer.Serialize(manifest));
manifestContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
// After dispatch, a lost response cannot tell us whether these segments are live.
publicationStarted = true;
await PutAsync(container + "/" + key, manifestContent, false, ct,
query: "?multipart-manifest=put");
}
catch
{
if (!publicationStarted) await RemoveAttemptedAsync(attempted);
throw;
}
}
private async Task RemoveAttemptedAsync(List<string> paths)
{
using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(15));
foreach (string path in paths)
{
try
{
using var request = new HttpRequestMessage(HttpMethod.Delete,
endpoint.AbsoluteUri.TrimEnd('/') + "/" + Encode(path));
request.Headers.Add("X-Auth-Token", token);
using var response = await http.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead, cleanup.Token);
if ((int)response.StatusCode != 204 && (int)response.StatusCode != 404)
Console.Error.WriteLine("An unreferenced segment could not be removed.");
}
catch (HttpRequestException) { Console.Error.WriteLine("Segment cleanup lost its connection."); }
catch (OperationCanceledException) { Console.Error.WriteLine("Segment cleanup timed out."); break; }
}
}
}
Handling large files
Each nonempty file gets a fresh segment prefix in <container>_segments. Sequential uploads keep
the payload buffer at 5 MiB. The manifest contains each segment’s path, ETag, and size, which Swift
validates before publishing the object. An empty file uses a regular empty PUT, avoiding
deployment-specific support for empty SLO manifests.
The remote object key can contain spaces and Unicode. Encoding preserves path separators and rejects empty, dot, and dot-dot segments to avoid URI normalization. Keep the local file unchanged throughout the upload; size checks cannot detect every concurrent same-size edit.
Implementing robust error handling
Container creation accepts HTTP 201 or 202; object and manifest uploads require 201. The segment checksum is compared with Swift’s raw ETag header, including deployments that omit quotation marks. Quota, authentication, checksum, connection, and cancellation failures exit nonzero. Ctrl+C and the five-minute deadline propagate to file reads and HTTP operations. This example does not retry.
Before manifest submission, a failed upload attempts to delete only its own exact segment names. Cleanup has a separate 15-second budget, so cancellation does not skip it; failed deletes can still leave orphaned segments. The existing object stays available until its replacement is published.
After manifest submission starts, the program retains the new segments on failure. A lost response can hide a successful publication, so deleting those segments could break the object. Inspect the current manifest before retrying or collecting unreferenced versions. A replacement uses a fresh prefix and an explicit segment list, preventing a shorter file from picking up an older tail. Old segments remain after replacement; collect only versions you own that have no remaining references or active readers. This example does not implement that maintenance workflow.
Performance optimization
Reuse the client for a batch of exports. The connection limit belongs to SocketsHttpHandler.
If you add concurrent uploads, bound the concurrency and budget one segment buffer per operation.
Practical example: complete implementation
With both environment variables configured, place the file you want to export at report.bin in
the project directory. This command creates or replaces the remote object at the given key:
dotnet run -- exports 'reports/report #1.bin' report.bin
The command creates exports and exports_segments if necessary, uploads the segments, and finally
publishes reports/report #1.bin. An empty input creates a regular empty object instead.
The command prints Uploaded. and exits zero only after the final
successful response. Download the object through Swift’s ordinary authenticated GET API to compare
its bytes with report.bin; a nonempty object’s SLO ETag is not the MD5 of the complete file.