Troubleshoot Object Storage

Find the symptom or the message you see, then follow the fix. Messages appear here exactly as the dashboard, the API or your S3 client shows them, and each fix links to the page that covers the task in full.

Cause

A new key takes up to a couple of minutes to reach storage in every region. Until then, storage refuses requests signed with it, usually with InvalidAccessKeyId.

Fix

Wait a couple of minutes and try again. If the key is still refused, check that the client has the whole access key ID, which starts with AKIA and is 24 characters long. Check too that the client sends requests to our endpoint: a client with no endpoint sends them to Amazon, which does not know the key. With the AWS CLI, add --debug to a command and look for s3.fra.coritan.com:7337 in the request URL (Connect an S3 client).

Cause

Storage answers InvalidAccessKeyId when it no longer accepts a key. That happens when you revoke the key, when the key's service is suspended or has ended, and when the key was limited to a bucket that you deleted. Each of these turns the key off within a couple of minutes.

Fix

Open the service's Access keys tab. A revoked key is no longer in the list, and a key that is turned off shows Inactive. For a suspended service, pay the overdue invoice (Failed payments and suspended services), and its keys work again within a couple of minutes of the service becoming active. In the other cases, create a new key and give it to the client.

Cause

Storage answers SignatureDoesNotMatch when the signature it computes differs from the one the client sent. The secret key is wrong, or only part of it was copied. The same error appears when the client sends the request to an address other than the endpoint, such as an IP address, or through a proxy that changes the Host header. Storage checks the signature against the endpoint's host and port.

Fix

Paste the secret key again from where you saved it, with no space or line break around it. Use the endpoint exactly as the dashboard gives it, such as https://s3.fra.coritan.com:7337, and send requests to it directly. If you lost the secret key, create a new key, because we cannot show a secret key again.

Cause

Storage answers AccessDenied when the key's permissions or scope do not cover the request. A read-only key cannot upload or delete, a key limited to one bucket cannot reach another, and no key can create or delete a bucket. That is why storage refuses aws s3 mb and aws s3 rb, and rclone when it tries to create the bucket before an upload. Storage also refuses a request with no signature, such as a bucket's address opened in a web browser.

Fix

Check the key's Scope and Permissions on the Access keys tab. To upload or delete, use a key with Read and write. For another bucket, use a key limited to that bucket, or one whose scope is All buckets. Create and delete buckets on the Buckets tab (Create and delete buckets). With rclone, set no_check_bucket = true in the remote (rclone). To share an object with someone who has no key, send a presigned URL.

Cause

Storage answers NoSuchBucket when the endpoint has no bucket with that name. The bucket lives in another region, the name lacks your prefix, or the bucket was deleted.

Fix

On the Buckets tab, find the bucket's full name and its Region. Send requests to that region's endpoint, which the Endpoints card on the Overview tab lists, and use the full name, such as u7-assets. A bucket that none of your services lists was deleted, together with its objects.

Cause

The client uses virtual-hosted style, which puts the bucket name in the host, such as u7-assets.s3.fra.coritan.com. Those hosts do not exist. A certificate error for the endpoint's own host, such as s3.fra.coritan.com, means the endpoint has no port, so the client connects to port 443, which serves other Coritan products.

Fix

Turn on path style, as each setup in Connect an S3 client does, and keep :7337 at the end of the endpoint.

Cause

Something between your client and us blocks outgoing connections to port 7337, such as a firewall, a proxy or a VPN. Object Storage answers only on that port.

Fix

Run curl -I https://s3.fra.coritan.com:7337 on the same machine. Any HTTP status in the answer, even 403, means your network reaches us, so check the endpoint in the client. No answer means the port is blocked: allow outgoing HTTPS on TCP port 7337, or ask whoever runs the network to allow it.

Cause

The client sends plain HTTP. Port 7337 accepts HTTPS only, and it answers a plain HTTP request with 400 Bad Request.

Fix

Start the endpoint with https://. In s3cmd, set use_https = True in ~/.s3cfg, or pass --ssl.

Cause

Storage answers 403 for a link that has expired. A link from the API lasts 15 minutes unless you asked for longer, and 1 hour at most. A link that an S3 client signed with your key also stops working when you revoke the key. Each link works only with the method it was signed for, and an upload link signed with a content type needs the same Content-Type header on the upload.

Fix

Create a new link (Share a file with a presigned URL). Send the request with the method the API returned and every header in headers. For a file that someone needs for longer, send a new link each time, or give them a read-only key limited to the bucket.

Cause

The name breaks a naming rule, or your account already uses it. The message under Name says which rule, such as Dots and hyphens cannot be next to each other. The message A bucket named u7-assets already exists means one of your services has a bucket with that name. u7-assets was deleted recently and is still being removed. Pick another name, or wait for the removal to finish. means you deleted a bucket with that name in the last 7 days.

Fix

Change the name to follow Bucket names, or choose another one. The name of a deleted bucket is free again 7 days after the delete.

Cause

The service holds 25 buckets, the most one service may hold. Create bucket… is greyed out, and the API answers 409 with This service already has 25 buckets, which is the most one service may hold.

Fix

Delete a bucket you no longer need (Delete a bucket). It stops counting as soon as you delete it. You can also order another service, which holds up to 25 buckets of its own.

Cause

No region can take a bucket at the moment, or the region you chose has stopped taking them. The order page says No region has a gateway right now, and the bucket dialog says No region is available right now. Creating a bucket in a region that has stopped taking them fails with That region does not offer object storage. An order for such a region fails during setup with the same message, and the service's status becomes failed.

Fix

Choose another region if the list offers one. Otherwise, try again later. For a service whose status is failed, contact support with its service ID.

Cause

The bucket holds objects that the last hourly measurement did not count, so the dialog offered no box to destroy them. We refuse the delete with u7-assets is not empty. Delete it with force to destroy its contents as well.

Fix

Wait for the next measurement and delete the bucket again with the box ticked. You can also delete the objects first (Delete objects and folders), or delete the bucket through the API with force=true (Delete a bucket through the API).

Cause

The service is not active. The banner above the tabs says why: Waiting for payment, Setting up or Suspended. Until the service is active, the dashboard hides Create bucket…, Delete bucket…, Create key…, Upload… and New folder…. You can still browse buckets, download objects and revoke keys.

Fix

For Waiting for payment, select Open invoices and pay the first invoice. Setting up takes under a minute, so reload the page. For Suspended, pay the overdue invoice (Failed payments and suspended services).

Cause

Your browser sends each upload straight to the region's endpoint on port 7337. The message after Could not upload and the file's name says what went wrong. Could not reach the storage gateway means your browser could not connect, usually because a firewall, proxy or VPN blocks the port. The upload link expired or was refused means storage answered 403. Upload failed with a number means storage answered with that HTTP status.

Fix

For Could not reach the storage gateway, try another network, or ask whoever runs yours to allow outgoing HTTPS on port 7337. For the other two, reload the page and upload the file again. Send a large file with an S3 client, which uploads it in parts (Connect an S3 client).

Cause

The folder has an empty folder object, whose key ends in /. New folder… writes one, and so do some S3 clients.

Fix

Delete the folder itself with Delete folder… in its menu, or tick it and delete it (Delete objects and folders).

Cause

We measure each bucket once an hour. The object count and size on the Buckets tab, in the object browser's header and on the Overview tab come from the last measurement.

Fix

Wait for the next measurement. The Measured column on the Buckets tab says when we last measured each bucket. The list of objects comes straight from storage, and the refresh button beside the bucket's name lists the folder again.

Cause

No measurements yet means we have not measured any of the service's buckets in the window you chose. A new service or bucket waits up to an hour for its first measurement, and a service with no buckets has nothing to measure.

Fix

Create a bucket if the service has none, and look again after the next hourly measurement.

Cause

At the last measurement, your buckets held more than the plan's allowance. The warning, such as 12.5 GB above the plan, gives the rate for the storage above it.

Fix

Delete objects or buckets you no longer need, and the warning clears after the next measurement. For a larger allowance, move to a larger plan as Change the plan describes. The warning's Change plan button opens the Billing tab, where a plan change keeps the same allowance.

Cause

We do not record when a key is used. The Last used column on the Access keys tab always shows Never, and the API always returns last_used_at as null.

Fix

Nothing is wrong with the key. To keep track of where each key is used, give it a label that names the place, such as ci deploy.

Cause

The dashboard and the API show a secret key only once, when you create the key.

Fix

Create a new key, move your applications to it, then revoke the old key (Create and revoke access keys).

Cause

We could not create it, or your account already had a bucket with that name, or had one that you deleted in the last 7 days. When we could not create it, the setup line in the Activity table on the Billing tab ends with The first bucket was not created: and a reason. When the name was taken, the line does not mention the bucket.

Fix

Create the bucket on the Buckets tab, with another name if the name is taken (Create a bucket).

400 An object key is required or Object keys are at most 1024 bytes

The presign request has no key, a key that ends in / with get or delete, or a key longer than 1024 bytes (Presign an object).

404 Object storage service not found

The service ID is not an Object Storage service on your account. List your services with GET /api/v1/client/object-storage/services (Object Storage).

404 Bucket not found

The bucket ID is not a bucket of this service, or you deleted the bucket. Take the id from List buckets.

404 Access key not found

The key ID is not a key of this service, or you revoked the key.

409 This service is pending; it can be changed once it is active

The service is not active, and the message names its status. See Buttons to create or change things are missing.

422

A value in the request breaks a rule, and detail says which one, such as Bucket names cannot have a dot next to another dot or a hyphen. Object Storage limits lists every rule.

502 with a message that starts Gateway refused, Could not list, Could not sign the request or Could not delete objects

The region's storage refused or failed the request. Try again in a minute. If it keeps happening, contact support with the whole message.

503 Object storage is not enabled on this platform

Storage in the bucket's region cannot take requests right now. Try again later, and contact support if it lasts.

Errors explains the shape of every error response.

Contact support and include:

• The Service ID from the service's Billing tab.
• The bucket's name and region.
• The access key ID of the key involved. Never send a secret key, to us or to anyone else.
• The endpoint your client uses, and the client's name and version.
• The whole error message, and when it happened.