Following gives short examples for accessing anonymous S3 buckets with different tools

rclone

Best choice for bulk transfers. Anonymous access simply means omitting access_key_id and secret_access_key.

Add a remote to ~/.config/rclone/rclone.conf:

[public]
type = s3
provider = Other
endpoint = https://s3.example.org
force_path_style = true

Then: 

rclone lsf "public:$BUCKET/$PREFIX" # list
rclone lsf "public:$BUCKET/" --dirs-only --max-depth 1 # top-level prefixes
rclone size "public:$BUCKET/$PREFIX" # object count and volume
rclone copy "public:$BUCKET/$KEY" ./download/ --progress # one object
rclone copy "public:$BUCKET/$PREFIX" ./download/ --progress --transfers 8
rclone cat "public:$BUCKET/$KEY" --count 8 | od -c # stream, no download


Without a config file, quote the endpoint — rclone splits connection strings on : and would otherwise read it as the bare string https:

rclone lsf ':s3,provider=Other,endpoint="'"$ENDPOINT"'":'"$BUCKET/$PREFIX"


AWS CLI

Use --no-sign-request for anonymous access and --endpoint-url for a non-AWS endpoint. The CLI uses path-style automatically with a custom endpoint.

export AWS_DEFAULT_REGION=us-east-1 # botocore insists on a region
export AWS_EC2_METADATA_DISABLED=true # skip the IMDS lookup
alias s3anon="aws s3 --no-sign-request --endpoint-url $ENDPOINT"

Then:


s3anon ls "s3://$BUCKET/" # top-level prefixes
s3anon ls "s3://$BUCKET/$PREFIX" # list a prefix
s3anon cp "s3://$BUCKET/$KEY" ./download/ # one object
s3anon sync "s3://$BUCKET/$PREFIX" ./download/ --dryrun # bulk, preview first

# object metadata / byte range
aws s3api head-object --no-sign-request --endpoint-url "$ENDPOINT" --bucket "$BUCKET" --key "$KEY" --query ContentLength
aws s3api get-object --no-sign-request --endpoint-url "$ENDPOINT" --bucket "$BUCKET" --key "$KEY" --range bytes=0-7 head.bin

Example

aws s3 ls --no-sign-request --endpoint-url https://s3.r1.cloud.eumetsat.int s3://seviri-meteosat-0-degree.fcdr.level15.netcdf/

Python — boto3

signature_version=UNSIGNED selects anonymous access.


import boto3
from botocore import UNSIGNED
from botocore.config import Config

ENDPOINT = "https://s3.example.org"
BUCKET = "my-bucket"
PREFIX = "some/path/"
KEY = "some/path/object.nc"

s3 = boto3.client(
	"s3",
	endpoint_url=ENDPOINT,
	config=Config(
		signature_version=UNSIGNED, # anonymous
		s3={"addressing_style": "path"}, # needed for dotted bucket names
	),
)

# top-level prefixes
for p in s3.list_objects_v2(Bucket=BUCKET, Delimiter="/").get("CommonPrefixes", []):
	print(p["Prefix"])

# every object under a prefix
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=PREFIX):
	for obj in page.get("Contents", []):
		print(obj["Key"], obj["Size"])

s3.head_object(Bucket=BUCKET, Key=KEY)["ContentLength"] # size
s3.get_object(Bucket=BUCKET, Key=KEY, Range="bytes=0-7")["Body"].read()
s3.download_file(BUCKET, KEY, "./local_copy.nc") # multipart-aware

Python — s3fs

anon=True selects anonymous access. Use this when you want a filesystem-like API, or to read part of a remote netCDF/HDF5 file without downloading it.


import s3fs

fs = s3fs.S3FileSystem(
	anon=True, # anonymous
	client_kwargs={"endpoint_url": "https://s3.example.org"},
	config_kwargs={"s3": {"addressing_style": "path"}}, # dotted bucket names
)

fs.ls(BUCKET) # top-level
fs.ls(f"{BUCKET}/{PREFIX}", detail=True) # with sizes
fs.info(f"{BUCKET}/{KEY}")["size"]
fs.get(f"{BUCKET}/{KEY}", "./local_copy.nc") # download

with fs.open(f"{BUCKET}/{KEY}", "rb") as f:
	f.read(8)

Reading a remote netCDF lazily (needs xarray, h5netcdf, h5py) — only the chunks you touch are fetched:


 import xarray as xr

with fs.open(f"{BUCKET}/{KEY}", "rb", block_size=4 * 1024 * 1024) as f:
	with xr.open_dataset(f, engine="h5netcdf", group="measurements") as ds:
		tile = ds["counts_hrv"][5000:5004, 2700:2706].values


Example — the snippet above on the SEVIRI bucket reads a 4×6 tile out of an 11136×5568 array in a 245 MiB file in about 2 seconds:
[[162 161 163 155 153 155]
[168 162 166 156 149 163]
[188 159 150 157 149 148]
[177 160 151 161 152 145]]

s3cmd — not supported

s3cmd cannot access anonymous buckets (with 2.4.0)



IPv6 note. These instructions assume your VM has IPv4, which EWC VMs do.

Some endpoints — s3.r1.cloud.eumetsat.int among them — publish both A and AAAA records but reject anonymous requests arriving over IPv6 with an empty-message 403 AccessDenied.

On a dual-stack host this looks like random failure (curl/rclone fail ~half the time; boto3/s3fs/AWS CLI fail every time, and AWS CLI v2 crashes with argument of type 'NoneType' is not a container or iterable) — in these cases, force IPv4 with curl -4, rclone --bind 0.0.0.0, or an /etc/hosts entry pinning the endpoint to its IPv4 address.

On an IPv6-only VM there is no workaround: use an IPv4-capable host, or ask the storage provider to fix the IPv6 path.






  • No labels