I am not being able to use the rhdf5 package to access the S3 data from NASA's earthaccess, but without success, can you help me out here? I do need to use the direct cloud access because I am actually using their cloud computing system and the machine is within the same region, so it can read very fast without the need for downloading the data.

I'm using this code (with the correct username and password), but it is throwing an error saying it was unable to open the file.

########################
# Get the s3 credentials
########################
user = "username"
password = "password"

library(httr2)
req <- request("https://data.nsidc.earthdatacloud.nasa.gov/s3credentials") %>% 
    req_auth_basic(user, password) 

res = req %>% httr2::req_options(followlocation=TRUE) %>% httr2::req_perform()
obj = res %>% httr2::resp_body_string() %>% jsonlite::parse_json()

s3_cred <- list(
    aws_region = "us-west-2",
    access_key_id = obj$accessKeyId,
    secret_access_key = obj$secretAccessKey
)


#############
## Search for the link of some data within NASA earthdata
#############
searchUrl = 'https://cmr.earthdata.nasa.gov/search/granules.json?short_name=ATL08&version=006&bounding_box=-44.1503575306397,-13.7583117535016,-44.1006630227247,-13.712436430646'

res = httr2::request(searchUrl) %>% httr2::req_perform()
granules = res %>% httr2::resp_body_json()


# Get the s3Link for the data
library(tidyr)
(s3Link = sapply(granuleATL08$links, `[[`, 'href') %>% grep(pattern = "s3:.*h5$", value=TRUE))

##  's3://nsidc-cumulus-prod-protected/ATLAS/ATL08/006/2018/10/19/ATL08_20181019181413_03220114_006_02.h5'

library(rhdf5)

rhdf5::h5ls(
     file = s3Link,
     s3 = TRUE,
     s3credentials = s3_cred
)

# Error in H5Fopen(file, flags = flags, fapl = fapl, native = native): HDF5. File accessibility. Unable to open file.
# Traceback:
# 
# 1. h5ls(file = s3Link, s3 = TRUE, s3credentials = s3_cred)
# 2. h5checktypeOrOpenLocS3(file, readonly = TRUE, fapl = fapl, native = native)
# 3. H5Fopen(file, flags = flags, fapl = fapl, native = native)

But if I use exactly the same credentials obj but using s3fs it works fine:

s3 = s3fs::S3FileSystem$new(
    aws_access_key_id = obj$accessKeyId,
    aws_secret_access_key = obj$secretAccessKey,
    aws_session_token = obj$sessionToken
)

s3$is_file(s3Link)

## TRUE

I wonder if it is because of the s3:// protocol instead of the https://, but I did try to change it to https://nsidc-cumulus-prod-protected.s3.us-west-2.amazonaws.com but it didn't work either.

I suspect this is a limitation in the HDF5 S3 vitrual file driver, which doesn't let you provide the session token. It looks like that's required to access this bucket, but there's no way to provide it to the HDF5 library, so rhdf5 won't work.

I tried using the s3fs approach to see if there was anyway I could modify rhdf5 to work with that, but I kept running into permissions issues. Running the code you provided with my own NASA credentials, and the running s3$file_copy(s3Link, "/tmp/new.h5") get's me a file containing:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>MHN75DG76917N5XG</RequestId>
<HostId>KAAozK6G/EtMkkv9OSUkAj9GrFQJuZCxIYS8slLQExZWfdJzz4uqH/bnKSLwKLDMFzXUBTlHqd9HIbx5BR/K7A==</HostId>
</Error>

Most of the documentation suggests you have to be running code in in the AWS us-west-2 region to access the bucket, which I am not doing. Are you doing that in the example that works. That might help me understand if I'm doing something wrong my end, or it's just the location I'm running the code.