How to list AWS files?

David_Cox · February 16, 2019, 11:27am

I am successfully using CURLSMBS.SetupAWS() to PUT, GET and DELETE files from my AWS bucket. The only other verb is POST.

How can I get a list of the files within a designated folder of my AWS bucket?

I cannot see a LIST-type example in the MBS examples folder. I want to be able to verify that my file was uploaded correctly (sometimes it disappears even though I receive error = 0).

Christian_Schmitz · February 16, 2019, 11:52am

You could make a GET request with

/BucketName/?list-type=2

and the bucket name in the URL.

David_Cox · February 16, 2019, 7:34pm

It sort of works too well!

tempCURLSMBSS3.OptionCustomRequest = "GET/" + BucketName + "/&delimiter=/&prefix=mySubFolder1/mySubFolder2/myFileName.txt/?list-type=2"

This works! But whether I add one subfolder or two (as above), or add a file name, it always returns an XML or EVERY file in EVERY folder. Now, I could parse this for just the file I was looking for, but the list could get massive over time.

I have tried using many combinations of &prefix= and &delimiter, but it returns the same (every file).

How may I look inside a sub-folder and for just a single file?

David_Cox · February 16, 2019, 7:36pm

I was using the documentation here:
https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

Christian_Schmitz · February 16, 2019, 7:56pm

there is a space missing after your GET!

Christian_Schmitz · February 16, 2019, 7:56pm

and this should really just go in the URL.
GET is by default. Just put the stuff in the URL.

Tim_Dietrich · February 16, 2019, 10:58pm

@David Cox:

I ran into a similar problem, and resolved it by passing the query params (as an array of strings) to the SetupAWS method.

For example:

[code]Dim AWSAccessKeyId as String = “your-access-key”
Dim AWSSecretAccessKey as String = “your-secrey”
Dim Region as String = “us-east-1”
Dim BucketName as String = “your-bucket-name”
Dim Service as String = “s3”
Dim Path as String = “/” + BucketName + “/”
Dim Domain as String
Dim Verb as String = “GET”
Dim HashedPayload as String

Dim QueryParams() As String
QueryParams.Append(“prefix=temp”)
QueryParams.Append(“list-type=2”)

Dim c As New CURLSMBS

Dim s As Boolean = c.SetupAWS(AWSAccessKeyId, AWSSecretAccessKey, Region, Service, Path, Domain, Verb, HashedPayload, Nil, QueryParams)

Dim p As Integer = c.Perform

Dim OutputData as String = c.OutputData
Dim HTTPResult as Integer = c.GetInfoResponseCode

TextArea1.Text = HTTPResult.ToText + EndOfLine + OutputData[/code]

Note the “QueryParams” array and its use as a param to the SetupAWS method. In my case, the subfolder is “temp.” I have not tested this with multi-level subfolders, but it should work.

You might want to check the version of the CURLSMBS plugin that you have installed to ensure that it supports the “QueryParams” parameter.

On a related note: @Christian Schmitz, the S3 support that you’ve added to the CURLSMBS plugin is amazing. Thank you!

Christian_Schmitz · February 17, 2019, 7:05am

Thanks.

David_Cox · February 17, 2019, 12:36pm

@Christian Schmitz
Even with a space after the GET, it still returned every file name

tempCURLSMBSS3.OptionCustomRequest = "GET /" + BucketName + "/?list-type=2&delimiter=/&prefix=Server0002/User000025/0000029675-0001.CubeiTz"

@Tim Dietrich
Adding one level of folder within bucket works, in that it shows me a list of all the files in that folder (see below). But

QueryParams.Append("prefix=Server0002")

if I add another folder level down,

QueryParams.Append("delimiter=/") QueryParams.Append("prefix=Server0002/User000025")
or

QueryParams.Append("prefix=Server0002/User000025/")

it gives me the OutputData error:

[code]<?xml version="1.0" encoding="UTF-8"?>
SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.MyAccessKeyAWS4-HMAC-SHA256
20190217T121045Z
20190217/us-west-2/s3/aws4_request
f2339160638268430805aa6f18f64521ceef8067004e612c7d62092cbbf2461e0d10fa51f0eee815af27d70e3835cded72d8b1a576515f05b7cca2748c84101741 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 39 30 32 31 37 54 31 32 31 30 34 35 5a 0a 32 30 31 39 30 32 31 37 2f 75 73 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 66 32 33 33 39 31 36 30 36 33 38 32 36 38 34 33 30 38 30 35 61 61 36 66 31 38 66 36 34 35 32 31 63 65 65 66 38 30 36 37 30 30 34 65 36 31 32 63 37 64 36 32 30 39 32 63 62 62 66 32 34 36 31 65GET
/cubeitz-aws/
list-type=2&prefix=Server0002%2FUser000025%2F
host:s3-us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190217T121045Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85547 45 54 0a 2f 63 75 62 65 69 74 7a 2d 61 77 73 2f 0a 6c 69 73 74 2d 74 79 70 65 3d 32 26 70 72 65 66 69 78 3d 53 65 72 76 65 72 30 30 30 32 25 32 46 55 73 65 72 30 30 30 30 32 35 25 32 46 0a 68 6f 73 74 3a 73 33 2d 75 73 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 39 30 32 31 37 54 31 32 31 30 34 35 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35FDCD48F2DCAE53FCi/K8lIayPfOPm+CX0yKTtzEd7b2pcxmoYKpdLyZc+lRpL0yg0swrA4Ogt4tBNLCiFUstc1Hldc4=[/code]

If I search for the file name alone:

QueryParams.Append("prefix=0000029675-0001.CubeiTz") QueryParams.Append("list-type=2")
it says it cannot be found:

<?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>cubeitz-aws</Name><Prefix>0000029675-0001.CubeiTz</Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
How can I get a listing within a sub-folder more than one level down?

Tim_Dietrich · February 17, 2019, 12:48pm

@David Cox:

I think you might need to url-encode the value.

Like this:

QueryParams.Append( "prefix=" + EncodeURLComponent( "Server0002/User000025/" ) )

Let me know if that helps. If not, I’ve got a demo project I can share with you.

Tim

David_Cox · February 17, 2019, 12:54pm

Tim, that fixed it! Running:

QueryParams.Append("list-type=2") QueryParams.Append("max-keys=1") QueryParams.Append("prefix=" + EncodeURLComponent("Server0002/User000025/0000029675-0001.CubeiTz"))
produced the resulting successful key:

<?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>cubeitz-aws</Name><Prefix>Server0002/User000025/0000029675-0001.CubeiTz</Prefix><KeyCount>1</KeyCount><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>Server0002/User000025/0000029675-0001.CubeiTz</Key><LastModified>2018-09-28T12:08:05.000Z</LastModified><ETag>"0ab9a8a6da9fb77aa398d7fe6b218796"</ETag><Size>438272</Size><StorageClass>ONEZONE_IA</StorageClass></Contents></ListBucketResult>

Hal_Gumbert · February 17, 2019, 3:15pm

Does anyone have an S3 Class that has methods for uploading, downloading, and getting folder contents? This is really cool!!

Christian_Schmitz · February 17, 2019, 5:07pm

Did you check the example coming with MBS Plugins?

I could add more if needed.

Hal_Gumbert · February 17, 2019, 5:11pm

Just checked out “CURLS Amazon S3 download.xojo_binary_project” and “CURLS Amazon S3 upload.xojo_binary_project”. Both are nice examples! Thank you, Christian.

What I’m looking for is one class or a module of classes that can init the AWS creds with modules to make doing the common things with S3 easy and reusable.

I’m adding S3 support to Xanadu soon, so I’ll work on a class and release it if no else already has something. I don’t see anything after searching the forums or github.

Hal_Gumbert · February 17, 2019, 10:57pm

Christian, could you add S3 example files for uploading a folderItem and downloading to a folderitem?

Christian_Schmitz · February 18, 2019, 7:52am

first to get my examples to work today, I had to change parameters a bit:

Region = "eu-central-1" domain = "monkeybreadsoftwaretest.s3.eu-central-1.amazonaws.com" FileName = "The-Big-Bang-Theory.jpg" BucketName = "monkeybreadsoftwaretest" AWSAccessKeyId = "xxx" AWSSecretAccessKey = "yyy" Path = "/" + Filename

so I had to change domain, which I didn’t need last time I tried a few months ago.

Christian_Schmitz · February 18, 2019, 7:55am

to stream from AWS to file, I can simply use

dim file as FolderItem = SpecialFolder.Desktop.Child("test.jpg") call d.CreateMTOutputFile(file)

and remove the d.CollectOutputData = true line.

For upload, you need to open file and calculate hash yourself:

[code]dim file as FolderItem = SpecialFolder.Desktop.Child(“test.jpg”)
call d.OpenMTInputFile(file)

HashedPayload = SHA256MBS.HashFile(file)[/code]

Christian_Schmitz · February 18, 2019, 7:57am

to list, we use:

[code]Path = “/”

dim headers() as string
dim queryparams() as string = array(“list-type=2”)
call d.SetupAWS(AWSAccessKeyId, AWSSecretAccessKey, Region, Service, Path, Domain, Verb, HashedPayload, headers, queryparams)[/code]

and than we get an XML like this:

[quote]<?xml version="1.0" encoding="UTF-8"?>
monkeybreadsoftwaretest21000falseThe-Big-Bang-Theory.jpg2019-02-18T07:52:57.000Z"6ed2e03cdd152abae8b6abfd3dc0f5ab"469744STANDARDtest.jpg2019-02-18T07:55:03.000Z"6ed2e03cdd152abae8b6abfd3dc0f5ab"469744STANDARD[/quote]

All questions answered?

Hal_Gumbert · February 18, 2019, 6:23pm

Thank you for the addtional notes Christian! I have another question.

I was able to upload successfully. But when I download, It’s creating a 0 byte file. Oddly, I can see the png file in the Curl.OutputData. I downloaded the file I uploaded via CyberDuck, so I know the file is valid.

I tried adding Curl.CloseMTOutputFile, but that didn’t help. Any ideas? Here’s the code I put together from your example file and applying your notes:

[code]FolderItemDownload( pKey as string, pFolderItem as FolderItem ) string

// Set AWS S3-related values.
Dim Service As String = “s3”
Dim Path As String = “/” + S3BucketName + “/” + pKey
Dim Verb As String = “GET”

// Create a cURL instance.
Dim Curl As New CURLSMBS

// Hash Nothing
Dim HashedPayload As String

// Prepare the cURL for the S3 API call.
Dim CurlSetup As Boolean = Curl.SetupAWS( S3AccessKey, S3SecretKey, S3Region, Service, Path, S3Domain, Verb, HashedPayload )

// Make the S3 API call.
Dim CurlPerform As Integer = Curl.Perform

// Get the response. 200 is Success
Dim HTTPResultCode As String = Curl.GetInfoResponseCode.ToText

// Save to a FolderItem
Dim FileCreated as Boolean
If HTTPResultCode = “200” Then
FileCreated = Curl.CreateMTOutputFile( pFolderItem )
Curl.CloseMTOutputFile
End If

Return HTTPResultCode[/code]

I call that method with the code below after setting S3AccessKey, S3SecretKey, S3Region, S3Domain, and S3BucketName:

Dim theFolderItem as FolderItem = SpecialFolder.Desktop.Child("test.png") Dim theFolderItemUpload as string theFolderItemUpload = theS3.FolderItemDownload( "Contacts/F30BB9DD-DE9A-4F13-91B1-79BA4A776585/PhotoURL/" + theFolderItem.Name, theFolderItem )

Christian_Schmitz · February 18, 2019, 7:02pm

Recent plugin?
You turned CollectOutputData off?