How to list AWS files?

  1. 4 months ago

    David C

    Feb 16 Pre-Release Testers, Xojo Pro Derby, ITM

    I am successfully using CURLSMBS.SetupAWS(…) to PUT, GET and DELETE files from my AWS bucket. The only other verb is POST.

    How can I get a list of the files within a designated folder of my AWS bucket?

    I cannot see a LIST-type example in the MBS examples folder. I want to be able to verify that my file was uploaded correctly (sometimes it disappears even though I receive error = 0).

    You could make a GET request with

    /BucketName/?list-type=2

    and the bucket name in the URL.

  2. Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Answer Germany

    You could make a GET request with

    /BucketName/?list-type=2

    and the bucket name in the URL.

  3. David C

    Feb 16 Pre-Release Testers, Xojo Pro Derby, ITM

    It sort of works too well!

    tempCURLSMBSS3.OptionCustomRequest = "GET/" + BucketName + "/&delimiter=/&prefix=mySubFolder1/mySubFolder2/myFileName.txt/?list-type=2"

    This works! But whether I add one subfolder or two (as above), or add a file name, it always returns an XML or EVERY file in EVERY folder. Now, I could parse this for just the file I was looking for, but the list could get massive over time.

    I have tried using many combinations of &prefix= and &delimiter, but it returns the same (every file).

    How may I look inside a sub-folder and for just a single file?

  4. David C

    Feb 16 Pre-Release Testers, Xojo Pro Derby, ITM

    I was using the documentation here:
    https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

  5. Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    there is a space missing after your GET!

  6. Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    and this should really just go in the URL.
    GET is by default. Just put the stuff in the URL.

  7. Tim D

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Richmond, VA

    @David C:

    I ran into a similar problem, and resolved it by passing the query params (as an array of strings) to the SetupAWS method.

    For example:

    Dim AWSAccessKeyId as String = "your-access-key"
    Dim AWSSecretAccessKey as String = "your-secrey"
    Dim Region as String = "us-east-1"
    Dim BucketName as String = "your-bucket-name"
    Dim Service as String = "s3"
    Dim Path as String = "/" + BucketName + "/"
    Dim Domain as String
    Dim Verb as String = "GET"
    Dim HashedPayload as String
    
    Dim QueryParams() As String
    QueryParams.Append("prefix=temp")
    QueryParams.Append("list-type=2")
    
    Dim c As New CURLSMBS
    
    Dim s As Boolean = c.SetupAWS(AWSAccessKeyId, AWSSecretAccessKey, Region, Service, Path, Domain, Verb, HashedPayload, Nil, QueryParams)
    
    Dim p As Integer = c.Perform
    
    Dim OutputData as String = c.OutputData
    Dim HTTPResult as Integer = c.GetInfoResponseCode
    
    TextArea1.Text = HTTPResult.ToText + EndOfLine + OutputData

    Note the "QueryParams" array and its use as a param to the SetupAWS method. In my case, the subfolder is "temp." I have not tested this with multi-level subfolders, but it should work.

    You might want to check the version of the CURLSMBS plugin that you have installed to ensure that it supports the "QueryParams" parameter.

    On a related note: @ChristianSchmitz, the S3 support that you've added to the CURLSMBS plugin is amazing. Thank you!

  8. Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Thanks.

  9. David C

    Feb 17 Pre-Release Testers, Xojo Pro Derby, ITM

    @ChristianSchmitz
    Even with a space after the GET, it still returned every file name

    tempCURLSMBSS3.OptionCustomRequest = "GET /" + BucketName + "/?list-type=2&delimiter=/&prefix=Server0002/User000025/0000029675-0001.CubeiTz"

    @Tim D
    Adding one level of folder within bucket works, in that it shows me a list of all the files in that folder (see below). But…

    QueryParams.Append("prefix=Server0002")

    if I add another folder level down,

    QueryParams.Append("delimiter=/")
    QueryParams.Append("prefix=Server0002/User000025")

    or

    QueryParams.Append("prefix=Server0002/User000025/")

    it gives me the OutputData error:

    <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>MyAccessKey</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
    20190217T121045Z
    20190217/us-west-2/s3/aws4_request
    f2339160638268430805aa6f18f64521ceef8067004e612c7d62092cbbf2461e</StringToSign><SignatureProvided>0d10fa51f0eee815af27d70e3835cded72d8b1a576515f05b7cca2748c841017</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 39 30 32 31 37 54 31 32 31 30 34 35 5a 0a 32 30 31 39 30 32 31 37 2f 75 73 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 66 32 33 33 39 31 36 30 36 33 38 32 36 38 34 33 30 38 30 35 61 61 36 66 31 38 66 36 34 35 32 31 63 65 65 66 38 30 36 37 30 30 34 65 36 31 32 63 37 64 36 32 30 39 32 63 62 62 66 32 34 36 31 65</StringToSignBytes><CanonicalRequest>GET
    /cubeitz-aws/
    list-type=2&amp;prefix=Server0002%2FUser000025%2F
    host:s3-us-west-2.amazonaws.com
    x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    x-amz-date:20190217T121045Z
    
    host;x-amz-content-sha256;x-amz-date
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 63 75 62 65 69 74 7a 2d 61 77 73 2f 0a 6c 69 73 74 2d 74 79 70 65 3d 32 26 70 72 65 66 69 78 3d 53 65 72 76 65 72 30 30 30 32 25 32 46 55 73 65 72 30 30 30 30 32 35 25 32 46 0a 68 6f 73 74 3a 73 33 2d 75 73 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 39 30 32 31 37 54 31 32 31 30 34 35 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>FDCD48F2DCAE53FC</RequestId><HostId>i/K8lIayPfOPm+CX0yKTtzEd7b2pcxmoYKpdLyZc+lRpL0yg0swrA4Ogt4tBNLCiFUstc1Hldc4=</HostId></Error>

    If I search for the file name alone:

    QueryParams.Append("prefix=0000029675-0001.CubeiTz")
    QueryParams.Append("list-type=2")

    it says it cannot be found:

    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>cubeitz-aws</Name><Prefix>0000029675-0001.CubeiTz</Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>

    How can I get a listing within a sub-folder more than one level down?

  10. Tim D

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Richmond, VA
    Edited 4 months ago

    @David C:

    I think you might need to url-encode the value.

    Like this:

    QueryParams.Append( "prefix=" + EncodeURLComponent( "Server0002/User000025/" ) )

    Let me know if that helps. If not, I've got a demo project I can share with you.

    - Tim

  11. David C

    Feb 17 Pre-Release Testers, Xojo Pro Derby, ITM

    Tim, that fixed it! Running:

    QueryParams.Append("list-type=2")
    QueryParams.Append("max-keys=1")
    QueryParams.Append("prefix=" + EncodeURLComponent("Server0002/User000025/0000029675-0001.CubeiTz"))

    produced the resulting successful key:

    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>cubeitz-aws</Name><Prefix>Server0002/User000025/0000029675-0001.CubeiTz</Prefix><KeyCount>1</KeyCount><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>Server0002/User000025/0000029675-0001.CubeiTz</Key><LastModified>2018-09-28T12:08:05.000Z</LastModified><ETag>&quot;0ab9a8a6da9fb77aa398d7fe6b218796&quot;</ETag><Size>438272</Size><StorageClass>ONEZONE_IA</StorageClass></Contents></ListBucketResult>
  12. Hal G

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    Does anyone have an S3 Class that has methods for uploading, downloading, and getting folder contents? This is really cool!!

  13. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Did you check the example coming with MBS Plugins?

    I could add more if needed.

  14. Hal G

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com
    Edited 4 months ago

    @ChristianSchmitz Did you check the example coming with MBS Plugins?

    Just checked out "CURLS Amazon S3 download.xojo_binary_project" and "CURLS Amazon S3 upload.xojo_binary_project". Both are nice examples! Thank you, Christian.

    What I'm looking for is one class or a module of classes that can init the AWS creds with modules to make doing the common things with S3 easy and reusable.

    I'm adding S3 support to Xanadu soon, so I'll work on a class and release it if no else already has something. I don't see anything after searching the forums or github.

  15. Hal G

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    @ChristianSchmitz I could add more if needed.

    Christian, could you add S3 example files for uploading a folderItem and downloading to a folderitem?

  16. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    first to get my examples to work today, I had to change parameters a bit:

    Region = "eu-central-1" 
    domain = "monkeybreadsoftwaretest.s3.eu-central-1.amazonaws.com"
    FileName = "The-Big-Bang-Theory.jpg"
    BucketName = "monkeybreadsoftwaretest"
    AWSAccessKeyId = "xxx"
    AWSSecretAccessKey = "yyy"
    Path = "/" + Filename

    so I had to change domain, which I didn't need last time I tried a few months ago.

  17. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    to stream from AWS to file, I can simply use

    dim file as FolderItem = SpecialFolder.Desktop.Child("test.jpg")
    call d.CreateMTOutputFile(file)

    and remove the d.CollectOutputData = true line.

    For upload, you need to open file and calculate hash yourself:

    dim file as FolderItem = SpecialFolder.Desktop.Child("test.jpg")
    call d.OpenMTInputFile(file)
    
    HashedPayload = SHA256MBS.HashFile(file)
  18. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    to list, we use:

    Path = "/"
    
    dim headers() as string
    dim queryparams() as string = array("list-type=2")
    call d.SetupAWS(AWSAccessKeyId, AWSSecretAccessKey, Region, Service, Path, Domain, Verb, HashedPayload, headers, queryparams)

    and than we get an XML like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>monkeybreadsoftwaretest</Name><Prefix></Prefix><KeyCount>2</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>The-Big-Bang-Theory.jpg</Key><LastModified>2019-02-18T07:52:57.000Z</LastModified><ETag>&quot;6ed2e03cdd152abae8b6abfd3dc0f5ab&quot;</ETag><Size>469744</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>test.jpg</Key><LastModified>2019-02-18T07:55:03.000Z</LastModified><ETag>&quot;6ed2e03cdd152abae8b6abfd3dc0f5ab&quot;</ETag><Size>469744</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

    All questions answered?

  19. Hal G

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    @ChristianSchmitz All questions answered?

    Thank you for the addtional notes Christian! I have another question.

    I was able to upload successfully. :) But when I download, It's creating a 0 byte file. :( Oddly, I can see the png file in the Curl.OutputData. I downloaded the file I uploaded via CyberDuck, so I know the file is valid.

    -image-

    I tried adding Curl.CloseMTOutputFile, but that didn't help. Any ideas? Here's the code I put together from your example file and applying your notes:

    FolderItemDownload( pKey as string, pFolderItem as FolderItem ) string
    
    // Set AWS S3-related values.
    Dim Service       As String = "s3"
    Dim Path          As String = "/" + S3BucketName + "/" + pKey
    Dim Verb          As String = "GET"
    
    // Create a cURL instance.
    Dim Curl As New CURLSMBS
    
    // Hash Nothing
    Dim HashedPayload As String
    
    // Prepare the cURL for the S3 API call.
    Dim CurlSetup As Boolean = Curl.SetupAWS( S3AccessKey, S3SecretKey, S3Region, Service, Path, S3Domain, Verb, HashedPayload )
    
    // Make the S3 API call.
    Dim CurlPerform As Integer = Curl.Perform
    
    // Get the response. 200 is Success
    Dim HTTPResultCode As String = Curl.GetInfoResponseCode.ToText
    
    // Save to a FolderItem
    Dim FileCreated as Boolean
    If HTTPResultCode = "200" Then
       FileCreated = Curl.CreateMTOutputFile( pFolderItem ) 
       Curl.CloseMTOutputFile
    End If
    
    Return HTTPResultCode

    I call that method with the code below after setting S3AccessKey, S3SecretKey, S3Region, S3Domain, and S3BucketName:

    Dim theFolderItem as FolderItem = SpecialFolder.Desktop.Child("test.png")
    Dim theFolderItemUpload as string
    theFolderItemUpload = theS3.FolderItemDownload( "Contacts/F30BB9DD-DE9A-4F13-91B1-79BA4A776585/PhotoURL/" + theFolderItem.Name, theFolderItem )
  20. Christian S

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Recent plugin?
    You turned CollectOutputData off?

  21. Newer ›

or Sign Up to reply!