How to list AWS files?

  1. ‹ Older
  2. 5 weeks ago

    Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    there is a space missing after your GET!

  3. Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    and this should really just go in the URL.
    GET is by default. Just put the stuff in the URL.

  4. Tim D

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Richmond, VA

    @David C:

    I ran into a similar problem, and resolved it by passing the query params (as an array of strings) to the SetupAWS method.

    For example:

    Dim AWSAccessKeyId as String = "your-access-key"
    Dim AWSSecretAccessKey as String = "your-secrey"
    Dim Region as String = "us-east-1"
    Dim BucketName as String = "your-bucket-name"
    Dim Service as String = "s3"
    Dim Path as String = "/" + BucketName + "/"
    Dim Domain as String
    Dim Verb as String = "GET"
    Dim HashedPayload as String
    
    Dim QueryParams() As String
    QueryParams.Append("prefix=temp")
    QueryParams.Append("list-type=2")
    
    Dim c As New CURLSMBS
    
    Dim s As Boolean = c.SetupAWS(AWSAccessKeyId, AWSSecretAccessKey, Region, Service, Path, Domain, Verb, HashedPayload, Nil, QueryParams)
    
    Dim p As Integer = c.Perform
    
    Dim OutputData as String = c.OutputData
    Dim HTTPResult as Integer = c.GetInfoResponseCode
    
    TextArea1.Text = HTTPResult.ToText + EndOfLine + OutputData

    Note the "QueryParams" array and its use as a param to the SetupAWS method. In my case, the subfolder is "temp." I have not tested this with multi-level subfolders, but it should work.

    You might want to check the version of the CURLSMBS plugin that you have installed to ensure that it supports the "QueryParams" parameter.

    On a related note: @ChristianSchmitz, the S3 support that you've added to the CURLSMBS plugin is amazing. Thank you!

  5. Christian S

    Feb 16 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Thanks.

  6. David C

    Feb 17 Pre-Release Testers, Xojo Pro Derby, ITM

    @ChristianSchmitz
    Even with a space after the GET, it still returned every file name

    tempCURLSMBSS3.OptionCustomRequest = "GET /" + BucketName + "/?list-type=2&delimiter=/&prefix=Server0002/User000025/0000029675-0001.CubeiTz"

    @Tim D
    Adding one level of folder within bucket works, in that it shows me a list of all the files in that folder (see below). But…

    QueryParams.Append("prefix=Server0002")

    if I add another folder level down,

    QueryParams.Append("delimiter=/")
    QueryParams.Append("prefix=Server0002/User000025")

    or

    QueryParams.Append("prefix=Server0002/User000025/")

    it gives me the OutputData error:

    <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>MyAccessKey</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
    20190217T121045Z
    20190217/us-west-2/s3/aws4_request
    f2339160638268430805aa6f18f64521ceef8067004e612c7d62092cbbf2461e</StringToSign><SignatureProvided>0d10fa51f0eee815af27d70e3835cded72d8b1a576515f05b7cca2748c841017</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 39 30 32 31 37 54 31 32 31 30 34 35 5a 0a 32 30 31 39 30 32 31 37 2f 75 73 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 66 32 33 33 39 31 36 30 36 33 38 32 36 38 34 33 30 38 30 35 61 61 36 66 31 38 66 36 34 35 32 31 63 65 65 66 38 30 36 37 30 30 34 65 36 31 32 63 37 64 36 32 30 39 32 63 62 62 66 32 34 36 31 65</StringToSignBytes><CanonicalRequest>GET
    /cubeitz-aws/
    list-type=2&amp;prefix=Server0002%2FUser000025%2F
    host:s3-us-west-2.amazonaws.com
    x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    x-amz-date:20190217T121045Z
    
    host;x-amz-content-sha256;x-amz-date
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 63 75 62 65 69 74 7a 2d 61 77 73 2f 0a 6c 69 73 74 2d 74 79 70 65 3d 32 26 70 72 65 66 69 78 3d 53 65 72 76 65 72 30 30 30 32 25 32 46 55 73 65 72 30 30 30 30 32 35 25 32 46 0a 68 6f 73 74 3a 73 33 2d 75 73 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 39 30 32 31 37 54 31 32 31 30 34 35 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>FDCD48F2DCAE53FC</RequestId><HostId>i/K8lIayPfOPm+CX0yKTtzEd7b2pcxmoYKpdLyZc+lRpL0yg0swrA4Ogt4tBNLCiFUstc1Hldc4=</HostId></Error>

    If I search for the file name alone:

    QueryParams.Append("prefix=0000029675-0001.CubeiTz")
    QueryParams.Append("list-type=2")

    it says it cannot be found:

    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>cubeitz-aws</Name><Prefix>0000029675-0001.CubeiTz</Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>

    How can I get a listing within a sub-folder more than one level down?

  7. Tim D

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Richmond, VA
    Edited 5 weeks ago

    @David C:

    I think you might need to url-encode the value.

    Like this:

    QueryParams.Append( "prefix=" + EncodeURLComponent( "Server0002/User000025/" ) )

    Let me know if that helps. If not, I've got a demo project I can share with you.

    - Tim

  8. David C

    Feb 17 Pre-Release Testers, Xojo Pro Derby, ITM

    Tim, that fixed it! Running:

    QueryParams.Append("list-type=2")
    QueryParams.Append("max-keys=1")
    QueryParams.Append("prefix=" + EncodeURLComponent("Server0002/User000025/0000029675-0001.CubeiTz"))

    produced the resulting successful key:

    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>cubeitz-aws</Name><Prefix>Server0002/User000025/0000029675-0001.CubeiTz</Prefix><KeyCount>1</KeyCount><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>Server0002/User000025/0000029675-0001.CubeiTz</Key><LastModified>2018-09-28T12:08:05.000Z</LastModified><ETag>&quot;0ab9a8a6da9fb77aa398d7fe6b218796&quot;</ETag><Size>438272</Size><StorageClass>ONEZONE_IA</StorageClass></Contents></ListBucketResult>
  9. Hal G

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    Does anyone have an S3 Class that has methods for uploading, downloading, and getting folder contents? This is really cool!!

  10. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Did you check the example coming with MBS Plugins?

    I could add more if needed.

  11. Hal G

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com
    Edited 5 weeks ago

    @ChristianSchmitz Did you check the example coming with MBS Plugins?

    Just checked out "CURLS Amazon S3 download.xojo_binary_project" and "CURLS Amazon S3 upload.xojo_binary_project". Both are nice examples! Thank you, Christian.

    What I'm looking for is one class or a module of classes that can init the AWS creds with modules to make doing the common things with S3 easy and reusable.

    I'm adding S3 support to Xanadu soon, so I'll work on a class and release it if no else already has something. I don't see anything after searching the forums or github.

  12. Hal G

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    @ChristianSchmitz I could add more if needed.

    Christian, could you add S3 example files for uploading a folderItem and downloading to a folderitem?

  13. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    first to get my examples to work today, I had to change parameters a bit:

    Region = "eu-central-1" 
    domain = "monkeybreadsoftwaretest.s3.eu-central-1.amazonaws.com"
    FileName = "The-Big-Bang-Theory.jpg"
    BucketName = "monkeybreadsoftwaretest"
    AWSAccessKeyId = "xxx"
    AWSSecretAccessKey = "yyy"
    Path = "/" + Filename

    so I had to change domain, which I didn't need last time I tried a few months ago.

  14. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    to stream from AWS to file, I can simply use

    dim file as FolderItem = SpecialFolder.Desktop.Child("test.jpg")
    call d.CreateMTOutputFile(file)

    and remove the d.CollectOutputData = true line.

    For upload, you need to open file and calculate hash yourself:

    dim file as FolderItem = SpecialFolder.Desktop.Child("test.jpg")
    call d.OpenMTInputFile(file)
    
    HashedPayload = SHA256MBS.HashFile(file)
  15. Christian S

    Feb 17 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    to list, we use:

    Path = "/"
    
    dim headers() as string
    dim queryparams() as string = array("list-type=2")
    call d.SetupAWS(AWSAccessKeyId, AWSSecretAccessKey, Region, Service, Path, Domain, Verb, HashedPayload, headers, queryparams)

    and than we get an XML like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>monkeybreadsoftwaretest</Name><Prefix></Prefix><KeyCount>2</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>The-Big-Bang-Theory.jpg</Key><LastModified>2019-02-18T07:52:57.000Z</LastModified><ETag>&quot;6ed2e03cdd152abae8b6abfd3dc0f5ab&quot;</ETag><Size>469744</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>test.jpg</Key><LastModified>2019-02-18T07:55:03.000Z</LastModified><ETag>&quot;6ed2e03cdd152abae8b6abfd3dc0f5ab&quot;</ETag><Size>469744</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

    All questions answered?

  16. Hal G

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    @ChristianSchmitz All questions answered?

    Thank you for the addtional notes Christian! I have another question.

    I was able to upload successfully. :) But when I download, It's creating a 0 byte file. :( Oddly, I can see the png file in the Curl.OutputData. I downloaded the file I uploaded via CyberDuck, so I know the file is valid.

    -image-

    I tried adding Curl.CloseMTOutputFile, but that didn't help. Any ideas? Here's the code I put together from your example file and applying your notes:

    FolderItemDownload( pKey as string, pFolderItem as FolderItem ) string
    
    // Set AWS S3-related values.
    Dim Service       As String = "s3"
    Dim Path          As String = "/" + S3BucketName + "/" + pKey
    Dim Verb          As String = "GET"
    
    // Create a cURL instance.
    Dim Curl As New CURLSMBS
    
    // Hash Nothing
    Dim HashedPayload As String
    
    // Prepare the cURL for the S3 API call.
    Dim CurlSetup As Boolean = Curl.SetupAWS( S3AccessKey, S3SecretKey, S3Region, Service, Path, S3Domain, Verb, HashedPayload )
    
    // Make the S3 API call.
    Dim CurlPerform As Integer = Curl.Perform
    
    // Get the response. 200 is Success
    Dim HTTPResultCode As String = Curl.GetInfoResponseCode.ToText
    
    // Save to a FolderItem
    Dim FileCreated as Boolean
    If HTTPResultCode = "200" Then
       FileCreated = Curl.CreateMTOutputFile( pFolderItem ) 
       Curl.CloseMTOutputFile
    End If
    
    Return HTTPResultCode

    I call that method with the code below after setting S3AccessKey, S3SecretKey, S3Region, S3Domain, and S3BucketName:

    Dim theFolderItem as FolderItem = SpecialFolder.Desktop.Child("test.png")
    Dim theFolderItemUpload as string
    theFolderItemUpload = theS3.FolderItemDownload( "Contacts/F30BB9DD-DE9A-4F13-91B1-79BA4A776585/PhotoURL/" + theFolderItem.Name, theFolderItem )
  17. Christian S

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Recent plugin?
    You turned CollectOutputData off?

  18. Hal G

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com
    Edited 5 weeks ago

    @ChristianSchmitz Recent plugin?
    You turned CollectOutputData off?

    I was using MBS 18.5 and now MBS 19.
    I tried Curl.CollectOutputData = false and removing that line entirely.

    CreateMTOutputFile( pFolderItem ) is creating 0 byte files for me.

    Here's the S3 Class so far with a calling example in the 'Example' Notes. ListGetXML, ListGetDict, and FolderItemUpload, StringUpload, StringDownload work.
    http://campsoftware.com/files/LongTerm/Xojo/S3.xojo_binary_code.zip

    Thanks to Christian and Tim Dietrich for the sample code!

  19. Christian S

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers Germany

    Please call CreateMTOutputFile before Perform.
    This is to stream to file, so you need to open before Perform call. It is closed automatically when Perform is done.

  20. Hal G

    Feb 18 Pre-Release Testers, Xojo Pro, XDC Speakers CampSoftware.com

    @ChristianSchmitz Please call CreateMTOutputFile before Perform

    You rock. That was the issue and make sense that it would need to be defined first. Thank you!

    I updated the S3 Class download file. I hope it's helpful to others! :)

or Sign Up to reply!