Adding form data to urlConnection POST request

General
1

I am trying to build a POST request to use with urlConnection. I have done quite a few of these, but never before with an attachment. In Postman I put together a request that looks like this:

Content-Type: multipart/form-data; boundary=--------------------------830792918371557810383557
Cookie: JSESSIONID=9684A4D4380F16330FDADCA4E5C56DC3
Content-Length: 207247

----------------------------830792918371557810383557
Content-Disposition: form-data; name=“file”; filename=“2007.13883.txt”
<2007.13883.txt>
----------------------------830792918371557810383557——————————————

This works fine with the api that I’m using, but I can’t work out how to reproduce this in Xojo. Specifically I can’t see where to put the boundary information that encloses the filename. The documentation is sparse on this. I saw an earlier thread from 2020 (URLConnection form data) which in turn refered back to a still earlier example using HTTPSocket. I was lost at that point.

Does anyone have an example of building a POST request that uses urlConnection and uploads a file attachment?

Any guidance on this will be gratefully received.

Thanks,

Ian.

2

I use URLConnection to upload PDF Files.

First i need to convert the binary Data to a BASE64 String:

Var f As FolderItem

f = Folderitem.ShowOpenFileDialog("*.pdf")

If f <> Nil Then // if the user didn't cancel..
  
  If f.Exists Then // if it is a valid file...
    
    Var ReadStream As BinaryStream = BinaryStream.Open(f, False)
    ReadStream.LittleEndian = True
    
    Var s As String = ReadStream.Read(f.Length)
    PDF_BASE64 = EncodeBase64(s, 0)
    
  End If
  
End If

Later then i create a JSON Object. I add the needed Form Data and add the above BASE64 String:

json.Value("pdfForm") = PDF_BASE64
1 Like
3

Thanks for the response. I am really looking for a worked example where the request is built. I’ve tried doing this (authentication blanked):

ucExtractConceptsFromFile.ClearRequestHeaders
// Use basic authentication
ucExtractConceptsFromFile.RequestHeader ("Authorization") = "Basic " + EncodeBase64("***" + ":" + "***")
ucExtractConceptsFromFile.RequestHeader("Content-Type") = "text/plain"
ucExtractConceptsFromFile.RequestHeader("Accept") = "text/plain"

// Put together the multipart package
requestPackage = "--------------------------830792918371557810383557"
requestPackage = requestPackage + EndOfLine.CRLF + "Content-Disposition: form-data; name=" + """file""" + "; filename="+f.NativePath
requestPackage = requestPackage + EndOfLine.CRLF + "<" + f.Name + ">"
requestPackage = requestPackage + EndOfLine.CRLF + "Content-Type: text/plain"
requestPackage = requestPackage + EndOfLine.CRLF + "--------------------------830792918371557810383557--"
// note the extra two dashes at the end of the line above; Postman adds these so I've reproduced that here

ucExtractConceptsFromFile.SetRequestContent(f.NativePath, "multipart/form-data; boundary=" + requestPackage)

requestURI = "https://[myuri]"

// get the response back into taRawTagJson
taRawTagJson.text = ucExtractConceptsFromFile.SendSync("POST", requestURI)

From reading Björn’s posting (URLConnection form data) I thought this might work. But I get a terse message back saying “Could not find acceptable representation”

I’m pretty sure there is a simple fix for this - I’m sending a plain text file and expecting plain text back (though it will end up as JSON). It works with Postman and I’m just trying to reproduce that.
Has anyone done this?

4

Replying to my own post here…
It occurred to me that maybe Xojo is expecting something different from what I am putting in the request. That would explain why Postman works and my Xojo code doesn’t. So I wondered whether there is a way to intercept a POSTed request so that I could see exactly what is being sent to the server.

Is this feasible?

Ian.
—

5

Here’s a Xojo Web 2.0 project that can be used to analyze HTTP requests.
https://timdietrich.me/blog/xojo-web-http-request-analyzer/

I hope you find that to be helpful.

Tim

3 Likes
6

Did you try using this open-source code from Björn?

4 Likes
7

That looks like what I need. I shall dive in and see if it helps.

I’m beginning to think that the problem lies in a mis-match between what I’ve been sending to the server and what it was expecting to receive. Hopefully this class will sort this out.

I thought I’d add a note on installing this, since there may be others in the future who are as woefully dim as me. I couldn’t use File > Open to open it, but I could drag the file from the Finder into the project. I don’t know if this is how it always works because I’ve never built or imported a class before.

Big thanks to Björn, Time and Jeremie for this code.

8

I’ve managed to use the open source code to properly build the api request, but it’s still causing trouble. I initially tested the api using Postman, and this works, bringing back the expected results. So I recorded the api request in Postman, and then wrote my Xojo code to reproduce this. The requests in Postman and Xojo, and the error response, are below.

So I’m confident right up to the point at which I do UrlConnection.SetRequestContent(requestdata, “multipart/form-data”). But it feels to me like somewhere downstream of SetRequestContent the request is being mangled.

The api developers (probably reasonably) say that if Postman works and Xojo doesn’t, the problem is in Xojo. So I need to probe further into this. I had a look at various “request bin” services but so far it’s not very illuminating.

Has anyone played with alternatives to urlConnection in Xojo? I suppose I could use curl, and maybe tcp socket, but ideally I’d like to stick with urlConnection because it’s easy and I’ve always previously had success with it.

Postman request (works)
=======
Authorization: Basic YXBpdXNlcjpwb29scGFydHk=
User-Agent: PostmanRuntime/7.41.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 2be2418a-2b04-4804-9e73-49646b8f25ae
Host: tellura.poolparty.biz
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------------------------457734510648452481990704
Cookie: JSESSIONID=02000A25B063D2E213F5E94F60921587
Content-Length: 1602219
 
----------------------------457734510648452481990704
Content-Disposition: form-data; name="file"; filename="2007.13883.pdf"
<2007.13883.pdf>
----------------------------457734510648452481990704--


Xojo request (fails - see error below)
====

Authorization: Basic YXBpdXNlcjpwb29scGFydHk=
User-Agent: PostmanRuntime/7.41.2
Accept: /
Cache-Control: no-cache
Host: tellura.poolparty.biz
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------------------------20240904095213
Content-Length: 1602002
----------------------------20240904095213
Content-Disposition: form-data; name=“file”; filename=“/Users/ianpiper/Documents/2007.13883.pdf”
<2007.13883.pdf>
----------------------------20240904095213–

The Xojo request gets this error back from the server:

{"responseBase":{"success":false,"status":500,"message":"Failed to parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadException: the request was rejected because no multipart boundary was found","resultType":"text/plain","result":null}}

9

The final boundary is using a long dash instead of a double dash.

Wrong:
----------------------------20240904095213–

Correct:
----------------------------20240904095213–

Edit: Or is the forum transforming double dash to long dash?

10

It is a double dash. Maybe the forum changed it.

——————————————20240904095213
Content-Disposition: form-data; name="file"; filename="/Users/ianpiper/Documents/2007.13883.pdf”
<2007.13883.pdf>
——————————————20240904095213—
11

There is a difference in content length between postman and Xojo although you are posting the same file.

Did you diff both requests with a text editor such as BBEdit?

I use URLConnection in all my apps and never had an issue. But I have never used MultipartFormData yet.

12

Yes, I noticed that. In the Finder the file size is 1,602,002 bytes (1.6 MB on disk), which matches Xojo. I don’t know how Postman measured the file size (which it reports as 217 bytes more). In Postman it doesn’t seem to be possible to load the whole request (i.e. including the file itself) unless I’m missing something. In Xojo, of course, I can’t get a handle on the actual submitted request, as it is hidden within the urlConnection object.
I agree re urlConnection in general. I have been using this (and the earlier version HTTPSocket) for years and have always managed to get it working. This is my first attempt at using multipart form data though.

13

Looks like the error comes from the boundary.
Maybe the server doesn’t like the boundary having a “+” sign in it.

In MultipartFormDataContent.SetURLConnectionMultipartContent method, change the first line to something like this:

'Dim boundary as String = Left(Str(Microseconds) + Str(Microseconds), 24) //Old code
Dim boundary as String = Left(format(Microseconds, "0") + format(Microseconds,"0"), 24) //new code
1 Like
14

One more thing.

In Xojo, you are encoding the entire file path instead of the filename.

15

Just FYI the forum changes two short dashes to an em dash – I use it from time to time

What I’m wondering about is

because the way you use that class, the class instance sets the request content. You are not supposed to be doing so. Check out the example code on the readme.

Dim uc as URLConnection = new URLConnection()
Dim multipartContent as MultipartFormDataContent = new MultipartFormDataContent()

multipartContent.Add("Component","PictureEffects")
multipartContent.Add("Path","Xojo x64")
multipartContent.Add("File",f)

multipartContent.SetURLConnectionMultipartContent(uc)

Dim result as String = uc.SendSync("POST", "http://localhost:5000/filecontroller/api/sendfile")
2 Likes
16

Tim, you are right. I misread the documentation. It is now working the way I would expect it to. I did need to make one change though. In the SetURLConnectionMultipartContent method, the line

uc.RequestHeader("Content-Type") = "multipart/form-data; boundary=--------------------------" + boundary

did not seem to result in a Content-Type request header being set. I needed to assign this to an app shared property in this method and then set the content type header back in the calling method. I have no idea why but it made the difference. I noticed this piece of code in SetURLConnectionMultipartContent:

// Handle Content-Type headers if needed
if mTypes(i) <> "" then
  formText = formText + "Content-Type: " + mTypes(i) + CRLF
  
end

Should I have used the .Add method to set this? Something like:

multipartContent.Add(“Content-Type”, “multipart/form-data”)

Anyway, big thanks to all of you who responded and helped me get my code back on track. Much appreciated.

17

I cannot speak to this experience.

That third parameter for Add is to specify that data being added is something like application/json, not that the whole request is multipart-form/data.

I’m not sure why you’re finding that you need to augment the class. If you could be more specific about the API that’s receiving the request, we can look into where the problem lies.

18

I’ll do my best.

I use a knowledge graph product called PoolParty. This has an extensive API for developers and many of its functions are usefully exposed through this. I’m currently using the Extractor API which allows you to upload a file, whereupon the server analyses the content compared with a taxonomy of concepts. It returns a json package of concepts and their relevance rankings (with some other stuff).
So in this case I am sending a pdf file off to the server. Here is my code:

// This method uses the urlConnection ucExtractConceptsFromFile to run the extract method using a file as source
// The method takes a FolderItem as parameter

dim requestURI As String
dim requestdata As string
dim rawTagString As string
dim rawTagJson As JSONNode

// // Set up the api method
// method is https://tellura.poolparty.biz/extractor/api/extract
// needs to use POST
// and have Content-Type: multipart/form-data

// Clear the request headers 
ucExtractConceptsFromFile.ClearRequestHeaders
// Use basic authentication
ucExtractConceptsFromFile.RequestHeader ("Authorization") = "Basic " + EncodeBase64("********" + ":" + "********")

// assemble the request URI
requestURI = "https://tellura.poolparty.biz/extractor/api/extract?projectId=23b9f288-cbf4-4e97-b318-e2b1cb522957&corpusScoring=corpus:25a645c3-4776-424f-aed7-fbabf08c3a9b&numberOfConcepts=20"

// Assemble the multipart form data. Uses MultipartFormDataContent class downloaded from here: 
// https://github.com/einhugur/MultipartFormDataContent

Dim multipartContent as MultipartFormDataContent = new MultipartFormDataContent()

multipartContent.Add("file", f)
multipartContent.SetURLConnectionMultipartContent(ucExtractConceptsFromFile)

// I need to do this or the Content-Type is not set. App.CTypeSharedString is set in SetURLConnectionMultipartContent 
ucExtractConceptsFromFile.RequestHeader("Content-Type") = App.CTypeSharedString

rawTagString = ucExtractConceptsFromFile.SendSync("POST", requestURI)

// Load and process JSON
rawTagJson = JSONNode.parse(rawTagString)
loadJsonToTagList(rawTagJson)

The only change that I’ve made in SetURLConnectionMultipartContent is here:

uc.RequestHeader("Content-Type") = "multipart/form-data; boundary=--------------------------" + boundary
App.CTypeSharedString = uc.RequestHeader("Content-Type")

I found that initially although the Content-Type was being set at the top of the method, if I later inspect the headers it just said “Content-Type” - i.e., no value. So I created an App level Shared Property and put the header there, then applied it in my method as shown above.
This now works, and retrieves the desired data, but I recognise that it is clumsy. I would welcome a more efficient way of doing this, so by all means suggest one.