Getting Web Pages.

I’m researching the affects of social media on Tourism. A pretty simple paper to begin, however I’m having issues with getting the html from youtube and wondering if there’s a trick I could use.
For example the webpage:
https://www.youtube.com/user/TourismBC/videos

When I save the html I only get part of the page. I’m guessing it has something to do with ‘loading’ I see in a web-browser and also the ‘load more’ button. Is there any way to get all the html so I can get links to all the videos on that page?

TIA

I know you’re here for development help, but I have an english tip instead (since you mentioned you were writing a paper)

The effects of social media on tourism
-or-
How social media affects tourism

You’d have to follow the loading ajax to get that list, additionally YouTube has an API which I would really recommend using.

English tip: Affect as a noun is an emotional state as contrasted to a cognition. Two control groups watching the same video, one with the social media content one without.

But thanks for your tip on ajax and the YouTube API both of which are far beyond my skills.

I must say I’m much more interested in understanding the use of affects in the original post than the actual question. I also would tend to agree with Tim, but I do see your argument. The sticking point for me is using it in the emotional state sense I would expect it to apply to something that can display an emotional affect. Maybe a person, character or animal even. After looking at a couple of dictionary sites I can see why this word is a tricky one.

It is the people in the 2 control groups. What different emotional reactions will be seen because of the social content shown. Testing to see if the group of people that get to see social ‘ratings’ (2,000 likes and 2 dislikes) react differently than the group that don’t get that information from the video.

Well, sounds interesting. Best of luck with it, and please post about the results. Probably in the off topic section though. I think I’ve nudged this thread off course. :slight_smile:

So after spending a day reading about YouTube v3 API, I have a question.
It seems like the browser gives me the information I’m looking for.
If I enter
https://www.googleapis.com/youtube/v3/playlists?part=snippet&channelId=UCpS9F8iOFksvPuYByurAvBQ&key={My Key}

I get easy to parse text on the webpage. If I use an HTTPSecureSocket and automate to process will this text be in the PageReceived or is this not the right class for me to be looking at.

TIA.

Here is a simple way of dealing with APIs using a synchronous request. The code will pause and wait for the answer for up to 20 seconds. I use this approach with several APIs from WordPress to HubSpot to other custom implementations - works great all day long. If you want an a synchronous request then you need to subclass the HTTPSecureSocket and put your code in the PageReceived event. That event includes a “Content” variable that will have your page data in it.

Dim s as new HTTPSecureSocket
dim responseJSON as new JSONItem

Dim response As String = s.Get("https://www.googleapis.com/youtube/v3/playlists?part=snippet&channelId=UCpS9F8iOFksvPuYByurAvBQ&key=<KEY>",20)

try
  responseJSON.Load(response)
 ' do something here
  
catch
  ' Bad JSON error handling here
  
end try

John Joyce Thanks for the tip. Is this correct thinking?

I will use 3 requests each dependent on the one prior

Essentially

  • Get Upload_List_ID
  • Get Upload List (using Upload_List_ID) <-- possibly 5 times as Max Pull of 50 videos per
  • Get Video & details (using Video ID from Upload List) <-- Asynchronous

I’m guessing your above code can just go into myMethod of a thread, then I can add all the VideoID’s to an array variable of the thread and get them from a HTTPSecureSocket variable of the thread?

I am not familiar with the specifics of the YouTube API - that said, the concept you have laid out sounds good to me. I do something very similar for a different API. I have a thread that calls different GET and UPDATE type methods that do the API work. You can of course pull in data from a property like an array.

Ok, I gave it whirl and I guess it worked because it stopped at the Break in Try and not catch. Now what? I looked at the JSON item and not a clue what to do with it.
response =
{
“kind”: “youtube#playlistListResponse”,
“etag”: “"m2yskBQFythfE4irbTIeOgYYfBU/VtCytYfz4m6y0zLztEhtkR4kVFU"”,
“nextPageToken”: “CAUQAA”,
“pageInfo”: {
“totalResults”: 34,
“resultsPerPage”: 5
},
“items”: [
{
“kind”: “youtube#playlist”,
“etag”: “"m2yskBQFythfE4irbTIeOgYYfBU/5UokzGsOn5eNymXBAlqFiik58JM"”,
“id”: “PL-iRZdaCoBuSRUcmoeJWoyyKdXNb6NH9k”,
“snippet”: {
“publishedAt”: “2017-01-26T22:46:00.000Z”,
“channelId”: “UCpS9F8iOFksvPuYByurAvBQ”,
“title”: “Explore BC With the Locals”,
“description”: “British Columbians share why they love where they live, from Vancouver Island to the Rocky Mountains, the Okanagan Valley to Haida Gwaii.”,
—Deleted thumbnail links to make this shorter —
“localized”: {
“title”: “Explore BC With the Locals”,
“description”: “British Columbians share why they love where they live, from Vancouver Island to the Rocky Mountains, the Okanagan Valley to Haida Gwaii.”
}
}
},
{
“kind”: “youtube#playlist”,

With 5 “kind”: “youtube#playlist”, one each of the 5 videos.
So lets say I needed
“nextPageToken”:
“id”: (there are 5 of these, 1 per the 5 videos listed.) What do I code?
TIA

You have to parse the JSON. Check the documentation for JSONItem here.

This is a playlist response item which is an array of playlists.

So you would get the first item (which is the array), then iterate through it to get each playlist. Maybe something like this (I am just making this up - not tested at all).


Dim s as new HTTPSecureSocket
dim responseJSON as new JSONItem
dim myIDs() as variant

Dim response As String = s.Get("https://www.googleapis.com/youtube/v3/playlists?part=snippet&channelId=UCpS9F8iOFksvPuYByurAvBQ&key=<KEY>",20)

try
  responseJSON.Load(response)
  dim responseItems as new JSONItem
  responseItems = responseJSON.Value("items")
  for i as integer = 0 to responseItems.count-1
    myIDs.append(responseItems.child(i).value("id"))
  next
      
catch
  ' Bad JSON error handling here
  
end try

… in the above example, nextPageToken should found at responseJSON.Value(“nextPageToken”) - sorry, missed that in initial read.

New framework makes this much easier. http://developer.xojo.com/xojo-data

Assuming response is a text variable:

[code]dim d as Xojo.Core.Dictionary
dim a() as auto
dim nextpagetoken, ids() as text

d = xojo.data.ParseJSON(response)

if d.HasKey(“nextPageToken”) then
nextpagetoken = d.Value(“nextPageToken”)
end if

if d.HasKey(“items”) then
a = d.value(“items”)
for each d in a
if d.HasKey(“kind”) and d.Value(“kind”) = “youtube#playlist” and d.HasKey(“id”) then
ids.Append(d.value(“id”))
end if
next
end if[/code]

nextpagetoken will hold the nextPageToken value and ids will be an array of the 5 ids.