Downloading HTML - Fail

  1. 2 years ago

    Can anyone tell me why is this code failing ?

    I'm trying to download the HTML from www.bna.com.ar with no luck...

     Dim HTTPSocket1 as new HTTPSocket
      Dim pageData As String
      pageData = HTTPSocket1.Get("http://www.bna.com.ar/", 30)

    thanks

    With this line added, it works... :-)

    HTTPSocket1.SetRequestHeader( "user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14" )

  2. With this line added, it works... :-)

    HTTPSocket1.SetRequestHeader( "user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14" )

  3. Tim P

    23 Dec 2017 Pre-Release Testers, Xojo Pro Rochester, NY

    Seems like they're trying to prevent unauthorized use and scraping of their website.
    If there is no API available, you should not be doing what you're doing.

  4. I understand. I just need to check once a day the official currency exchange rate. I guess there's not harm in that....

  5. Tim P

    24 Dec 2017 Pre-Release Testers, Xojo Pro Rochester, NY

    I wonder if this will help, http://fixer.io
    It's a special API specifically for currency conversions. It seems to be very easy to use as the data comes in as JSON. Be sure to follow the github links at the bottom for more generic implementation help :)

  6. Alberto D

    24 Dec 2017 Pre-Release Testers, Xojo Pro

    I understand why Roman need to access that page, that's the official exchange rate for the Country. Usually other currency conversions sites use market value and not the official exchange rate set for a specific date.

    I also need to access the official website, in Mexico, for the daily currency exchange with the dollar. Other official exchange rates are set every month (my guess is they calculate an average for the previous month) but with the dollar the rate is set daily (well only working weekdays, no weekend or holidays).

    So for me, and I guess for Roman too, is going to a specific site everyday and manually get the value, or set your program to do it for you. I'm thinking of setting a webpage with the information pulled every day, that way my program only check the info once a day, update my website and then use my website if more tests/checks are needed each day (or more users). The goal is to ask for the information once, then use it more times if needed without going to the official site again.

  7. Tim P

    24 Dec 2017 Pre-Release Testers, Xojo Pro Rochester, NY
    Edited 2 years ago

    I haven't searched the website, but perhaps they offer an API for gathering the exchange rate officially.

    One of the things you can run into by scraping websites without authorization or permission is that the website can change without notice. Then your app goes down, and your customers are mad at you because your app isn't working.

    The thing that bothers me is that everyone here is so protective of their apps (just go look for the DRM threads) yet people seem to have no issue stealing data. It's incredibly unethical and gets me worked up real fast.

  8. @Roman V I understand. I just need to check once a day the official currency exchange rate. I guess there's not harm in that....

    Tell me you're not serious. Is this really your thinking? Be a professional and do it the right way with an approved API

    Use either fixer.io or OpenExchange, openexchangerates.org. Most API's give you the first 1000 requests for free so you'll have no issues with once a day.

  9. Alberto D

    24 Dec 2017 Pre-Release Testers, Xojo Pro

    Sorry this is a big issue and you get mad about it.

    At least for Mexico's official site, there is no API and I don't think it will ever be. The information is public and it is needed to calculate every transaction done in foreign currency, this to ultimate pay the correct taxes.

    fixer.io say they use information from European Central Bank, so I can't use that. I guess openexchangerates.org doesn't use the official daily exchange rate that Banco de Mexico calculates and provide to the official publication from the government.

    I'm a newbie, so I really don't know what is the correct way to deal with this. I only see the following options:
    a) go to the official website, read the html with the program an extract the exchange rate
    b) ask the client to visit the site, and manually enter the value into the program updating a database (the program will only be used by 1 client)
    c) create a website with the information needed for the client, so I go to the official site, see the exchange rate, manually update my website and use that with the client's program (create own site/API)

    Thank you for your input.

  10. Tim P

    24 Dec 2017 Pre-Release Testers, Xojo Pro Rochester, NY

    I looked into it a little to try to help you out, and I found this statement on the Banco de Mexico website in reference to the exchange rates: "Banco de México da a conocer estos tipos de cambio con fines estrictamente informativos y sin carácter oficial." (or "3 / Banco de México discloses these exchange rates for strictly informative purposes and without official character." per Google). That to me sounds like the Banco de Mexico (banxico.org.mx ) website wouldn't be any more usable than any other API source.

    The Banco de Mexico website states that they get their data from the International Monetary Fund, and it seems the data on the IMF website (https://imf.org/external/np/fin/data/param_rms_mth.aspx ) does indeed match. I also found on the IMF website what they call the "Exchange Rate Query Tool" which allows you to make queries to their database. In addition they also specifically outline their terms for usage of the data (https://www.imf.org/external/terms.htm ) stating that you cannot use it for commercial purposes without authorization.

    Considering that's where it looks like most of this exchange rate data comes from, you should be requesting permission from them to use the data. They may even offer proper channels to do so (an API or officially delivered data).

    We're developers, we code and make things. We cannot simply take data or information from wherever we so happen to find it and use it without permission just because we can. Getting into client requests is a grey area and that's what lawyers are for, however I personally wouldn't consider it any more appropriate to use the data at that point.

    Best of luck to you with your exchange rates, and if there's anything more I can help with please let me know!

  11. Alberto D

    25 Dec 2017 Pre-Release Testers, Xojo Pro

    Thank you Tim, I always see that you go above and beyond what other people are willing to do to help out. I will gladly buy you a drink/coffee sometime.

    Maybe the stealing data remark took me by surprise. When I found Xojo the advertising said it is beginner-friendly and even some documents that I read said it could be used by people with little or no programming background. It really was an honest misinformation that it is different to scrap a webpage once a day vs visiting that page and manually copy the daily value. In my mind it seemed like the same thing only automatic. Like building a robot/drone to pick up the free newsletter from the corner store, the one that say free-pick one. I didn't understand the difference between going myself to pick one up and sending the robot/drone to pick one up. Sorry I'm not a developer and I guess I lack some (much) training.

    It will be nice if Xojo docs include a warning saying something like what you said: "be aware that you cannot simply take data or information from wherever website and use it without permission, check if there is an official API, the requirements or ask permission to use their site"

    Going back to the Mexican official dollar/peso exchange rate: Banxico provide the information about noon every working day that will be published the next day on the Diario Oficial de la Federación (Federation Official Newspaper). The information is not official until published. The exchange rate published on dof.gob.mx it should be the official one, and I say it should because they sometimes make mistakes, you need to confirm what was published on PDF or paper (at least once the website had the same date with 2 different values and one missing date, as you guess the 2 values were for different dates, one day and the next).

    Thanks to you I searched and found out that there is an API in beta from Banxico . The output I can get is html, json or xml. I already have my token, I understand the limits and I'll do some tests to learn how to program that in Xojo.

  12. Alright guys, thanks everybody for their feedback... Please consider not everyone here is an absolute pro. There are things many of us don't know.

    I don't mean to enter an ethical argument here. Maybe it is wrong (if it is I didn't know it, and I don't understand WHY it is wrong, since it's public information and the BNA it's a public institution supported by the people (me as one of them) which has as one of its responsibilities to PROVIDE that info), but I still think I am not technically causing harm checking public data once a day on their site by a not publicly distributed app.

    I also understand, that when scraping data, if/when the site changes the mechanism will stop working.

    Again, let's not get deep into ethical issues here, please.. as it is not the subject of the topic, it is not the problem I am facing here, and basically I am NOT stating that scraping data it is the right thing to do...

    Believe me, If I had found a free API I would certainly have used it. But I did not. The rates http://fixer.io/ offer are simply not good as they are not official and worse, they don't even have ARS as a currency, listed.

    The same goes for https://openexchangerates.org. It returns values that are still not official (although are very similar).

    At last, there's always the option to force users to go into BNA's website and copy/paste the three numbers manually....

    Of course your thoughts and ideas are always welcome...

    Roman

  13. Peter J

    is not verified 26 Dec 2017 KwaZulu-Natal, South Africa

    @Tim P The thing that bothers me is that everyone here is so protective of their apps (just go look for the DRM threads) yet people seem to have no issue stealing data. It's incredibly unethical and gets me worked up real fast.

    This is interesting. I think one must look to the law, rather than what one regards as ethical. Argentina is a signatory to the Berne convention, so the general conditions would apply, as set out in their copyright laws. As I understand things ( IANAL ) copying for personal use is OK, but re-publishing or on-selling information would be a violation or an offence. Local laws do apply though.

    The interesting thing I did read is that USA federal government publications - presumably including websites - are in the public domain within the USA ( some exceptions ). Maybe Argentina has a provision like that.

  14. 2 weeks ago

    Mariano P

    Jan 11 Pre-Release Testers, Xojo Pro Argentina

    @Roman V ... probably you had already solved this problem but in the following github repository you will find the XOJO code and the instructions you will need in order to use the APIs of the Central Bank of the Argentinien Republic (BCRA) to obtain the dolar rate, and also a lot of other historic and actual parameters of the Argentinien economy.

    https://github.com/bMD-biomedical/BCRA-APis

  15. Cool. I'll give it a look.
    Thanks

or Sign Up to reply!