Stocks abroad open database | Tools & Userscripts | TORN

Stocks abroad open database

​

    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Thread created on 15:39:18 - 22/05/20 (6 years ago)
    |
    Last replied 19:00:54 - 01/07/20 (6 years ago)
    Hey,

    I've been having the "can YATA displays stocks abroad" discussion over and over again with a lot of different players.
    I've always said no because doctorn was there and nobody could get the same user base. Unfortunately it seems to be down for quite a while.

    So I came up with the solution of an open read/write API that can give us a sustainable alternative to doctorn and eventually reach a decent user base.

    All is explained here: https://yata.alwaysdata.net/bazaar/abroad/ (tl;tr I think there is no point in having several private databases. I propose a shared one.)

    I'm waiting for your feedback.

    Note: if I missed the 'writing' part of the arson warehouse API or any other tool that would provide the same feature then we can forget about this and I'm sorry for wasting your time.

    EDIT: First userscript from Pyrit https://raw.githubusercontent.com/TotallyNot/yata-abroad/master/yata_abroad.user.js
    Last edited by Kivou on 20:56:00 - 24/05/20 (6 years ago)

    WesstWorld

    • DeKleineKobini [2114440]
    • Role: Committee
    • Level: 100
    • Posts: 4,597
    • Karma: 6,536
    • Last Action: Now
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 16:43:15 - 22/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    Interesting, I'd recommend to contact Sulsay tho, maybe he can make AWH use this too so we have 1 central place.

    I might even make a small comeback in scripting to work a bit with this. After a month of leaving, I'm already missing it lol.

    e4312a60-1195-482c-a23d-f99e4bbb7ffc-2114440.gif

    • Quarantino [2531034]
    • Role: Civilian
    • Level: 40
    • Posts: 540
    • Karma: 650
    • Last Action: 6 years
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 18:37:15 - 22/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    This is great :)

    I think decentralization is the right path.
    • Manuito [2225097]
    • Role: Committee
    • Level: 83
    • Posts: 1,220
    • Karma: 2,517
    • Last Action: 9 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 19:22:08 - 22/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    I think this is a great idea.

    From my side I think we could also gather information from mobile phone users via Torn PDA. So we might end up having several sources, which is ideal.
    • Mephiles [2087524]
    • Role: Civilian
    • Level: 30
    • Posts: 513
    • Karma: 1,117
    • Last Action: 1 year
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 20:03:21 - 22/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    That's great. I will implement it to my TornTools so it will surely reach lots of "script users" through me.

    I don't think anyone would do this really but what if there are false POSTs to ruin it for others? Could u filter those in any way?

    TornTools

    • Sulsay [2173590]
    • Role: Tester
    • Level: 91
    • Posts: 570
    • Karma: 1,275
    • Last Action: 8 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 04:50:24 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    I like the idea, I'll implement this in AWH! Not sure when quite yet, but I will.

    The fake reports problem can be solved by giving each client (read: app) a secret to include in the posts. I wouldn't bother though until it becomes an issue, if ever.

    The post body of your write request looks off. For "items", you seem to expect an object with item_ids as keys, but you're using array syntax in your example. If up to me, I'd use an array where each entry is an object with keys item_id, quantity and cost.

    I also feel it makes more sense for your API to detect items that do belong in a country but aren't available at this time. That means clients won't need to include those items in their post body (with zero quantity), centralizing this logic in your API instead of requiring each app to re-invent that wheel.

    For the GET request; I think it'd be good to allow no filters at all - and return all item stocks from all countries in that case. You could significantly reduce the size of the response by only returning item_id, country, stock data + cost, instead of including all the item meta data if bandwidth is a concern.

    Arson Warehouse - Foreign stock table and trading software

    • Mephiles [2087524]
    • Role: Civilian
    • Level: 30
    • Posts: 513
    • Karma: 1,117
    • Last Action: 1 year
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 09:01:50 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link

    Sulsay [2173590]

    I like the idea, I'll implement this in AWH! Not sure when quite yet, but I will. The fake reports problem can be solved by giving each client (read: app) a secret to include in the posts. I wouldn't bother though until it becomes an issue, if ever. The post body of your write request looks off. For "items", you seem to expect an object with item_ids as keys, but you're using array syntax in your example. If up to me, I'd use an array where each entry is an object with keys item_id, quantity and cost. I also feel it makes more sense for your API to detect items that do belong in a country but aren't available at this time. That means clients won't need to include those items in their post body (with zero quantity), centralizing this logic in your API instead of requiring each app to re-invent that wheel. For the GET request; I think it'd be good to allow no filters at all - and return all item stocks from all countries in that case. You could significantly reduce the size of the response by only returning item_id, country, stock data + cost, instead of including all the item meta data if bandwidth is a concern.
    I see the syntax error in write request but I think that the syntax of "id: {}" should stay as this is the way Torn does it too for example when fetching the itemlist (it's { id: {}, id: {}}). It's easier to find items like that when you have the ID becuase you don't have to loop through.

    As of the "zero quantity" I would say that just make it optional because even I don't know all the items that you can buy from certain country. So if a user lands in Mexico and sends a POST with a list of item and an item is not included count it as quantity 0.

    I also found a typo in the GET request ("item_maket_value"). I didn't get tho why there was the Country and it's ID added to each item object in the GET request. And from my personal perspective I don't find it necessary to include all the other stuff (item name, type, market value, sell price) because I always use Torn's itemlist anyway when dealing with items and get that information from there. It might just decrease the load and data transfered to remove those. Also isn't buy price and abroad price the same thing as you can't buy those items from shops anyway?

    I'd also discuss the fly_time key as this is probably the default fly time? I don't think that's needed as most players (I assume) use a private airstrip anyway. And if a player uses the private jet or business class ticket then the app should re-calculate the fly time anyway.

    TornTools

    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 13:36:03 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    Thanks for the feedback. I'm glad you are interested in the project.

    Sulsay
    Indeed there is an error in the documentation. It used to be a array syntax with item id inside but in the backend it was easier to make it a dictionary (for the reason mentioned by Mephiles, it's easier to lookup for items. Sometimes torn API return the array structure and it's a real pain). I could accept both if it's easier for the client side.

    The backend does detect 0 items not available at the time the client scrap the data. It sets the quantity to 0 and the cost to the previous cost registered.

    And the GET request is for now just an idea of what can be pulled out. It's true that except weekly tendency everything is directly available with the API. It just make the job easier... And yes a full list will be available also (maybe just the full list).

    Mephiles
    For the 0 quantity: the list is kind of automatically built in the backend by comparing the payload to previous import and looking for missing entries. That would require all clients to scrap the whole page and not just plushies / flowers and drugs for example. If it happens to be a problem for certain clients I can add a option but I don't think it will.

    Yes indeed there is a typo ^^. I'll fix it asap. I added the country within each item to have an homogeneous output and in case for full export or filter based on type it's important to know the country corresponding to the cost/quantity.

    Yes fly_time is the default. It was just to avoid every client to get their own table. Then a simple factor can be applied (airstrip / WLT) to get the actual time. It's not a big deal anyway.


    ---


    So to summarize.

    - The item list of the POST payload could accept two formats:
    1. list (python) or array of objects (json)
    2. dictionary (python) or nested objects (json)
    (if I'm not mistaken about the json way of calling things...). I'm waiting for your point of view: Is the second option really easier for the clients? I will avoid a more complex backend if I can.

    - the GET request is to be worked out. Not sure the filters are very useful. Maybe just spit out the full list. Additional information doesn't seem to fill you with joy so I might just remove it. Better have full list with less info than filters with more info.


    Thanks again for your feeback. Much appreciated.
    Last edited by Kivou on 13:41:29 - 23/05/20 (6 years ago)

    WesstWorld

    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 14:42:48 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    I made a couple of changes in the output payload with different formats, hopefully pleasing everybody.
    The documentation has been updated https://yata.alwaysdata.net/bazaar/abroad/
    Last edited by Kivou on 14:43:04 - 23/05/20 (6 years ago)

    WesstWorld

    • Sulsay [2173590]
    • Role: Tester
    • Level: 91
    • Posts: 570
    • Karma: 1,275
    • Last Action: 8 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 14:55:06 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    Mephiles
    The way TORN handles collections of items in the api shouldn't matter. Don't base your approach on how hard it is to write. Go for the best api, not the easiest. It's a long term initiative and public facing, so it makes sense to go with whatever describes the domain best. In this case, it's a "list of items". Using an object forces the IDs to be strings, which they aren't. They are integers, so using an object is semantically incorrect.

    Making items with zero stock optional in the post body is what I meant to say. With ArsonWarehouse I scan the table abroad and take all items, but ones that are out of stock are simply not included in the html, and therefor not posted. So, counting an item that's excluded from the post body as zero seems to be te most sensible thing to do on the side of YATA. Ofcourse, sending 0 has the same effect but again, should not be required.

    Kivou
    I think it's sensible to expect all apps to post the full stock table to your api. I say this to convince you to build that auto-zero assignment. I can't see why any app would benefit from making partial posts regarding stock data while everything is clearly there on a single page. At least, not if their goal is to provide transparency for all players.

    For the post data choice between array or object, I think we should approach the data from the domain point of view. It's a list of items, not a hash/dictionary (which also isn't primarily meant to iterate over).

    The GET request can be as simple as possible, yeah. Let consumers fetch their own item data if they want to. I feel it doesn't make much sense to include it all in the an endpoint that is meant to get the foreign stock.


    EDIT:
    Oh yeah, I wanted to suggest sticking with one way of doing things. Don't support multiple formats, such as for the country. Either accept an integer representing a country or an exact string. It provides clarity for api consumers and prevents possible confusion. The client feeding the database will know how to properly transform data before POSTing it. Or, well, they should :-)
    Last edited by Sulsay on 15:01:10 - 23/05/20 (6 years ago)

    Arson Warehouse - Foreign stock table and trading software

    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 15:17:31 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link

    Sulsay [2173590]

    Mephiles The way TORN handles collections of items in the api shouldn't matter. Don't base your approach on how hard it is to write. Go for the best api, not the easiest. It's a long term initiative and public facing, so it makes sense to go with whatever describes the domain best. In this case, it's a "list of items". Using an object forces the IDs to be strings, which they aren't. They are integers, so using an object is semantically incorrect. Making items with zero stock optional in the post body is what I meant to say. With ArsonWarehouse I scan the table abroad and take all items, but ones that are out of stock are simply not included in the html, and therefor not posted. So, counting an item that's excluded from the post body as zero seems to be te most sensible thing to do on the side of YATA. Ofcourse, sending 0 has the same effect but again, should not be required. Kivou I think it's sensible to expect all apps to post the full stock table to your api. I say this to convince you to build that auto-zero assignment. I can't see why any app would benefit from making partial posts regarding stock data while everything is clearly there on a single page. At least, not if their goal is to provide transparency for all players. For the post data choice between array or object, I think we should approach the data from the domain point of view. It's a list of items, not a hash/dictionary (which also isn't primarily meant to iterate over). The GET request can be as simple as possible, yeah. Let consumers fetch their own item data if they want to. I feel it doesn't make much sense to include it all in the an endpoint that is meant to get the foreign stock. EDIT: Oh yeah, I wanted to suggest sticking with one way of doing things. Don't support multiple formats, such as for the country. Either accept an integer representing a country or an exact string. It provides clarity for api consumers and prevents possible confusion. The client feeding the database will know how to properly transform data before POSTing it. Or, well, they should :-)
    Quick response: the auto 0 assignment has been implemented on my end since the beginning
    Last edited by Kivou on 15:53:20 - 23/05/20 (6 years ago)

    WesstWorld

    • Mephiles [2087524]
    • Role: Civilian
    • Level: 30
    • Posts: 513
    • Karma: 1,117
    • Last Action: 1 year
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 16:40:09 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link

    Sulsay [2173590]

    Mephiles The way TORN handles collections of items in the api shouldn't matter. Don't base your approach on how hard it is to write. Go for the best api, not the easiest. It's a long term initiative and public facing, so it makes sense to go with whatever describes the domain best. In this case, it's a "list of items". Using an object forces the IDs to be strings, which they aren't. They are integers, so using an object is semantically incorrect. Making items with zero stock optional in the post body is what I meant to say. With ArsonWarehouse I scan the table abroad and take all items, but ones that are out of stock are simply not included in the html, and therefor not posted. So, counting an item that's excluded from the post body as zero seems to be te most sensible thing to do on the side of YATA. Ofcourse, sending 0 has the same effect but again, should not be required. Kivou I think it's sensible to expect all apps to post the full stock table to your api. I say this to convince you to build that auto-zero assignment. I can't see why any app would benefit from making partial posts regarding stock data while everything is clearly there on a single page. At least, not if their goal is to provide transparency for all players. For the post data choice between array or object, I think we should approach the data from the domain point of view. It's a list of items, not a hash/dictionary (which also isn't primarily meant to iterate over). The GET request can be as simple as possible, yeah. Let consumers fetch their own item data if they want to. I feel it doesn't make much sense to include it all in the an endpoint that is meant to get the foreign stock. EDIT: Oh yeah, I wanted to suggest sticking with one way of doing things. Don't support multiple formats, such as for the country. Either accept an integer representing a country or an exact string. It provides clarity for api consumers and prevents possible confusion. The client feeding the database will know how to properly transform data before POSTing it. Or, well, they should :-)
    I understand what you mean by "It's a long term initiative and public facing, so it makes sense to go with whatever describes the domain best. In this case, it's a "list of items". Using an object forces the IDs to be strings, which they aren't. They are integers, so using an object is semantically incorrect."

    But my reasoning on why we should go with object rather than an array is because Torn is using it already. I think it matters because it would be better to match Torn's format. Like you said "Either accept an integer representing a country or an exact string. It provides clarity for api consumers and prevents possible confusion". Matching Torn's format also provides clarity.

    TornTools

    • DeKleineKobini [2114440]
    • Role: Committee
    • Level: 100
    • Posts: 4,597
    • Karma: 6,536
    • Last Action: Now
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 16:42:35 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link

    Sulsay [2173590]

    Mephiles The way TORN handles collections of items in the api shouldn't matter. Don't base your approach on how hard it is to write. Go for the best api, not the easiest. It's a long term initiative and public facing, so it makes sense to go with whatever describes the domain best. In this case, it's a "list of items". Using an object forces the IDs to be strings, which they aren't. They are integers, so using an object is semantically incorrect. Making items with zero stock optional in the post body is what I meant to say. With ArsonWarehouse I scan the table abroad and take all items, but ones that are out of stock are simply not included in the html, and therefor not posted. So, counting an item that's excluded from the post body as zero seems to be te most sensible thing to do on the side of YATA. Ofcourse, sending 0 has the same effect but again, should not be required. Kivou I think it's sensible to expect all apps to post the full stock table to your api. I say this to convince you to build that auto-zero assignment. I can't see why any app would benefit from making partial posts regarding stock data while everything is clearly there on a single page. At least, not if their goal is to provide transparency for all players. For the post data choice between array or object, I think we should approach the data from the domain point of view. It's a list of items, not a hash/dictionary (which also isn't primarily meant to iterate over). The GET request can be as simple as possible, yeah. Let consumers fetch their own item data if they want to. I feel it doesn't make much sense to include it all in the an endpoint that is meant to get the foreign stock. EDIT: Oh yeah, I wanted to suggest sticking with one way of doing things. Don't support multiple formats, such as for the country. Either accept an integer representing a country or an exact string. It provides clarity for api consumers and prevents possible confusion. The client feeding the database will know how to properly transform data before POSTing it. Or, well, they should :-)

    Mephiles [2087524]

    I understand what you mean by "[color=#333333][size=12][i]It's a long term initiative and public facing, so it makes sense to go with whatever describes the domain best. In this case, it's a "list of items". Using an object forces the IDs to be strings, which they aren't. They are integers, so using an object is semantically incorrect.[/i][/size][/color]" But my reasoning on why we should go with object rather than an array is because Torn is using it already. I think it matters because it would be better to match Torn's format. Like you said "[color=#333333][size=12][i]Either accept an integer representing a country or an exact string. It provides clarity for api consumers and prevents possible confusion[/i]". Matching Torn's format also provides clarity.[/size][/color]
    Matching Torn's format also provides clarity.
    No, a good documentation provided clarity, something Torn doesn't give us.

    e4312a60-1195-482c-a23d-f99e4bbb7ffc-2114440.gif

    • Sulsay [2173590]
    • Role: Tester
    • Level: 91
    • Posts: 570
    • Karma: 1,275
    • Last Action: 8 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 17:07:58 - 23/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    The TORN api doesn’t quite set the right example imo. It’s okay to stray from what is already there if the result is an improvement. Especially with this being a new api altogether. I’m good either way! :-)

    Arson Warehouse - Foreign stock table and trading software

    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 00:44:25 - 24/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    Based on your remarks I made the API more simple and the documentation cleaner.
    I moved back to array of objects for the item list in the post request. Even if I don't like this format I like Sulsay's argument.
    Countries are only defined by their 3 first letters (lowercase). I think it's the easiest/safest for both sides.

    WesstWorld

    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 20:55:19 - 24/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    We've got our first userscript thanks to Pyrit: https://raw.githubusercontent.com/TotallyNot/yata-abroad/master/yata_abroad.user.js
    I've made a display on YATA too.

    Now it's all about people using it...

    WesstWorld

    • Seluar [2425756]
    • Role: Civilian
    • Level: 15
    • Posts: 75
    • Karma: 27
    • Last Action: 4 years
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 00:07:57 - 25/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    Nice, installing this. Great job guys :-)
    • GreekPoser [2527893]
    • Role: Civilian
    • Level: 100
    • Posts: 1,344
    • Karma: 1,754
    • Last Action: 6 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 00:09:40 - 25/05/20 (6 years ago)
    Post link copied to clipboard Copy post link
    Thanks for the hard work. I'm coming from the YATA site.

    As I understand, we need to use the script to share data when we are abroad right? I will share it with my faction mates
    • Kivou [2000607]
    • Role: Civilian
    • Level: 100
    • Posts: 1,694
    • Karma: 2,594
    • Last Action: 1 hour
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 00:18:33 - 25/05/20 (6 years ago)
    Post link copied to clipboard Copy post link

    GreekPoser [2527893]

    Thanks for the hard work. I'm coming from the YATA site. As I understand, we need to use the script to share data when we are abroad right? I will share it with my faction mates
    Yes that's the idea, thanks!

    WesstWorld

    • GreekPoser [2527893]
    • Role: Civilian
    • Level: 100
    • Posts: 1,344
    • Karma: 1,754
    • Last Action: 6 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 01:21:54 - 25/05/20 (6 years ago)
    Post link copied to clipboard Copy post link

    GreekPoser [2527893]

    Thanks for the hard work. I'm coming from the YATA site. As I understand, we need to use the script to share data when we are abroad right? I will share it with my faction mates

    Kivou [2000607]

    Yes that's the idea, thanks!
    I think there is a problem with it when taking the numbers on the stock. Right now on the site is the updated number of crocus by me, but it says 4 instead of 4,225 which is the correct amount. If I can correct it after leaving the canadian hospital, I will edit the correction here

    EDIT: the problem the parseInt on line 21 when scraping the data. When you try to do parseInt("4,225"), which is essentially what is doing, you get a 4. You need to do parseInt("4,225".replace(/,/g,'')) Basically removing the commas to get the proper result.

    Replace

    quantity: parseInt(
    item.querySelector(".stck-amount").innerText
    )
    With

    quantity: parseInt(
    item.querySelector(".stck-amount").innerText.replace(/,/g,'')
    )

    to fix
    Last edited by GreekPoser on 01:48:48 - 25/05/20 (6 years ago)
Reply
Thread Title: