API: Messages and events | Bugs & Issues | TORN

API: Messages and events

​

    • Mauk [1494436]
    • Role: Civilian
    • Level: 75
    • Posts: 1,759
    • Karma: 5,335
    • Last Action: 4 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Thread created on 19:41:00 - 28/05/16 (10 years ago)
    |
    Last replied 18:26:16 - 30/05/16 (10 years ago)
    Hello,

    I know this could technically be considered a feature request, but the messages and events API selections can't be efficiently used for a majority of purposes, so much so I'd consider it an API design bug. Please, bear with me.

    There are currently 3 selections we can use to retrieve messages and events:

    Notifications: Returns the number of unread events and messages
    Messages: Returns the list of the last 100 messages
    Events: Returns list of the last 100 events

    There are two main problems with this.

    First, the API responses are not gzipped. Downloading both lists takes whopping 30KB, while a gzipped version would be around 5KB. That's a significant bandwidth overhead. It wouldn't be a problem if we could reliably cache the lists, but this leads us to...

    The second and more important problem: users of the API can't display notifications without requesting the whole list every time. The number returned by the notifications selection isn't enough for us to be able to cache the list. As long as it is different than zero, we have to download the full list. If we are trying to get updates every 10 seconds, à lá City Watch, it's very likely we'll have to request both lists every 10 seconds.

    Why can't we use the "unread messages and events" number from the notifications selection? Let's see:

    T+0min: Request notifications. Unread messages: 1. Request messages. Display notification and cache list.
    T+1min: Request notifications. Unread messages: 1. Great, we don't need to display the notification because we already cached that... Or did we?

    Full picture:

    T+0min: Request notifications. Unread messages: 1. Request messages. Display notification and cache list.
    T+30s: Player reads message.
    T+50s: Player receives a new message.
    T+1min: Request notifications. Unread messages: 1. Great, we don't need to display the notification because we already cached that... Except we didn't.

    As you can see, the number of unread messages is not a good indicator and can't be used for lots of use-cases. For us to develop a dependable notifications system, the notifications API isn't that useful (it fits other use-cases), and we need to request messages,events every single update. That's okay, really, but the problem then becomes bandwidth: at 30KB per request, it could consume GBs of data per month.

    I know there's currently no filtering in the API, but instead of adding a generic filtering system, I'd request at least a "timestamp" parameter for lists. Consider making https://api.torn.com/user/?selections=messages,events×tamp=XXX&key=YYY return a list containing only the messages and events with a timestamp greater than XXX.

    The messages selection should also include the sender's nickname. Yes, we can get it with the user id, but that's an unnecessary round-trip that adds a ton of latency and bandwidth. I'd argue that 99% of use-cases (yes, I took this out of thin air.. please show me examples otherwise) will not use a message title without a sender name, so why force a second request on everybody? Plus, it wouldn't use that much extra bandwidth, especially if gzipped (certainly much less than a whole new request -- and also, gzip excels at repeated content).

    Thank you!
    Last edited by Mauk on 21:26:54 - 28/05/16 (10 years ago)

    MAUK

    • McNeo [864688]
    • Role: Civilian
    • Level: 81
    • Posts: 2,519
    • Karma: 1,955
    • Last Action: 4 years
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 19:48:06 - 28/05/16 (10 years ago)
    Post link copied to clipboard Copy post link

    Mauk [1494436]

    Hello, I know this could technically be considered a feature request, but the messages and events API selections [b]can't[/b] be efficiently used for a majority of purposes, so much so I'd consider it an API design bug. Please, bear with me. There are currently 3 selections we can use to retrieve messages and events: [i]Notifications[/i]: Returns the number of unread events and messages [i]Messages[/i]: Returns the list of the last 100 messages [i]Events[/i]: Returns list of the last 100 events [b]There are two main problems with this. [/b]First, the API responses are not gzipped. Downloading both lists takes whopping 30KB, while a gzipped version would be around 5KB. That's a significant bandwidth overhead. It wouldn't be a problem if we could reliably cache the lists, but this leads us to... The second and more important problem: users of the API [b]can't[/b] display notifications without requesting the whole list [b]every time[/b]. The number returned by the [i]notifications[/i] selection isn't enough for us to be able to cache the list. As long as it is different than zero, we have to download the full list. If we are trying to get updates every 10 seconds, à lá City Watch, it's very likely we'll have to request both lists every 10 seconds. [i]Why can't we use the "unread messages and events" number from the notifications selection?[/i] Let's see: T+0min: Request [i]notifications.[/i] Unread messages: 1. Request [i]messages[/i]. Display notification and cache list. T+1min: Request [i]notifications[/i]. Unread messages: 1. Great, we don't need to display the notification because we already cached that... Or did we? Full picture: T+0min: Request [i]notifications.[/i] Unread messages: [b]1[/b]. Request [i]messages[/i]. Display notification and cache list. T+30s: Player reads message. T+50s: Player receives a [b]new [/b]message. T+1min: Request [i]notifications[/i]. Unread messages: [b]1[/b]. Great, we don't need to display the notification because we already cached that... Except we didn't. As you can see, the number of unread messages is not a good indicator and can't be used for lots of use-cases. For us to develop a dependable notifications system, the [i]notifications[/i] API isn't that useful (it fits other use-cases), and we need to request [i]messages,events[/i] every single update. That's okay, really, but the problem then becomes bandwidth: at 30KB per request, it could consume GBs of data per month. I know there's currently no filtering in the API, but instead of adding a generic filtering system, I'd request at least a "timestamp" parameter for lists. Consider making https://api.torn.com/user/?selections=messages,events×tamp=XXX&key=YYY return a list containing only the messages and events with a timestamp greater than XXX. The [i]messages[/i] selection should also include the sender's nickname. Yes, we can get it with the user id, but that's an unnecessary round-trip that adds a ton of latency and bandwidth. I'd argue that 99% of use-cases (yes, I took this out of thin air.. please show me examples otherwise) will not use a message title without a sender name, so why force a second request on everybody? Plus, it wouldn't use that much extra bandwidth, especially if gzipped (certainly much less than a whole new request -- and also, gzip excels at repeated content). Thank you!
    R+ and I'll second this.

    Torn Tools does exactly as you describe and uses the notifications selection to see if there's a new message and/or event, then makes a second call appropriately if needed - and, as you say, then needs to get ALL the messages/events.

    The messages and events all already have a timestamp parameter anyway, so by allowing us to make API calls with a timestamp argument would allow us to only get the newest messages/events - thus saving bandwidth, a few milliseconds, and potentially even server load.
    • Mauk [1494436]
    • Role: Civilian
    • Level: 75
    • Posts: 1,759
    • Karma: 5,335
    • Last Action: 4 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 20:04:37 - 28/05/16 (10 years ago)
    Post link copied to clipboard Copy post link
    And while we're at it, why not make them arrays instead of objects? While it's possible to loop through keys of an object, they aren't optimized for that like arrays (in every language), and getting the total number of keys is much slower than getting the length of an array.

    Lists are simply not conceptually the same thing as maps. Make them arrays of objects with an id property instead of objects with id as keys.
    Last edited by Mauk on 20:24:52 - 28/05/16 (10 years ago)

    MAUK

    • Marc [24377]
    • Role: Civilian
    • Level: 100
    • Posts: 8,301
    • Karma: 2,537
    • Last Action: 4 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 21:01:56 - 28/05/16 (10 years ago)
    Post link copied to clipboard Copy post link
    I'll pass on to Chedburn for his thoughts.
    • Chedburn [1]
    • Role: Admin
    • Level: 15
    • Posts: 29,253
    • Karma: 74,084
    • Last Action: 7 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 17:14:12 - 29/05/16 (10 years ago)
    Post link copied to clipboard Copy post link
    Why not just new 'unseenmessages' and 'unseenevents'? That only show messages and events that have not been seen on Torn by user?
    • Mauk [1494436]
    • Role: Civilian
    • Level: 75
    • Posts: 1,759
    • Karma: 5,335
    • Last Action: 4 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 17:18:52 - 29/05/16 (10 years ago)
    Post link copied to clipboard Copy post link

    Chedburn [1]

    Why not just new 'unseenmessages' and 'unseenevents'? That only show messages and events that have not been seen on Torn by user?
    Yes, that would be better than what we currently have. I don't think it'd be easier to implement compared to timestamps, though-- and would still require us to keep re-downloading the list of unseen messages/events.

    5 hours of 10 unseen events updated every 10 seconds?
    1.800 requests containing 18.000 events in total using unseen_events.
    1.800 requests containing 10 events in total using timestamps.

    Still much better than the 1800 requests containing 180.000, of course.
    Last edited by Mauk on 17:21:17 - 29/05/16 (10 years ago)

    MAUK

    • Chedburn [1]
    • Role: Admin
    • Level: 15
    • Posts: 29,253
    • Karma: 74,084
    • Last Action: 7 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 16:53:59 - 30/05/16 (10 years ago)
    Post link copied to clipboard Copy post link
    `timestamp` has been added for messages and events. On messages, I've also changed "from" to "ID" and "name" of sender.

    I expect this will be live tomorrow.
    • McNeo [864688]
    • Role: Civilian
    • Level: 81
    • Posts: 2,519
    • Karma: 1,955
    • Last Action: 4 years
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 18:23:11 - 30/05/16 (10 years ago)
    Post link copied to clipboard Copy post link

    Chedburn [1]

    `timestamp` has been added for messages and events. On messages, I've also changed "from" to "ID" and "name" of sender. I expect this will be live tomorrow.
    "timestamp" already exists for messages and events, so where did it get added to and how do we use it? Can you provide an example of an API call that would utilize it?

    Is it something like api.torn.com/user/?selections=messages×tamp=123&key=
    • Mauk [1494436]
    • Role: Civilian
    • Level: 75
    • Posts: 1,759
    • Karma: 5,335
    • Last Action: 4 hours
      • 0
    • Reason:
      Are you sure you want to report this post to staff?
      Cancel
    Posted on 18:26:16 - 30/05/16 (10 years ago)
    Post link copied to clipboard Copy post link

    Chedburn [1]

    `timestamp` has been added for messages and events. On messages, I've also changed "from" to "ID" and "name" of sender. I expect this will be live tomorrow.

    McNeo [864688]

    "timestamp" already exists for messages and events, so where did it get added to and how do we use it? Can you provide an example of an API call that would utilize it? Is it something like api.torn.com/user/?selections=messages×tamp=123&key=
    We were talking about filtering, so, yeah, I believe that's the case -- it's on the request, not response. Pass a timestamp, only messages and events newer than that will be returned.

    MAUK

Reply
Thread Title: