MixRack rate limiting on external connections

Forums Forums dLive Forums dLive troubleshooting MixRack rate limiting on external connections

This topic contains 2 replies, has 2 voices, and was last updated by Profile photo of James James 2 years ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #106374
    Profile photo of m-vo
    m-vo
    Participant

    In https://www.allen-heath.com/media/AH-dLive-for-IT-managers.pdf under “1.2 External Control” the following is stated (emphasis by me):

    TCP Rendezvous port 51325 is used for accepting external / non AHNet control messages often sent by DAWs or similar
    3rd party applications. These are formatted as TCP encapsulated MIDI (hex) messages. Flow rates are dependent on client
    configuration, and limited by sliding window protocol from the server
    .

    So this means the MixRack is effectively rate limiting requests send via Midi over TCP, right? For instance SysEx messages for getting fader or send levels/mute status/channel name or color and so on might not come through. From the perspective of code implementing this interface, running into this is very bad: There is no acknowledgement of successfully transferred packages or for instance a (mirrored) header that would allow to identify the source of the request. As you cannot detect when this happens, doing client side rate limiting for sending might be the best option. :-/ And, by the way, the cap seems to be extremely conservative, so this is not a theoretical problem.

    I’ve basically got two questions that might help me getting around this:

    * Are there any specs on how the MixRack rate limiting is working or what’s the actual quota in total/per connection? I also don’t get what’s meant with “dependent on client
    configuration”.
    * Is this also affecting outbound traffic, i.e. the MixRack transmitting send levels for instance?

    Maybe AH can shed some on this.
    Thanks in advance!

    #106391
    Profile photo of m-vo
    m-vo
    Participant

    This unfortunately turns out to be a major issue for our application. šŸ™

    When sending too many MIDI messages (esp. larger ones like setting labels) to the MR, execution of the MIDI commands halts completely and then – after a while – is executed at the normal speed again. No messages seem to be dropped, though. It’s just as if the worker reading the buffer stops working for a time but input still being queued. To be honest, this would be a rather silly implementation of rate limiting. šŸ™‚ So I assume it’s likely a bug.

    What is even worse: MIDI messages, sent by the MR (as a result of the requested commands), could also get slowed down. If you hit the above condition and have sent enough data, it could take several minutes (!) after the outgoing messages have been completely sent. It’s about only 1-2 message(s) per second then. During this time, it is impossible to retrieve status information (e.g. messages to get fader/send level/mute statusā€¦) as they simply seem to get queued afterwards.

    How to reproduce:
    1. Open up two connections, one where you send lots of data (for instance label all Ip channels), one where you listen to the announced changes.
    2. The first messages will be received with little delay, after a while they appear super slow.
    3. When monitoring the same thing in the dLive Director, you’ll notice the execution of the commands halting at some point (you might try sending scene recall messages in between, so this gets better observable).

    Possible solutions:
    * A quick fix could maybe be to loosen the constraints and allow a bigger rate per time unit. I’m hitting these limits even with rate limiting on my side and it’s very hard and cumbersome to work around the limitations. All in all the sent payloads are super small compared to the available network bandwidth.
    * If stalling the MR/Director is an issue, there should IMHO be a smoother throttling at the input side (sliding window, currently feels like its a fixed one). But not sure this is really an issue, as the system seems to be perfectly OK with being hammered by multiple Director instancesā€¦
    * A more sophisticated solution could be to selectively drop queued messages if they would get overridden without side effects by a later one (e.g. for labeling/coloring/muting/setting fader and send levels).
    * Currently, there is no feedback send for commands on the same connection. So, for instance requesting to “mute channel A”, won’t issue a “channel A muted” message. In order to build a stateless system or to monitor states, you’ll always need a second connection (which I assume also eats up the budget). Not sure if this is a feature and changing it would be considered a BC break or it’s a bug.

    Feel free to contact me for more information. I’m very happy to help track this issue down and improve the situation.

    #106471
    Profile photo of James
    James
    Keymaster

    Hi m-vo,

    The statement ā€œflow rates are dependent on client configuration, and limited by sliding window protocol from the serverā€, does not mean we are doing any rate limiting, or throttling. Nor, does it mean any messages will be dropped.
    The connection is TCP, which means that any messages sent will get there, unless the connection is dropped (i.e. there is no route left between the client and server, and all re-tries are exhausted), as is normal with any TCP connection, which is why there is no need for acknowledgements.
    There are always limits to the amount of data that can flow between the end points, these limits depend upon many factors, including the client computer speed and loading, the MixRackā€™s speed and loading, and the network infrastructure that the connection is made over. The bandwidth and performance is high and should not cause any practical problems, indeed the TCP/MIDI is similar in data through-put to the AHNet connection that is used by the dLive Surface & Director etc.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.