MQTT Client app - Beta testing

[Daniel]

We are inviting everyone to join beta testing for our new MQTT client app.

https://dl.openrb.com/pkg/mqtt-client-20240926.ipk

The app will run on all controllers, none LM too.

The first step is to establish communication with the Broker. Under the gear icon the MQTT configuration can be set. After saving the dot in the top right corner will highlight if communication was established or not. Green means connected, red the opposite.

There are also connection logs and help with a link to this thread.

---

Add topic

Common options

• Direction - defines the communication direction, it can be in Both, In (From MQTT) and Out (To MQTT)
  
• Topic name is what the name defines.
  
• Mapped object is the object from which the value will be read/written. This option is not needed if Direction is In (From MQTT) as the payload can contain several values.
  

Out (to MQTT) options

• Publish QoS - Quality of Service level when publishing data to MQTT. Use QoS 0 when using a local broker on the same LM. More info here.
  
• Retain message tells broker to keep the last value. Any new client subscribing to the topic will receive the last value, otherwise will need to wait for a change. More info here.
  
• Publish current value when connected to broker - when enabled the current object value is published when a connection to the broker is established.
  

In (from MQTT) options

• Publish QoS - Quality of Service level when subscribing to data to MQTT. Use QoS 0 when using a local broker on the same LM.
  
• Send delta - minimum value difference to trigger group
  
• Throttle time (seconds) - set a minimum time between MQTT messages. A timer is started when a message is received. Any new messages are ignored until the time expires.
  
• Write to bus - defines where to write values to KNX/TP or not.
  
• JSON value path - extracts value from a JSON payload, see example below.
  

---

JSON value path example

MQTT payload is the following JSON, battery level % needs to be obtained.

Code:

{
  "id": "0",
  "battery": {
    "V": 2.99,
    "percent": 93
  }
}

The root object contains an object named battery which has a percent value. The resulting JSON value path is battery.percent

Arrays are also supported with indexes starting from 1 because the JSON payload is converted to Lua.

No write is performed if the payload is not a valid JSON or the path cannot be found.

---

Scripts

In MQTT there is no definition of how the data is structured. To overcome this problem each topic has a script attached to convert the data to a correct format if needed. Script example and more info on how the conversion works is defined in this guide. Script errors can be viewed in the LM Error log. Each script has its own global environment, meaning that global variable values are kept between each execution but these variables are not shared between scripts.

Please report any issues or questions here.

[myg]

Hi,

Very promising library, hides complexities of integration while allowing to focus on actual value conversion routines. I found one issue and one suggestion:

1) ISSUE: lack of scope isolation between scripts. If code will be reused by copy&paste, any issue where user failed to use "local" definition (variable or function) will cause to overwrite values between scripts, which can cause hard to debug issues. 
However, if fixed, i would like you to retain global scope within script, that is any defined global variable keeps its value between script executions. If not fixed, this should be explicitly mentioned in the documentation, and also suggestion that follows my post should be implemented  . 
One of the examples where it's beneficial having global variable retain its value is having Throttle function implementation like this:

Code:

local data = json.pdecode(payload)
if type(data) == 'table' then
  val = data.illuminance
 
  -- global variable here that should retain its value between executions
  last_update = last_update == nil and 0 or last_update
  ---
  local now = os.time()
  if((now - last_update) < 60) then
    return nil
  else
    last_update = now
    return val
  end
end

2) SUGGESTION: like in example above, i would like to have Throttle as a function that can be reused multiple times without having to copy-paste it. Right now with global scope creep this can be done indirectly, but it's a bad practice and also will depend on event execution order (event that defines function must be guaranteed to be launched first). It would be great to have separate script editor where i can define all global functions/variables.

UPD: after thinking for a while, it might be a bit more tricky to implement this function, as Throttle usually implemented via capturing state variable in closure, creating new function instance for each usage. Need to think more about how to best implement it in this plugin. Anyway, having set of global functions is still a good addition.

---

3) SUGGESTION: add enable/disable flag to each object to allow for quick temporary enable and disable of mapping

---

question: how data can be backed up / restored? if i recover LM from backup, how i can make sure that application configuration is restored as well?

[Daniel]

Thanks for feedback.
As any other app the app is backed up with the device backup.
We will think about the throttle.
Why would you want to disable mapping?

[myg]

Why would you want to disable mapping?

not all mapping, just certain ones for debugging or other purposes. Like you can disable individual event scripts.

We will think about the throttle.

Throttle is just one of the possible filter functions. But to be honest, i've checked my current mappings and this is the only stateful function that i'm using, so maybe build-in implementation will be sufficient. Still some kind of global functions might be useful in the future.

[admin]

Having a local environment for each script is possible but this way certain functions might misbehave. This needs testing.

Reusable functions: a possible solution are user libraries so only one require call is needed in each script.

Configuration is placed in storage so it's a part of the backup as Daniel already mentioned.

[myg]

Reusable functions: a possible solution are user libraries so only one require call is needed in each script.

good idea, i forgot i can use require() call, will try this appoach.


I'm currently doing exercise remapping all zigbee devices from my script to this new application, and i found few suggestions that i think will make it easier.

1) According to (industry) standard of zigbee2mqtt, device state should be set via same topic with "/set" suffix. So to control and read state of the device (for ex. relay), I cannot use "Both" direction, since value and control topics are different. I need to create 2 separate rules, one for IN and one for OUT.
- SUGGESTION: for "Both" setting, add separate optional SET topic to the configuration window

2) I think (from my case) 90% of zigbee devices are sending single usable value in a json property, the rest of the values are system info like link quality or battery info, which are usually not mapped. With current implementation it is not possible to extract this single value without writing several strings of code over and over again. 
- SUGGESTION: add support for extracting attribute value directly. Potentially topic syntax can be extended to something like this: "zigbee2mqtt/sensor_name[@temperature]". You can even extend this to passing named variables to the script, like "temperature" becomes local variable in the script holding extracted value.

---

BUG: mappings list is not sorted, please implement sorting by topic name, then direction, then mapped object, so in/out events can be grouped and displayed correctly

[myg]

UPD: i moved all my rules to the new application (54 total) hope it won't break anything :-D

so regarding throttle, the following approach seems to be working, with the knowledge that i should use unique names per each script

"In" event:

Code:

ls_bedroom_throttle = ls_bedroom_throttle or require('user.mqttclient_common').throttle(60)

local data = json.pdecode(payload)
if type(data) == 'table' then
  return ls_bedroom_throttle(data.illuminance)
end

user library:

Code:

return {
  throttle = function(timeout)
    local last_update = 0
    return function(val)
      if(val ~= nil) then
        local now = os.time()
        if((now - last_update) < timeout) then
          return nil
        else
          last_update = now
          return val
        end
      end
    end
  end
}

---

SUGGESTION: 
1) in error log make it obvious which script contains error, so i can fix it easily. For example this error does not have information where to look:

error: failed to run script - [string "in-26"]:9: attempt to compare nil with number

2) output errors in global error log so they can be noticed in time

[Daniel]

There is new version in the original post.