Linking Multiple LM's

[sjfp]

I need to Link 5 LM (GSM) machine's back to a central LM Lite, (This a demo system to prove concept to a client) 
Was looking how I can send various Object values between then all, sending alerts back to the Central and the Central LM sending required control cammands like operating relays etc.
Thinking to use as a temporary quick fix for proof of concept Remote Services, but I would like to establish a more scalable and secure solution going forward, as there could be a significant amount of remote LM's on sites (100+)
So at the present all LM will sit on a OPEN VPN Network, so using Remote Service is OK for now. 

Been reading on the forum about MQTT and Broker based solutions, which appears to be the way forward.

Just seeing if there is any information or examples of how to get ALL LM's communicating between remote site and the central LM as above.

Any suggestions or advise would be useful.

[Daniel]

The web services would work but MQTT might be better option for such bigger installation.
For your demo you can install MQTT broker app, in attachment
You need RC2 fw but if you use VPN then you already have it. 
As for clients you can use this script
https://forum.logicmachine.net/showthrea...6#pid10926

On the client with broker in use ip 172.0.0.1

For installation for 100LMs it would be recommended to install MQTT broker on your VPN server. 
Hard to say how many clients broker on LM can handle.

BR

[sjfp]

thank you Daniel. will have a read

[sjfp]

Daniel, thank you for the example.
Have tested the solutions (thank you) and works well. Just what I was looking for.
Now tested, I am going to move the broker onto a cloud, done and works great. but now need to secure connection to the broker. How can I implement a user name a password for each client to use. Stopping unauthorized use of broker,

[Daniel]

If you use it inside the VPN then it is already secured. Only devices which are able to connect to VPN will be able to access broker. Do you want to do it outside of VPN?

[sjfp]

Yes please.
The VPN is on one clients site, but another project isn't, so securing the MQTT is important as well.
Plus how can add info on which LM is connected.
On the broker I can see active clients (only a number like 4 client), but cant see which ones.
Is there a way to find out, and also no if the client disappears ??
Sorry if this is an issue.

[Daniel]

For connection with encryption use this example. Enable MQTT broker with encryption and add a user and password in the Broker app
You may need to reboot LM with broker after changing settings.

Code:

if not broker then
  broker = '192.168.0.10'
 
  username = 'user'
  password = 'password'
 
  socket = require('socket')

  port = 8883

  function multiply(mult)
    return function(value)
      local num = tonumber(value)
      if num then
        return num * mult
      else
        return value
      end
    end
  end

  -- topic to object map
  mqtt_to_object = {
    ['in/topic1'] = '1/0/0',
    ['in/topic2'] = '1/0/19',
  }

  -- optional topic value conversion function
  mqtt_to_object_conv = {
  --  ['in/topic1'] = multiply(100),
  --  ['in/topic2'] = multiply(0.01),
  }

  -- object to topic map
  object_to_mqtt = {
    ['1/0/0'] = 'out/topic1',
    ['1/0/19'] = 'out/topic2',
  }

  datatypes = {}

  grp.sender = 'mq'
  require('socket')

  for addr, _ in pairs(object_to_mqtt) do
    local obj = grp.find(addr)
    if obj then
      datatypes[ addr ] = obj.datatype
    end
  end

  mclient = require('mosquitto').new()

  mclient.ON_CONNECT = function(res, ...)
    log('mqtt connect status', res, ...)

    if res then
      for topic, _ in pairs(mqtt_to_object) do
        mclient:subscribe(topic)
      end
    else
      mclient:disconnect()
    end
  end

  mclient.ON_MESSAGE = function(mid, topic, payload)
    local addr = mqtt_to_object[ topic ]
    if addr then
      local fn = mqtt_to_object_conv[ topic ]

      if fn then
        payload = fn(payload)
      end

      grp.write(addr, payload)
    end
  end

  mclient.ON_DISCONNECT = function(...)
    log('mqtt disconnect', ...)
    mclientfd = nil
  end

  function mconnect()
    local status, rc, msg, fd

    status, rc, msg = mclient:connect(broker, port)

    if not status then
      log('mqtt connect failed ' .. tostring(msg))
    end

    fd = mclient:socket()
    if fd then
      mclientfd = fd
    end
  end
 
    mclient:tls_insecure_set(true)
  mclient:login_set(username, password or '')
 
  mconnect()

  function publishvalue(event)
    -- message from us or client is not connected
    if event.sender == 'mq' or not mclientfd then
      return
    end

    local addr = event.dst
    local dpt = datatypes[ addr ]
    local topic = object_to_mqtt[ addr ]

    -- unknown object
    if not dpt or not topic then
      return
    end

    local value = busdatatype.decode(event.datahex, dpt)
    if value ~= nil then
      if type(value) == 'boolean' then
        value = value and 1 or 0
      end

      mclient:publish(topic, tostring(value))
    end
  end

  lbclient = require('localbus').new(1)
  lbclient:sethandler('groupwrite', publishvalue)

  lbclientfd = socket.fdmaskset(lbclient:getfd(), 'r')

  -- run timer every 5 seconds
  timer = require('timerfd').new(5)
  timerfd = socket.fdmaskset(timer:getfd(), 'r')
end

-- mqtt connected
if mclientfd then
  mclientfdset = socket.fdmaskset(mclientfd, mclient:want_write() and 'rw' or 'r')
  res, lbclientstat, timerstat, mclientstat =
      socket.selectfds(10, lbclientfd, timerfd, mclientfdset)
-- mqtt not connected
else
  res, lbclientstat, timerstat =
    socket.selectfds(10, lbclientfd, timerfd)
end

if mclientstat then
  if socket.fdmaskread(mclientstat) then
    mclient:loop_read()
  end

  if socket.fdmaskwrite(mclientstat) then
    mclient:loop_write()
  end
end

if lbclientstat then
  lbclient:step()
end

if timerstat then
  -- clear armed timer
  timer:read()

  if mclientfd then
    mclient:loop_misc()
  else
    mconnect()
  end
end

[sjfp]

Daniel, just tried the provided code, change IP address and port number (1883) but wont connect to the broker.
Tried the connection with a app on android phone with the same credentials, and connect with no issue. So I think there may be an issue in the code provided. Sorry :-(

[Daniel]

You have to use port 8883 I tested it and works.  Make sure you  Enable MQTT broker with encryption, add user and password then reboot LM.

[admin]

If you don't use encryption you need to remove the line that calls tls_insecure_set

[sjfp]

Thank you both.
Tried the tls_insecure_set and worked fine. Will have a go at yours Daniel later today.
Thank you both for your help.

[sjfp]

Having an issue sending 10.3 byte time / day . object data between LM's
Is there a conversation that needs to be preformed ?

[Daniel]

The script you use is sending data in string format. This dpt is a table and will not work. Use date/time in numeric format.

[Daniel]

Use this script then data will be send in json format so even dpt date and time can be sent.  You can also set qos and retain

Code:

if not broker then
  socket = require('socket')
  json = require('json')

  broker = '192.168.0.10'
  port = 8883

  username = 'user'
  password = 'password'

  -- topic to object map
  mqtt_to_object = {
    ['in/topic1'] = {
      addr = '1/0/0',
      convert = json.pdecode,
    },
    ['in/topic2'] = {
      addr = '1/0/19',
    }
  }

  -- object to topic map
  object_to_mqtt = {
    ['1/0/0'] = {
      topic = 'out/topic1',
      qos = 1,
      retain = true,
      convert = json.encode,
    },
    ['1/0/19'] = {
      topic = 'out/topic2',
    }
  }

  datatypes = {}

  grp.sender = 'mq'

  for addr, _ in pairs(object_to_mqtt) do
    local obj = grp.find(addr)
    if obj then
      datatypes[ addr ] = obj.datatype
    end
  end

  mclient = require('mosquitto').new()

  mclient.ON_CONNECT = function(res, ...)
    log('mqtt connect status', res, ...)

    if res then
      for topic, _ in pairs(mqtt_to_object) do
        mclient:subscribe(topic)
      end
    else
      mclient:disconnect()
    end
  end

  mclient.ON_MESSAGE = function(mid, topic, payload)
    local map = mqtt_to_object[ topic ]
    if map then
      if map.convert then
        payload = map.convert(payload)
      end

      grp.write(map.addr, payload)
    end
  end

  mclient.ON_DISCONNECT = function(...)
    log('mqtt disconnect', ...)
    mclientfd = nil
  end

  function mconnect()
    local status, rc, msg, fd

    status, rc, msg = mclient:connect(broker, port)

    if not status then
      log('mqtt connect failed ' .. tostring(msg))
    end

    fd = mclient:socket()
    if fd then
      mclientfd = fd
    end
  end

  mclient:tls_insecure_set(true)
  mclient:login_set(username, password or '')

  mconnect()

  function publishvalue(event)
    -- message from us or client is not connected
    if event.sender == 'mq' or not mclientfd then
      return
    end

    local addr = event.dst
    local dpt = datatypes[ addr ]
    local map = object_to_mqtt[ addr ]

    -- unknown object
    if not dpt or not map then
      return
    end

    local value = busdatatype.decode(event.datahex, dpt)
    if value ~= nil then
      if map.convert then
        value = map.convert(value)
      elseif type(value) == 'boolean' then
        value = value and 1 or 0
      end

      mclient:publish(map.topic, tostring(value), map.qos or 0, map.retain)
    end
  end

  lbclient = require('localbus').new(1)
  lbclient:sethandler('groupwrite', publishvalue)

  lbclientfd = socket.fdmaskset(lbclient:getfd(), 'r')

  -- run timer every 5 seconds
  timer = require('timerfd').new(5)
  timerfd = socket.fdmaskset(timer:getfd(), 'r')
end

-- mqtt connected
if mclientfd then
  mclientfdset = socket.fdmaskset(mclientfd, mclient:want_write() and 'rw' or 'r')
  res, lbclientstat, timerstat, mclientstat =
      socket.selectfds(10, lbclientfd, timerfd, mclientfdset)
-- mqtt not connected
else
  res, lbclientstat, timerstat =
    socket.selectfds(10, lbclientfd, timerfd)
end

if mclientstat then
  if socket.fdmaskread(mclientstat) then
    mclient:loop_read()
  end

  if socket.fdmaskwrite(mclientstat) then
    mclient:loop_write()
  end
end

if lbclientstat then
  lbclient:step()
end

if timerstat then
  -- clear armed timer
  timer:read()

  if mclientfd then
    mclient:loop_misc()
  else
    mconnect()
  end
end

[Erwin van der Zwart]

Hi Daniel,

I see in your last sample that you have added the field "qos" (quality of service), i have a couple of questions about that:

For the qos, when i set it to 1 how do we handle the PUBACK packet? If we set it to 2 how do we handle the PUBREC, PUBREL and PUBCOMP packages? Is this handled in the lib already and the re-sending of a topic value is automatically handled and we don't have to take care of this in the client script?

Another question is:  Can we simply set a extra field 'lwt' in the object to mqtt table to set the last will and testament message?

BR,

Erwin

[Daniel]

Hi Erwin

QOS is handled automatically in lib.

You can set 'Last will' like this
client:will_set(topic, message, qos = 0, retain = false)
So in example script it will looks like that.
mclient:will_set('in/topic2', 'disconnected', 0, true)

Add this after line
mclient:login_set(username, password or '')

[Erwin van der Zwart]

Hi Daniel,

Thanks!

Works perfect, now that we have a MQTT client in Ecostruxure Building Operation 3.2 (still BETA) i have a nice testing broker and it opens up a lot of options for integration.

BR,

Erwin

[Erwin van der Zwart]

Hi Daniel,

It's possible to send a null value (retained) to a topic to clear the retained value, but is there also a command to clear retained values from a topic including all subtopics or even all topics? Otherwise it's quite a pita to clear retained values if needed (for example during testing).

BR,

Erwin

[admin]

From MQTT standard:

A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes will be processed as normal by the Server and sent to Clients with a subscription matching the topic name. Additionally any existing retained message with the same topic name MUST be removed and any future subscribers for the topic will not receive a retained message. “As normal” means that the RETAIN flag is not set in the message received by existing Clients. A zero byte retained message MUST NOT be stored as a retained message on the Server.

[jmir]

Hi,

I've been testing this code using one LM as broker+publisher and a second one as publisher.
I realized that when rebooting the LM that is not the broker, it can't reconnect until I disable and enable the script...


This is the log:

* arg: 1
  * string: mqtt disconnect
* arg: 2
  * bool: false
* arg: 3
  * number: 14
* arg: 4
  * string: unexpected disconnect
* arg: 5
  * nil

I've added the line "broker = nil" here and now it's working, but i'm not sure that it is the best way to solve it...

Code:

  mclient.ON_DISCONNECT = function(...)
    log('mqtt disconnect', ...)
    mclientfd = nil
    broker = nil
  end

Is it correct?
Thanks!