MqTT in Wiser for KNX

[JRP]

Hello
I have been trying to use wiser for KNX to connect to an MQTT broker (mosquitto) but am getting nothing.
The broker is running on my PC, I check the topics and they work, in the script I put the ip address of the pc where the broker runs and I have the addresses defined in the objects tab, but without result.
I don't know if wiser have what it takes to connect or if it is necessary to install libmosquitto.ipk, luamosquitto.ipk and luasockets.ipk as it comes in the Gbridge.io guide that I found in the forum.

Would someone be so kind as to tell me how to proceed?
Thank you.
a greeting

[Daniel]

First we have to see your script. What fw do you use?
Have you seen this thread?
https://forum.logicmachine.net/showthread.php?tid=2628

[JRP]

First we have to see your script. What fw do you use?
Have you seen this thread?

Hi Daniel thanks for replying.

Wiser has the fw 2.4.0 as you can see in the following image.


I use the script that I publish Admin in this thread
https://forum.logicmachine.net/showthrea...light=mqtt
I have created the resident script at interval 0 and changed the IP address to the address of the pc where the mosquitto broker runs.
The object mapping I have used the same as a test.

Code:

if not broker then
  broker = '192.168.0.12'

  function multiply(mult)
    return function(value)
      local num = tonumber(value)
      if num then
        return num * mult
      else
        return value
      end
    end
  end

  -- topic to object map
  mqtt_to_object = {
    ['in/topic1'] = '32/1/1',
    ['in/topic2'] = '32/1/2',
  }

   -- object to topic map
  object_to_mqtt = {
    ['1/1/1'] = 'out/topic1',
    ['1/1/2'] = 'out/topic2',
  }

  datatypes = {}

  grp.sender = 'mq'
  require('socket')

  for addr, _ in pairs(object_to_mqtt) do
    local obj = grp.find(addr)
    if obj then
      datatypes[ addr ] = obj.datatype
    end
  end

  mclient = require('mosquitto').new()

  mclient.ON_CONNECT = function(res, ...)
    log('mqtt connect status', res, ...)

    if res then
      for topic, _ in pairs(mqtt_to_object) do
        mclient:subscribe(topic)
      end
    else
      mclient:disconnect()
    end
  end

  mclient.ON_MESSAGE = function(mid, topic, payload)
    local addr = mqtt_to_object[ topic ]
    if addr then
      local fn = mqtt_to_object_conv[ topic ]

      if fn then
        payload = fn(payload)
      end

      grp.write(addr, payload)
    end
  end

  mclient.ON_DISCONNECT = function(...)
    log('mqtt disconnect', ...)
    mclientfd = nil
  end

  function mconnect()
    local fd

    mclient:connect(broker)
    fd = mclient:socket()

    -- fd ref is valid
    if fd then
      mclientfd = fd
    end
  end

  mconnect()

  function publishvalue(event)
    -- message from us or client is not connected
    if event.sender == 'mq' or not mclientfd then
      return
    end

    local addr = event.dst
    local dpt = datatypes[ addr ]
    local topic = object_to_mqtt[ addr ]

    -- unknown object
    if not dpt or not topic then
      return
    end

    local value = busdatatype.decode(event.datahex, dpt)
    if value ~= nil then
      if type(value) == 'boolean' then
        value = value and 1 or 0
      end

      mclient:publish(topic, tostring(value))
    end
  end

  lbclient = require('localbus').new(1)
  lbclient:sethandler('groupwrite', publishvalue)

  lbclientfd = socket.fdmaskset(lbclient:getfd(), 'r')

  -- run timer every 5 seconds
  timer = require('timerfd').new(5)
  timerfd = socket.fdmaskset(timer:getfd(), 'r')
end

-- mqtt connected
if mclientfd then
  mclientfdset = socket.fdmaskset(mclientfd, mclient:want_write() and 'rw' or 'r')
  res, lbclientstat, timerstat, mclientstat =
      socket.selectfds(10, lbclientfd, timerfd, mclientfdset)
-- mqtt not connected
else
  res, lbclientstat, timerstat =
    socket.selectfds(10, lbclientfd, timerfd)
end

if mclientstat then
  if socket.fdmaskread(mclientstat) then
    mclient:loop_read()
  end

  if socket.fdmaskwrite(mclientstat) then
    mclient:loop_write()
  end
end

if lbclientstat then
  lbclient:step()
end

if timerstat then
  -- clear armed timer
  timer:read()

  if mclientfd then
    mclient:loop_misc()
  else
    mconnect()
  end
end

Putting the log lines suggested in that same post, this is the result.
MQTT 20.05.2020 16:03:19
Resident script:130: bad argument #2 to 'selectfds' (number expected, got nil)
stack traceback:
[C]: in function 'selectfds'
Is it necessary to install a library or package?
The topics are published and the broker subscribed to them.
Thank you.
a greeting

[Daniel]

In the link which I originally forgot to paste you can find how to use encrypted communication
https://forum.logicmachine.net/showthread.php?tid=2628

[JRP]

In the link which I originally forgot to paste you can find how to use encrypted communication
https://forum.logicmachine.net/showthread.php?tid=2628

Hello
Thank you for responding.
In the broker I have not activated any user or password, as it is only a test for now.
I'm still wondering if it's necessary to install some library in wiser.
Greetings and thanks for helping

[Erwin van der Zwart]

Hi,

There is already a lib in the Wiser 2.4 firmware however i don't think it's the most recent version as 2.4 is out for quite a while.

I'm currently testing on FW 2.5 that will be released (don't pinpoint me on it) somewhere around this summer, and all features that are in Daniels latest script work on FW 2.5 including the broker app.

So you need to wait a couple of weeks (:

BR,

Erwin

[Daniel]

I just tried this script
https://forum.logicmachine.net/showthrea...6#pid16896
on SL with fw 2.4.0 and it works just fine. I used broker on LM.

[JRP]

Hi,

There is already a lib in the Wiser 2.4 firmware however i don't think it's the most recent version as 2.4 is out for quite a while.

I'm currently testing on FW 2.5 that will be released (don't pinpoint me on it) somewhere around this summer, and all features that are in Daniels latest script work on FW 2.5 including the broker app.

So you need to wait a couple of weeks (:

BR,

Erwin

I just tried this script
https://forum.logicmachine.net/showthrea...6#pid16896
on SL with fw 2.4.0 and it works just fine. I used broker on LM.

Hello
Thanks Erwin for the information on the fw update. While the update arrives I will do some more tests.

Hellol Daniel.
Thanks for answering.
I have activated user and password in the mosquito broker, I have copied the script that you indicate and changed ip, user, password and port (1883 is where mosquitto runs on my PC).
I use the MQTT.fx client and post both topic in / topic1 and out / topic1.
I subscribe with the same MQTT.fx and with mosquitto_sub and it works perfectly, but I don't see any changes in the wiser.
In the register tab I see that it does not connect to the broker, put this.

* string: mqtt connect failed Operation timed out
This is the script (I remove the password).

Code:

if not broker then
  socket = require('socket')
  json = require('json')

  broker = '192.168.0.12'
  port = 1883

  username = 'wiser'
  password = 'mypassword'

  -- topic to object map
  mqtt_to_object = {
    ['in/topic1'] = {
      addr = '1/0/0',
      convert = json.pdecode,
    },
    ['in/topic2'] = {
      addr = '1/0/19',
    }
  }

  -- object to topic map
  object_to_mqtt = {
    ['1/0/0'] = {
      topic = 'out/topic1',
      qos = 1,
      retain = true,
      convert = json.encode,
    },
    ['1/0/19'] = {
      topic = 'out/topic2',
    }
  }

  datatypes = {}

  grp.sender = 'mq'

  for addr, _ in pairs(object_to_mqtt) do
    local obj = grp.find(addr)
    if obj then
      datatypes[ addr ] = obj.datatype
    end
  end

  mclient = require('mosquitto').new()

  mclient.ON_CONNECT = function(res, ...)
    log('mqtt connect status', res, ...)

    if res then
      for topic, _ in pairs(mqtt_to_object) do
        mclient:subscribe(topic)
      end
    else
      mclient:disconnect()
    end
  end

  mclient.ON_MESSAGE = function(mid, topic, payload)
    local map = mqtt_to_object[ topic ]
    if map then
      if map.convert then
        payload = map.convert(payload)
      end

      grp.write(map.addr, payload)
    end
  end

  mclient.ON_DISCONNECT = function(...)
    log('mqtt disconnect', ...)
    mclientfd = nil
  end

  function mconnect()
    local status, rc, msg, fd

    status, rc, msg = mclient:connect(broker, port)

    if not status then
      log('mqtt connect failed ' .. tostring(msg))
    end

    fd = mclient:socket()
    if fd then
      mclientfd = fd
    end
  end

  mclient:tls_insecure_set(true)
  mclient:login_set(username, password or '')

  mconnect()

  function publishvalue(event)
    -- message from us or client is not connected
    if event.sender == 'mq' or not mclientfd then
      return
    end

    local addr = event.dst
    local dpt = datatypes[ addr ]
    local map = object_to_mqtt[ addr ]

    -- unknown object
    if not dpt or not map then
      return
    end

    local value = busdatatype.decode(event.datahex, dpt)
    if value ~= nil then
      if map.convert then
        value = map.convert(value)
      elseif type(value) == 'boolean' then
        value = value and 1 or 0
      end

      mclient:publish(map.topic, tostring(value), map.qos or 0, map.retain)
    end
  end

  lbclient = require('localbus').new(1)
  lbclient:sethandler('groupwrite', publishvalue)

  lbclientfd = socket.fdmaskset(lbclient:getfd(), 'r')

  -- run timer every 5 seconds
  timer = require('timerfd').new(5)
  timerfd = socket.fdmaskset(timer:getfd(), 'r')
end

-- mqtt connected
if mclientfd then
  mclientfdset = socket.fdmaskset(mclientfd, mclient:want_write() and 'rw' or 'r')
  res, lbclientstat, timerstat, mclientstat =
      socket.selectfds(10, lbclientfd, timerfd, mclientfdset)
-- mqtt not connected
else
  res, lbclientstat, timerstat =
    socket.selectfds(10, lbclientfd, timerfd)
end

if mclientstat then
  if socket.fdmaskread(mclientstat) then
    mclient:loop_read()
  end

  if socket.fdmaskwrite(mclientstat) then
    mclient:loop_write()
  end
end

if lbclientstat then
  lbclient:step()
end

if timerstat then
  -- clear armed timer
  timer:read()

  if mclientfd then
    mclient:loop_misc()
  else
    mconnect()
  end
end

The port number should not be between ''?
He does not put them in the thread from which he copies it.
Thank you very much for the help.
Greetings

[Daniel]

If you don't use encryption (port 1883)
Then change this

mclient:tls_insecure_set(true)

to

mclient:tls_insecure_set(false)

[JRP]

If you don't use encryption (port 1883)
Then change this

mclient:tls_insecure_set(true)

to

mclient:tls_insecure_set(false)

Hello
I have changed it and the same error continues.
Thanks for answering.
a greeting

[Daniel]

This work with my broker so your issue is somewhere there.  Can you connect to your broker by using  MQTT Explorer  client but from another machine?

[JRP]

This work with my broker so your issue is somewhere there.  Can you connect to your broker by using  MQTT Explorer  client but from another machine?

Hello
I have tried with Mqtt explorer and MQtt.fx and neither of them connects from another pc. I put the IP of the pc where I have mosquitto installed and the username and password and nothing.
Surely the problem comes from here.
I have all the computers on the same network, I ping and they all respond.
If in MQtt.fx installed on the same pc as the broker, I put instead of address 127 ......., the ip address of the same pc 192.168.012. It connects and on netstat I see the connection to port 1883.
Will I be the router that blocks traffic? It is what occurs to me.
Thanks for the help.
a greeting

[Daniel]

It is most likely your PC firewall.

[JRP]

It is most likely your PC firewall.

Hello
Eureka!
Effectively it is the w10 firewall that was blocking traffic, I have opened port 1883 and it has connected and worked the first time.
After writing the last message it occurred to me that it could be the firewall, but until you told me to try from another PC, the light bulb did not light up.
Thank you very much Daniel for your interest and your reconmedances.
a greeting

[manos@dynamitec]

Hello Erwin,

I have a SpaceLynk (V1.3 FW 2.7.0) that I would like to use as MQTT client and publish data to a cloud platform here in Belgium "Eniris". This device will serve only the purpose of MQTT client.
Could you or anyone assist me with the MQTT client script to be able to publish data in a certain json structured format?

The format should be like mentioned in the link below:

https://wiki.eniris.be/wiki/publicinform...3A%20MQTT/

In short something like this:

Topic = 'standart1/rp_one_m/T/userid/' and then in the data packet the following
 

Code:

{
  "time" :  1661506491 , --In Unix format
  "extraTags" : {
    "subId" :  "T19"
  },
  "fields" : {
    "temperature_degC" :  16.76 --this must be the value of the KNX object as a float datatype towards the broker)
  }
}

I have tested the MQTT client script below and the connection with the broker is ok. And data is being published but not in the certain format sctructure their platform is expecting it.

Code:

if not broker then
  broker = 'mqtt.eniris.be'
 
  username = 'xxx'
  password = 'xxx'
 
  socket = require('socket')

  port = 1883

  function multiply(mult)
    return function(value)
      local num = tonumber(value)
      if num then
        return num * mult
      else
        return value
      end
    end
  end

  -- topic to object map
  mqtt_to_object = {
    ['Actual Temp/Hal A/S2'] = '4/0/1',
    ['Actual Temp/Hal B/S1'] = '4/0/2',
  }

  -- optional topic value conversion function
  --mqtt_to_object_conv = {
  --  ['in/topic1'] = multiply(100),
  --  ['in/topic2'] = multiply(0.01),
  --}

  -- object to topic map
  object_to_mqtt = {
    ['4/0/1'] = 'Actual Temp/Hal A/S2',
    ['4/0/2'] = 'Actual Temp/Hal B/S1',
  }

  datatypes = {}

  grp.sender = 'mq'
  require('socket')

  for addr, _ in pairs(object_to_mqtt) do
    local obj = grp.find(addr)
    if obj then
      datatypes[ addr ] = obj.datatype
    end
  end

  mclient = require('mosquitto').new()

  mclient.ON_CONNECT = function(res, ...)
    log('mqtt connect status', res, ...)

    if res then
      for topic, _ in pairs(mqtt_to_object) do
        mclient:subscribe(topic)
      end
    else
      mclient:disconnect()
    end
  end

  mclient.ON_MESSAGE = function(mid, topic, payload)
    local addr = mqtt_to_object[ topic ]
    if addr then
      local fn = mqtt_to_object_conv[ topic ]

      if fn then
        payload = fn(payload)
      end

      grp.write(addr, payload)
    end
  end

  mclient.ON_DISCONNECT = function(...)
    log('mqtt disconnect', ...)
    mclientfd = nil
  end

  function mconnect()
    local status, rc, msg, fd

    status, rc, msg = mclient:connect(broker, port)

    if not status then
      log('mqtt connect failed ' .. tostring(msg))
    end

    fd = mclient:socket()
    if fd then
      mclientfd = fd
    end
  end
 
--mclient:tls_insecure_set(true)
  mclient:login_set(username, password or '')
 
  mconnect()

  function publishvalue(event)
    -- message from us or client is not connected
    if event.sender == 'mq' or not mclientfd then
      return
    end

    local addr = event.dst
    local dpt = datatypes[ addr ]
    local topic = object_to_mqtt[ addr ]

    -- unknown object
    if not dpt or not topic then
      return
    end

    local value = busdatatype.decode(event.datahex, dpt)
    if value ~= nil then
      if type(value) == 'boolean' then
        value = value and 1 or 0
      end

      mclient:publish(topic, tostring(value))
    end
  end

  lbclient = require('localbus').new(1)
  lbclient:sethandler('groupwrite', publishvalue)

  lbclientfd = socket.fdmaskset(lbclient:getfd(), 'r')

  -- run timer every 5 seconds
  timer = require('timerfd').new(5)
  timerfd = socket.fdmaskset(timer:getfd(), 'r')
end

-- mqtt connected
if mclientfd then
  mclientfdset = socket.fdmaskset(mclientfd, mclient:want_write() and 'rw' or 'r')
  res, lbclientstat, timerstat, mclientstat =
      socket.selectfds(10, lbclientfd, timerfd, mclientfdset)
-- mqtt not connected
else
  res, lbclientstat, timerstat =
    socket.selectfds(10, lbclientfd, timerfd)
end

if mclientstat then
  if socket.fdmaskread(mclientstat) then
    mclient:loop_read()
  end

  if socket.fdmaskwrite(mclientstat) then
    mclient:loop_write()
  end
end

if lbclientstat then
  lbclient:step()
end

if timerstat then
  -- clear armed timer
  timer:read()

  if mclientfd then
    mclient:loop_misc()
  else
    mconnect()
  end
end

Thank you in advance for your responce.

Regards

[admin]

If you don't have a lot of frequent updates you can use event script to publish new values then disconnect.

Add to Common functions and change username/password:

Code:

function mqttpublish(topic, data)
  local broker = 'mqtt.eniris.be'

  local username = 'xxx'
  local password = 'xxx'

  local port = 1883

  local mqtt = require('mosquitto')
  local client = mqtt.new()

  if type(data) == 'table' then
    data = require('json').encode(data)
  end

  client.ON_CONNECT = function(status, rc, msg)
    if status then
      log('mqtt cloud connected')
      client:publish(topic, data)
    else
      log('mqtt cloud connect failed ' .. tostring(msg))
      client:disconnect()
    end
  end

  client.ON_PUBLISH = function()
    log('disconnecting')
    client:disconnect()
  end

  client:login_set(username, password)
  local status, rc, msg = client:connect(broker, port)

  if status then
    client:loop_forever()
  else
    log('cloud connect failed: ' .. tostring(msg))
  end
end

Event script example:

Code:

topic = 'Actual Temp/Hal A/S2'
data = {
  time = os.time(),
  extraTags = {
    subId = 'T19'
  },
  fields = {
    temperature_degC = event.getvalue()
  }
}

mqttpublish(topic, data)

[manos@dynamitec]

If you don't have a lot of frequent updates you can use event script to publish new values then disconnect.

Add to Common functions and change username/password:

Code:

function mqttpublish(topic, data)
  local broker = 'mqtt.eniris.be'

  local username = 'xxx'
  local password = 'xxx'

  local port = 1883

  local mqtt = require('mosquitto')
  local client = mqtt.new()

  if type(data) == 'table' then
    data = require('json').encode(data)
  end

  client.ON_CONNECT = function(status, rc, msg)
    if status then
      log('mqtt cloud connected')
      client:publish(topic, data)
    else
      log('mqtt cloud connect failed ' .. tostring(msg))
      client:disconnect()
    end
  end

  client.ON_PUBLISH = function()
    log('disconnecting')
    client:disconnect()
  end

  client:login_set(username, password)
  local status, rc, msg = client:connect(broker, port)

  if status then
    client:loop_forever()
  else
    log('cloud connect failed: ' .. tostring(msg))
  end
end

Event script example:

Code:

topic = 'Actual Temp/Hal A/S2'
data = {
  time = os.time(),
  extraTags = {
    subId = 'T19'
  },
  fields = {
    temperature_degC = event.getvalue()
  }
}

mqttpublish(topic, data)

Hello Admin,

Thank you for the quick reply.
Unfortunately I will have to publish around 200 objects and its not that they come in the bus very frequently.
Is there and option to publish data every one minute? Like adding a resident script with sleep internal of 1min?

Regards,

[admin]

Are all 200 objects temperature sensors or are there different values that affect the JSON message format? Is extraTags = { subId = 'T19' } common for all messages or not?

[manos@dynamitec]

Are all 200 objects temperature sensors or are there different values that affect the JSON message format? Is extraTags = { subId = 'T19' } common for all messages or not?

Hello Admin,
 
The following things are to be published. 


WaterMeter
Data:
standard1/rp_one_m/waterMetrics/<username>
{
    "time": <UNIX timestamp>,
    "extraTags": {
        "subId": <string>
    }
    "fields": {
        "absVolume_m3": <float>,
        "flow_m3ph": <float>
    }
}

Weather Station
Data:
standard1/rp_one_m/weatherMetrics/<username>
{
    "time": <UNIX timestamp>,
    "extraTags": {
        "subId": <string>
    }
    "fields": {
        "outdoorTemperature_degC": <float>,
        "outdoorBrightness_lux": <float>,
        "windSpeed_kmph": <float>
    }
}

GasMeter
Data:
standard1/rp_one_m/gasMetrics/<username>
{
    "time": <UNIX timestamp>,
    "extraTags": {
        "subId": <string>
    }
    "fields": {
        "absVolume_Nm3": <float>
    }
}

Errors & Messages:
standard1/autogen/alarms/<username>
{
    "time": <UNIX timestamp>,
    "extraTags": {
        "subId": <string>
    }
    "fields": {
        "alarmActive": <boolean>,
        "msg": <srtring>,
        "mbusStatus": <int>    //zero = OK otherwise alarm
    }
}


Thank you again for the great support.

[admin]

You can create a scheduled script that runs every minute and publishes all values. Each message has to specified manually due to a variable format. Add as many addmessage calls as needed and change grp.getvalue addresses. The script will automatically disconnect once all messages are published.

Code:

broker = 'mqtt.eniris.be'

username = 'xxx'
password = 'xxx'

port = 1883

messages = {}
time = os.time()

function addmessage(topic, data)
  messages[ #messages + 1 ] = {
    topic = topic,
    data = require('json').encode(data),
  }
end

addmessage('standard1/rp_one_m/waterMetrics/' .. username, {
  time = time,
  extraTags = {
    subId = 'T19',
  },
  fields = {
    absVolume_m3 = grp.getvalue('1/1/1'),
    flow_m3ph = grp.getvalue('1/1/2'),
  }
})

addmessage('standard1/rp_one_m/weatherMetrics/' .. username, {
  time = time,
  extraTags = {
    subId = 'T19',
  },
  fields = {
    outdoorTemperature_degC = grp.getvalue('1/1/3'),
    outdoorBrightness_lux = grp.getvalue('1/1/4'),
    windSpeed_kmph = grp.getvalue('1/1/5'),
  }
})

addmessage('standard1/rp_one_m/gasMetrics/' .. username, {
  time = time,
  extraTags = {
    subId = 'T19',
  },
  fields = {
    absVolume_Nm3 = grp.getvalue('1/1/6'),
  }
})

mqtt = require('mosquitto')
client = mqtt.new()

mids = {}
pubcount = 0

client.ON_CONNECT = function(status, rc, msg)
  if status then
    log('mqtt cloud connected')

    for _, message in ipairs(messages) do
      local mid = client:publish(message.topic, message.data)

      if mid then
        pubcount = pubcount + 1
        mids[ mid ] = true
      end
    end

    if pubcount == 0 then
      client:disconnect()
    end
  else
    log('mqtt cloud connect failed ' .. tostring(msg))
    client:disconnect()
  end
end

client.ON_PUBLISH = function(mid)
  if mids[ mid ] then
    mids[ mid ] = nil
    pubcount = pubcount - 1

    if pubcount == 0 then
      log('disconnecting')
      client:disconnect()
    end
  end
end

client:login_set(username, password)
status, rc, msg = client:connect(broker, port)

if status then
  client:loop_forever()
else
  log('cloud connect failed: ' .. tostring(msg))
end