massive data logging

[gdimaria]

Hi, I need to trend log about 250/300 KNX objects. 

I know that's over LM possibilities due the 21 Mb archive limit.

Can you suggest an appropriate device to do this job?

I already knows BAB TECHNOLOGIE GmbH DATALOGGER V2 (https://bab-technologie.com/datalogger-v2/?lang=en), it could be ok.

But I trust in your experience and I would receive a suggest.

Thanks

Peppe

[admin]

With low resolution / retention settings you could fit 300 trends into LM. An alternative solution is to use Grafana - either cloud-hosted or by installing it locally on a small PC. Installation can be a bit tricky but it's a very flexible solution for storing time-series data.

[gdimaria]

Grafana seems to be very cool! And how to connect LM to it, sending telegrams via remote access?

[admin]

LM is sending data to InfluxDB using HTTP. This thread has working examples: https://forum.logicmachine.net/showthread.php?tid=1531
Another option is MQTT: https://grafana.com/blog/2021/08/12/stre...fana-live/

[gdimaria]

LM is sending data to InfluxDB using HTTP. This thread has working examples: https://forum.logicmachine.net/showthread.php?tid=1531
Another option is MQTT: https://grafana.com/blog/2021/08/12/stre...fana-live/

Things are little changed on InfluxDB:

https://docs.influxdata.com/influxdb/clo...fluxdb-api

it seems a little too complicated to me... no shorter ways to implement a massive data logging?

Peppe

[admin]

You only need to create a bucket once. It will store your data for a certain amount of time. There are examples of sending data to influx v2 in the topic I've linked to.
The setup part is a bit complicated but you only have to do it once

[gdimaria]

You only need to create a bucket once. It will store your data for a certain amount of time. There are examples of sending data to influx v2 in the topic I've linked to.
The setup part is a bit complicated but you only have to do it once

I accepted the challenge!
But right now I get error messages:


* arg: 1
  * string: error sending to influx
* arg: 2
  * string: {"code":"invalid","message":"at line 1:36: cannot parse value for field key \"value\": invalid float value syntax (check rejected_points in your _monitoring bucket for further information)"}
* arg: 3
  * number: 400
* arg: 4
  * string: rawdata,name=ora,addr=33/1/9 value=10:57


and


* arg: 1
  * string: error sending to influx
* arg: 2
  * string: {"code":"invalid","message":"at line 1:47: cannot parse value for field key \"value\": invalid bool value \"Thu\" (check rejected_points in your _monitoring bucket for further information)"}
* arg: 3
  * number: 400
* arg: 4
  * string: rawdata,name=giornomeseanno,addr=33/1/8 value=Thu 15-09-2022


the script I used is the following, can you help me?

Code:

local socket = require('socket')
local http = require('socket.http')
http.TIMEOUT = 5


local dt_cache = {}
function get_value(addr, datahex)
  local dt = dt_cache[addr]

  if(not dt) then
    dt = grp.find(addr)
    dt_cache[addr] = dt
  end
 
  return knxdatatype.decode(datahex, dt.datatype), dt
end

function knx_callback(event)
  local addr = event.dst
  local value, dt = get_value(addr, event.datahex)

  send_metric('rawdata', dt.name, addr, value)
end


function send_metric(table, name, addr, value)
  if name == nil or name == '' then
    return
  end

  name = string.gsub(name, ' ', '\\ ')

  local url = 'https://europe-west1-1.gcp.cloud2.influxdata.com/api/v2/write?bucket=XX&org=XXX'

  local body

  if type(value) == 'boolean' then
    body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)
  else
    -- most likely number
    body = string.format('%s,name=%s,addr=%s value=%s', table, name, addr, value)
  end

  local res, code = http.request({
    url = url,
    method = 'POST',
    body = body,
    headers = {
      Authorization = 'Token XXXXXXXXXXXXXXXXXXX'
    }
  })

  if code ~= 204 then
    log('error sending to influx', res, code, body)
  end
end

function run_loop()
  local bus = require('localbus').new(1)
  bus:sethandler('groupwrite', knx_callback)
  local busfd = socket.fdmaskset(bus:getfd(), 'r')

 
  while(true) do
    res, fbus = socket.selectfds(10, busfd)
    if(fbus) then
      bus:step()
    end
  end
end

run_loop()

[admin]

You're sending string data (10:57 and Thu 15-09-2022) which Influx does not understand. String values must be enclosed in quotes.
Try replacing lines 35..42 with this:

Code:

local body

if type(value) == 'boolean' or type(value) == 'number' then
  body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)
elseif type(value) == 'string' then
  body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value)
else
  log('invalid data type', addr, type(value))
  return
end

Currently this script sends values for all objects. There should be some filtering implemented so only the required data is pushed.

[gdimaria]

You're sending string data (10:57 and Thu 15-09-2022) which Influx does not understand. String values must be enclosed in quotes.
Try replacing lines 35..42 with this:

Code:

  local body

if type(value) == 'boolean' or type(value) == 'number' then
  body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)
elseif type(value) == 'string' then
  body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value)
else
  log('invalid data type', addr, type(value))
  return
end

Currently this script sends values for all objects. There should be some filtering implemented so only the required data is pushed.

ok, now it's collecting data.... and yes, I need to filter them.

I guess I have to modify the function knx_callback(event), but my object addresses are not in a specific range, so I need to implement a sort of list of addresses to send.

[admin]

This example will push only objects that have influx tag attached.

Code:

local http = require('socket.http')
http.TIMEOUT = 5

local objs = {}
local tagobjs = grp.tag('influx')

for _, obj in ipairs(tagobjs) do
  objs[ obj.address ] = {
    name = obj.name:gsub(' ', '\\ '),
    datatype = obj.datatype,
  }
end

function callback(event)
  local addr = event.dst
  local obj = objs[ addr ]

  if obj then
    local value = knxdatatype.decode(event.datahex, obj.datatype)
    sendmetric('rawdata', obj.name, addr, value)
  end
end

function sendmetric(table, name, addr, value)
  local url = 'https://europe-west1-1.gcp.cloud2.influxdata.com/api/v2/write?bucket=XX&org=XXX'
  local body

  if type(value) == 'boolean' or type(value) == 'number' then
    body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)
  elseif type(value) == 'string' then
    body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value)
  else
    log('invalid data type', addr, type(value))
    return
  end

  local res, code = http.request({
    url = url,
    method = 'POST',
    body = body,
    headers = {
      Authorization = 'Token XXXXXXXXXXXXXXXXXXX'
    }
  })

  if code ~= 204 then
    log('error sending to influx', res, code, body)
  end
end

local busclient = require('localbus').new()
busclient:sethandler('groupwrite', callback)

while true do
  busclient:loop(1)
end

[gdimaria]

This example will push only objects that have influx tag attached.

Code:

local http = require('socket.http')
http.TIMEOUT = 5

local objs = {}
local tagobjs = grp.tag('influx')

for _, obj in ipairs(tagobjs) do
  objs[ obj.address ] = {
    name = obj.name:gsub(' ', '\\ '),
    datatype = obj.datatype,
  }
end

function callback(event)
  local addr = event.dst
  local obj = objs[ addr ]

  if obj then
    local value = knxdatatype.decode(event.datahex, obj.datatype)
    sendmetric('rawdata', obj.name, addr, value)
  end
end

function sendmetric(table, name, addr, value)
  local url = 'https://europe-west1-1.gcp.cloud2.influxdata.com/api/v2/write?bucket=XX&org=XXX'
  local body

  if type(value) == 'boolean' or type(value) == 'number' then
    body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)
  elseif type(value) == 'string' then
    body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value)
  else
    log('invalid data type', addr, type(value))
    return
  end

  local res, code = http.request({
    url = url,
    method = 'POST',
    body = body,
    headers = {
      Authorization = 'Token XXXXXXXXXXXXXXXXXXX'
    }
  })

  if code ~= 204 then
    log('error sending to influx', res, code, body)
  end
end

local busclient = require('localbus').new()
busclient:sethandler('groupwrite', callback)

while true do
  busclient:loop(1)
end

Everything is fine (although I still have to learn how to handle the data to create a decent report  )

Now  I have the final step: to connect influxdb to Grafana.

I don't know how to choose the right authentication metod to connect it. Where to put the token and org name??!

[admin]

If you switch to Flux query language then you can specify access token, org and bucket.
This might be helpful:
https://medium.com/@nanditasahu031/integ...b4aebb3368
https://blog.devgenius.io/grafana-influx...561e89b207