massive data logging

[gdimaria]

Hi, I need to trend log about 250/300 KNX objects. 

I know that's over LM possibilities due the 21 Mb archive limit.

Can you suggest an appropriate device to do this job?

I already knows BAB TECHNOLOGIE GmbH DATALOGGER V2 (https://bab-technologie.com/datalogger-v2/?lang=en), it could be ok.

But I trust in your experience and I would receive a suggest.

Thanks

Peppe

[admin]

With low resolution / retention settings you could fit 300 trends into LM. An alternative solution is to use Grafana - either cloud-hosted or by installing it locally on a small PC. Installation can be a bit tricky but it's a very flexible solution for storing time-series data.

[gdimaria]

Grafana seems to be very cool! And how to connect LM to it, sending telegrams via remote access?

[admin]

LM is sending data to InfluxDB using HTTP. This thread has working examples: https://forum.logicmachine.net/showthread.php?tid=1531
Another option is MQTT: https://grafana.com/blog/2021/08/12/stre...fana-live/

[gdimaria]

LM is sending data to InfluxDB using HTTP. This thread has working examples: https://forum.logicmachine.net/showthread.php?tid=1531
Another option is MQTT: https://grafana.com/blog/2021/08/12/stre...fana-live/

Things are little changed on InfluxDB:

https://docs.influxdata.com/influxdb/clo...fluxdb-api

it seems a little too complicated to me... no shorter ways to implement a massive data logging?

Peppe

[admin]

You only need to create a bucket once. It will store your data for a certain amount of time. There are examples of sending data to influx v2 in the topic I've linked to.
The setup part is a bit complicated but you only have to do it once

[gdimaria]

You only need to create a bucket once. It will store your data for a certain amount of time. There are examples of sending data to influx v2 in the topic I've linked to.
The setup part is a bit complicated but you only have to do it once

I accepted the challenge!
But right now I get error messages:


* arg: 1
  * string: error sending to influx
* arg: 2
  * string: {"code":"invalid","message":"at line 1:36: cannot parse value for field key \"value\": invalid float value syntax (check rejected_points in your _monitoring bucket for further information)"}
* arg: 3
  * number: 400
* arg: 4
  * string: rawdata,name=ora,addr=33/1/9 value=10:57


and


* arg: 1
  * string: error sending to influx
* arg: 2
  * string: {"code":"invalid","message":"at line 1:47: cannot parse value for field key \"value\": invalid bool value \"Thu\" (check rejected_points in your _monitoring bucket for further information)"}
* arg: 3
  * number: 400
* arg: 4
  * string: rawdata,name=giornomeseanno,addr=33/1/8 value=Thu 15-09-2022


the script I used is the following, can you help me?

Code:

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172

local socket = require('socket') local http = require('socket.http') http.TIMEOUT = 5 local dt_cache = {} function get_value(addr, datahex)   local dt = dt_cache[addr]   if(not dt) then     dt = grp.find(addr)     dt_cache[addr] = dt   end     return knxdatatype.decode(datahex, dt.datatype), dt end function knx_callback(event)   local addr = event.dst   local value, dt = get_value(addr, event.datahex)   send_metric('rawdata', dt.name, addr, value) end function send_metric(table, name, addr, value)   if name == nil or name == '' then     return   end   name = string.gsub(name, ' ', '\\ ')   local url = 'https://europe-west1-1.gcp.cloud2.influxdata.com/api/v2/write?bucket=XX&org=XXX'   local body   if type(value) == 'boolean' then     body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)   else     -- most likely number     body = string.format('%s,name=%s,addr=%s value=%s', table, name, addr, value)   end   local res, code = http.request({     url = url,     method = 'POST',     body = body,     headers = {       Authorization = 'Token XXXXXXXXXXXXXXXXXXX'     }   })   if code ~= 204 then     log('error sending to influx', res, code, body)   end end function run_loop()   local bus = require('localbus').new(1)   bus:sethandler('groupwrite', knx_callback)   local busfd = socket.fdmaskset(bus:getfd(), 'r')     while(true) do     res, fbus = socket.selectfds(10, busfd)     if(fbus) then       bus:step()     end   end end run_loop()

[admin]

You're sending string data (10:57 and Thu 15-09-2022) which Influx does not understand. String values must be enclosed in quotes.
Try replacing lines 35..42 with this:

Code:

12345678910

local body if type(value) == 'boolean' or type(value) == 'number' then   body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value) elseif type(value) == 'string' then   body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value) else   log('invalid data type', addr, type(value))   return end

Currently this script sends values for all objects. There should be some filtering implemented so only the required data is pushed.

[gdimaria]

You're sending string data (10:57 and Thu 15-09-2022) which Influx does not understand. String values must be enclosed in quotes.
Try replacing lines 35..42 with this:

Code:

12345678910

  local body if type(value) == 'boolean' or type(value) == 'number' then   body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value) elseif type(value) == 'string' then   body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value) else   log('invalid data type', addr, type(value))   return end

Currently this script sends values for all objects. There should be some filtering implemented so only the required data is pushed.

ok, now it's collecting data.... and yes, I need to filter them.

I guess I have to modify the function knx_callback(event), but my object addresses are not in a specific range, so I need to implement a sort of list of addresses to send.

[admin]

This example will push only objects that have influx tag attached.

Code:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556

local http = require('socket.http') http.TIMEOUT = 5 local objs = {} local tagobjs = grp.tag('influx') for _, obj in ipairs(tagobjs) do   objs[ obj.address ] = {     name = obj.name:gsub(' ', '\\ '),     datatype = obj.datatype,   } end function callback(event)   local addr = event.dst   local obj = objs[ addr ]   if obj then     local value = knxdatatype.decode(event.datahex, obj.datatype)     sendmetric('rawdata', obj.name, addr, value)   end end function sendmetric(table, name, addr, value)   local url = 'https://europe-west1-1.gcp.cloud2.influxdata.com/api/v2/write?bucket=XX&org=XXX'   local body   if type(value) == 'boolean' or type(value) == 'number' then     body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)   elseif type(value) == 'string' then     body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value)   else     log('invalid data type', addr, type(value))     return   end   local res, code = http.request({     url = url,     method = 'POST',     body = body,     headers = {       Authorization = 'Token XXXXXXXXXXXXXXXXXXX'     }   })   if code ~= 204 then     log('error sending to influx', res, code, body)   end end local busclient = require('localbus').new() busclient:sethandler('groupwrite', callback) while true do   busclient:loop(1) end

[gdimaria]

This example will push only objects that have influx tag attached.

Code:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556

local http = require('socket.http') http.TIMEOUT = 5 local objs = {} local tagobjs = grp.tag('influx') for _, obj in ipairs(tagobjs) do   objs[ obj.address ] = {     name = obj.name:gsub(' ', '\\ '),     datatype = obj.datatype,   } end function callback(event)   local addr = event.dst   local obj = objs[ addr ]   if obj then     local value = knxdatatype.decode(event.datahex, obj.datatype)     sendmetric('rawdata', obj.name, addr, value)   end end function sendmetric(table, name, addr, value)   local url = 'https://europe-west1-1.gcp.cloud2.influxdata.com/api/v2/write?bucket=XX&org=XXX'   local body   if type(value) == 'boolean' or type(value) == 'number' then     body = string.format('%s,name=%s,addr=%s state=%s', table, name, addr, value)   elseif type(value) == 'string' then     body = string.format('%s,name=%s,addr=%s value=%q', table, name, addr, value)   else     log('invalid data type', addr, type(value))     return   end   local res, code = http.request({     url = url,     method = 'POST',     body = body,     headers = {       Authorization = 'Token XXXXXXXXXXXXXXXXXXX'     }   })   if code ~= 204 then     log('error sending to influx', res, code, body)   end end local busclient = require('localbus').new() busclient:sethandler('groupwrite', callback) while true do   busclient:loop(1) end

Everything is fine (although I still have to learn how to handle the data to create a decent report  )

Now  I have the final step: to connect influxdb to Grafana.

I don't know how to choose the right authentication metod to connect it. Where to put the token and org name??!

[admin]

If you switch to Flux query language then you can specify access token, org and bucket.
This might be helpful:
https://medium.com/@nanditasahu031/integ...b4aebb3368
https://blog.devgenius.io/grafana-influx...561e89b207