Storage is pushed on localbus

[Smeagol]

Hello,

I have a resident script with interval 0 that is reading out a 'smart' electrical meter (dutch). The meter pushes a new telegram out every second. Which gets parsed and processed by the script. First question, what does the interval 0 mean? Is it run continiously? Does this have negative effects? It seems to work fine... 

To save data between script iterations I use the storage. I save quite a lot of parameters to the storage since the data requires some processing to be useable in trends. I have noticed that this in turn results in a 'heavy' load of the websocket (localbus) since every set storage is pushed out the socket. Is there a way to prevent this? What is the best way to save data between iterations? I can think of one improvement and this is to try and save data in a table instead of separate parameters but this still will be pushed trough the localbus...

The script below:

Code:

lastchar = '!'

require('serial')
port = serial.open('/dev/ttyUSB0', { -- Let op type van de slimme meter voor de onderstaande instellingen.
  baudrate = 115200,             --DSMR 2/3 = 9600        DSMR 4 = 115200
  databits = 8,                    --DSMR 2/3 = 7            DSMR 4 = 8
  stopbits = 1,                    --DSMR 2/3 = 1            DSMR 4 = 1
  parity = 'none',            --DSMR 2/3 = even        DSMR 4 = none
  duplex = 'full'
})

function readdata(port)
  local char, buf

  buf = {}

  while true do
    char = port:read(1)
    -- error (timeout) or newline, stop
    if char == nil or char == lastchar then
      break
    -- ignore cr char
    elseif char ~= '\r' then
      table.insert(buf, char)
    end
  end

  return table.concat(buf)
end

function splitString(s, delimiter)
    local result = {};
    for match in string.gmatch(s, "([^"..delimiter.."]+)") do            
        table.insert(result, match);
    end
    return result;
end

function parseTelegram(data)     
    local parsed = {} 
  for line in data:gmatch("([^\n]*)\n?") do
    local pristineLine = string.gsub(string.sub(line, 5), "%)", "")    
    local keyValue = splitString(pristineLine, "%(")
    if (keyValue[1]) then
        parsed[keyValue[1]] = keyValue[2]
    end
    end 
  return parsed
end

data = readdata(port)
port:flush()

---------------------------------------------------------------

dailyConsumption = storage.get('dailyConsumption')
dailyInjection = storage.get('dailyInjection')
intervalConsumption = storage.get('intervalConsumption')
intervalInjection = storage.get('intervalInjection')
intervalPoints = storage.get('intervalPoints')

baseDayConsumption = storage.get('baseDayConsumption')
baseDayInjection = storage.get('baseDayInjection')
baseNightConsumption = storage.get('baseNightConsumption')
baseNightInjection = storage.get('baseNightInjection')

parsed = parseTelegram(data)

totalDayConsumption = tonumber(splitString(parsed["1.8.1"], "%*")[1])
totalDayInjection = tonumber(splitString(parsed["2.8.1"], "%*")[1])
totalNightConsumption = tonumber(splitString(parsed["1.8.2"], "%*")[1])
totalNightInjection = tonumber(splitString(parsed["2.8.2"], "%*")[1])

currentConsumption = tonumber(splitString(parsed["1.7.0"], "%*")[1])
currentInjection = tonumber(splitString(parsed["2.7.0"], "%*")[1])

now = os.date('*t')

if ((not dailyConsumption)) then
  dailyConsumption = 0
end

if ((not dailyInjection)) then
  dailyInjection = 0
end

if (not intervalConsumption) then
  intervalConsumption = currentConsumption
end

if (not intervalInjection) then
  intervalInjection = currentInjection
end

if (not intervalPoints) then
  intervalPoints = 0
end

intervalPoints = intervalPoints + 1

if ((not baseDayConsumption)) then 
  baseDayConsumption = totalDayConsumption
  storage.set('baseDayConsumption', baseDayConsumption)
end

if ((not baseDayInjection)) then
  baseDayInjection = totalDayInjection
  storage.set('baseDayInjection', baseDayInjection)
end

if ((not baseNightConsumption)) then
  baseNightConsumption = totalNightConsumption
  storage.set('baseNightConsumption', baseNightConsumption)
end

if ((not baseNightInjection)) then
  baseNightInjection = totalNightInjection
  storage.set('baseNightInjection', baseNightInjection)
end

currentConsumtionPhase1 = tonumber(splitString(parsed["21.7.0"], "%*")[1])
currentConsumtionPhase2 = tonumber(splitString(parsed["41.7.0"], "%*")[1])
currentConsumtionPhase3 = tonumber(splitString(parsed["61.7.0"], "%*")[1])

currentInjectionPhase1 = tonumber(splitString(parsed["22.7.0"], "%*")[1])
currentInjectionPhase2 = tonumber(splitString(parsed["42.7.0"], "%*")[1])
currentInjectionPhase3 = tonumber(splitString(parsed["62.7.0"], "%*")[1])

intervalConsumption = intervalConsumption + currentConsumption
intervalInjection = intervalInjection + currentInjection

dailyConsumption = dailyConsumption + (currentConsumption * (1/3600))
storage.set('dailyConsumption', dailyConsumption)

dailyInjection = dailyInjection + (currentInjection * (1/3600))
storage.set('dailyInjection', dailyInjection)

if (math.fmod(now["min"], 5) == 0 and now["sec"] == 0) then  
  grp.write('x/x/x', dailyConsumption)
  grp.write('x/x/x', dailyInjection)
  grp.write('x/x/x', intervalConsumption / intervalPoints)
  grp.write('x/x/x', intervalInjection / intervalPoints)
  grp.write('x/x/x', currentConsumtionPhase1)
  grp.write('x/x/x', currentConsumtionPhase2)
  grp.write('x/x/x', currentConsumtionPhase3)
 
  grp.write('x/x/x', currentInjectionPhase1)
  grp.write('x/x/x', currentInjectionPhase2)
  grp.write('x/x/x', currentInjectionPhase3)

  storage.set('intervalConsumption', nil)
  storage.set('intervalInjection', nil)
  storage.set('intervalPoints', nil)
else
    storage.set('intervalConsumption', intervalConsumption)
  storage.set('intervalInjection', intervalInjection)
  storage.set('intervalPoints', intervalPoints)
end

if (now["min"] == 0 and now["sec"] == 0) then
    grp.write('32/6/8', totalDayConsumption - baseDayConsumption)
  grp.write('x/x/x', totalDayInjection - baseDayInjection)
  grp.write('x/x/x', totalNightConsumption - baseNightConsumption)
  grp.write('x/x/x', totalNightInjection - baseNightInjection)
end

-- At the end of the day after all was set, reset the values for the next day

if ((now["hour"] == 0 and now["min"] == 0 and now["sec"] == 0)) then
  dailyConsumption = 0
  grp.write('x/x/x', dailyConsumption)
  storage.set('dailyConsumption', dailyConsumption)
end

if ((now["hour"] == 0 and now["min"] == 0 and now["sec"] == 0)) then
  dailyInjection = 0
  grp.write('x/x/x', dailyInjection)
  storage.set('dailyInjection', dailyInjection)
end

if ((now["hour"] == 0 and now["min"] == 0 and now["sec"] == 0)) then
  log('reset consumption for day')
  baseDayConsumption = totalDayConsumption
  grp.write('x/x/x', totalDayConsumption - baseDayConsumption)
  storage.set('baseDayConsumption', baseDayConsumption)
end

if ((now["hour"] == 0 and now["min"] == 0 and now["sec"] == 0)) then
  baseDayInjection = totalDayInjection
  grp.write('x/x/x', totalDayInjection - baseDayInjection)
  storage.set('baseDayInjection', baseDayInjection)
end

if ((now["hour"] == 0 and now["min"] == 0 and now["sec"] == 0)) then
  baseNightConsumption = totalNightConsumption
  grp.write('x/x/x', totalNightConsumption - baseNightConsumption)
  storage.set('baseNightConsumption', baseNightConsumption)
end

if ((now["hour"] == 0 and now["min"] == 0 and now["sec"] == 0)) then
  baseNightInjection = totalNightInjection
  grp.write('x/x/x', totalNightInjection - baseNightInjection)
  storage.set('baseNightInjection', baseNightInjection)
end

Any advise would be appreciated!

Kind regards

[Erwin van der Zwart]

Hi,

The resident to 0 is no problem, the socket waits for new chars and the smart meter is pushing out data every 10 seconds so the socket delays the execution of the script until new chars are received.

Some questions: 

1) Without having to "understand" what you have build can you eleborate on what your final goal is?
2) What do you mean with "Every set storage is send out of the socket" ? The meter is read only, and localbus is not in your script so i don't understand that question.

BR,

Erwin

PS: Ben je een Nederlandse gebruiker van de Wiser for KNX?

[Smeagol]

Hi,

The resident to 0 is no problem, the socket waits for new chars and the smart meter is pushing out data every 10 seconds so the socket delays the execution of the script until new chars are received.

Some questions: 

1) Without having to "understand" what you have build can you eleborate on what your final goal is?
2) What do you mean with "Every set storage is send out of the socket" ? The meter is read only, and localbus is not in your script so i don't understand that question.

BR,

Erwin

PS: Ben je een Nederlandse gebruiker van de Wiser for KNX?

Hello,

Thank you for answering! The smart meter is pushing data out every second instead of every 10 seconds but i suppose given your explanation that that does not matter... As long as there is some delay it won't eat up the CPU.

The script i shared works fine and does what i require it to do:

• Segment the data and extract the usefull parts
  
• At the start of a new day (00:00:00) store the meter values (day consumtion, night consumption, day injection and night injection) in memory and for the rest of the day substract these value from the new meter values. This gives you your daily consumption and injection. I wirte this to a virtual address every hour so i can use it in a trend.
  
• For more fine grained trends i use the actual power consumption and injection values of the meter and calculate a 'running' discrete integral with them (so at the end of the day you once again have your daily consumption) and push this on a virtual address every 5 minuts for use in detail trends. I also do this for the power consumtion of each phase.
  

Since the script runs every ~1 seconds and i need data between iteration of the script, the storing of values is done with 'storage.set'. Again this works fine. My main problem, however, is that i noticed that when you use an app like Mosaic (or a custom app) that relies on the localbus.js.gz library, the websocket the library esthablishes with the LM comes under 'heavy' load since all the storage.set method calls trigger an update from the LM to the app via the websocket. 

So in short storage.set triggers a websocket message with it's new value to apps. Since i use storage.set quite heavily and frequently (~20 x every second) i fear for the load on the websocket. The data that i require to store for this script is also only important for this script and should not be on the websocket. I assume however that it is normal for storage to be shared with the apps. I am wondering however if there is a better way of storing data between script iteration or at the very least to find a way to limit it?

Thanks for the advise!

Kind regards

P.S. Ik ben een Belgische gebruiker van een Logic Machine (LM5 Lite) en heb geen Wiser 

[admin]

All global variables in resident scripts are retained after each loop. Only script restart clears these variables. So I'm not sure whether storage is needed at all. Alternatively you can add app: prefix to the storage key. Such keys are not sent over localbus. For example:

Code:

storage.set('app:dailyInjection', dailyInjection)

[Bitver]

All global variables in resident scripts are retained after each loop. Only script restart clears these variables. So I'm not sure whether storage is needed at all. Alternatively you can add app: prefix to the storage key. Such keys are sent over localbus. For example:

Code:

storage.set('app:dailyInjection', dailyInjection)

You mean "Such keys are not sent over localbus"? because they're not 

[admin]

Yes, they are not sent. There was a typo in my post

[Smeagol]

Yes, they are not sent. There was a typo in my post

Thank you! If global variable state is retained between iteration, then indeed i do not even need storage.