Trend clean

[domotiqa]

Hi,
I hae majority of my project, where I need to setup LM before all counter are installed (modbus, powertag....).
And when electrician installe it, I have my total script for category for example that 'got a peak.
   

So I have this kind of curve. I often export in xml, then re-import in the backup with the trend ID, but it's such a pain.

I was thinking to option:
1- reset the trend file to a blank one by script
2- I would prefer to create a script that monitor all my curve and if value goes above a certain max I setup, it clear it from the rdd file.
I don't know where to start to modifiy a value, is there existing base script I could modify ?

I would definitiv prefer second option.

Regards

[Daniel]

You probably use trend type 'counter' and you have negative delta, Use 'counter with negative delta' to not have such spikes.

[domotiqa]

You probably use trend type 'counter' and you have negative delta, Use 'counter with negative delta' to not have such spikes.

I Daniel,
The problem is not that in my caze. I understand what you mean.

I have script that add for example all light counter, all hvac, hall socket.... for the building.
But sometimes, the real counter is not cable.

so it s value is 0. When the electrician fix it,maybe one day or one week later. The Total increase a lot (the counter is not any more 0 but the index value from modbus).
So this point is not an error, however I want to delete it.

So I need to export in xml, find the value in the day and month storeage area, then export in rrd, then import my backup.

I'm looking for a way by script to find value that exeed for example XXXX (setup in the script), then replace it with zero).


This day I have a SE project with 100 counter at least, and for example, the water counter install in modbus are not well connected. The electrician will come next week to fix it (there is more than 10 area in this building).  When he will do that I will have an amount collected and a peak on the curve because the index is in reality already in count.

However my HMI is already build with all srcipt to do the TOTAL by category, area, type energy, M3....
Each HMI page have trend id in it and customer already use it. So very hard to generate again everything when all counter will be well wired.

[Daniel]

I don't think that this is possible, maybe admin will have some idea.

[domotiqa]

I was thinking:
1- Easiest way is to copy to the trend file in /tmp/trends an empty template
2- the best, a script that look for value above XXX, then replace it by value YYYY (I will calculate average of last day and then add same value)

any help appreciate

[admin]

You can simply delete the trend file then it will be re-created automatically.

Code:

id = 99
file = '/tmp/trends/t' .. id .. '.trend'
os.remove(file)

Making a script that modifies internal trend values is possible but it's a complex task.

[domotiqa]

You can simply delete the trend file then it will be re-created automatically.

Code:

id = 99
file = '/tmp/trends/t' .. id .. '.trend'
os.remove(file)

Making a script that modifies internal trend values is possible but it's a complex task.

thks admin and daniel.
This first solution will be cool.

Can you send some key I will dig in it for the second one ?

For the first solution, this scipt with your help do the job.
I would prefer the second solution where I don't need to put the name... but only a value max

Code:

tabCurveName= {
'ECLAIRAGE-TOTAL-KWH',
'GENERAL_ELEC-TOTAL-KWH'
}

for _, curveName in ipairs(tabCurveName) do
 
  id = db:getone('SELECT id FROM trends WHERE name=?', curveName)
 
  if id then
    file = '/tmp/trends/t' .. id .. '.trend'
    os.remove(file)
  end
 
end




script.disable(_SCRIPTNAME)

[domotiqa]

----------------


--------------

Hi Admin, can you give me example to find trend value greater than xxx and replace it by 0 for example ?
thks

[admin]

This example will change all valid trend values by x2. Recommended way of testing is to create a new trend log with the same parameters as the source.

Code:

infile = '/tmp/trends/t1.trend'
outfile = '/tmp/trends/t7.trend'

function adjust(value)
  value = value * 2
  return value
end

ffi = require('ffi')

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

function doubletostring(val)
  local dbl = ffi.new('double[1]')
  dbl[ 0 ] = val
  return ffi.string(dbl, 8)
end

info = require('rrd').info(infile)

data = io.readfile(infile)
offset = info.header_size

buf = { data:sub(1, offset) }

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      value = adjust(value)
    end

    buf[ #buf + 1 ] = doubletostring(value)
  else
    break
  end

  offset = offset + 8
end

data = table.concat(buf)
io.writefile(outfile, data)

For counter type the editing is trickier. The stored value is not the actual counter value but a per-second change rate. This example will log the maximum of all stored values without writing to a different trend log file. You can use this value for replacement in the previous script.

Code:

infile = '/tmp/trends/t1.trend'

max = 0

function check(value)
  max = math.max(max, value)
end

ffi = require('ffi')

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

info = require('rrd').info(infile)

data = io.readfile(infile)
offset = info.header_size

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      check(value)
    end
  else
    break
  end

  offset = offset + 8
end

log(max)

Note that when replacing values you should not use direct comparison (==) for floating point values due to possible rounding errors.

[domotiqa]

great thanks for helping, I will try to test in the next day.

[domotiqa]

great this work for me !! Your top

Code:

ffi = require('ffi')

typName=true
typCat=false
typTotal=false
maxVal=4/60/60


function check(value)
  max = math.max(max, value)
  log(value)
  if value > maxVal then
    value=maxVal
    --log(value)
  end
  return value
end

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

function doubletostring(val)
  local dbl = ffi.new('double[1]')
  dbl[ 0 ] = val
  return ffi.string(dbl, 8)
end

function wash(filename,maxVal)
  max = 0
  info = require('rrd').info(filename)

  data = io.readfile(filename)
  offset = info.header_size

  buf = { data:sub(1, offset) }

  while true do
    chunk = data:sub(offset + 1, offset + 8)

    if #chunk == 8 then
      value = stringtodouble(chunk)

      -- skip NaN (empty/undefined value)
      if value == value then
        value=check(value)
      end
     

      buf[ #buf + 1 ] = doubletostring(value)
    else
      break
    end

    offset = offset + 8
  end

  log('MAX:'..max)
  data = table.concat(buf)
  io.writefile(filename, data)
 
end






if typName then
  tabCurveName= {
  --'ADLC-TOTAL-KWH',
  --'TG2-TOTAL-KWH',
  --'ECLAIRAGE-TOTAL-KWH',
  --'GENERAL_ELEC-TOTAL-KWH',
  --'ADLC-01_ECLAIRAGE-KWH-ECL',
  --'CHAUFFAGE_CLIMATISATION_VENTILATION-TOTAL-KWH',
  --'RECE-01_ECLAIRAGE-KWH-ECL'
  --'ADN0-TOTAL-KWH',
  --'RECE-TOTAL-KWH'
    --'DIVERS-TOTAL-KWH',
    --'EAU_CHAUDE-TOTAL-KWH',
    --'PRISES-TOTAL-KWH',
    --'TG1-TOTAL-KWH',
    --'RECE-01_LOCAL_FROID-M3-FLU',
    'ADN2-01_ECLAIRAGE-KWH-ECL'
  }

  for _, curveName in ipairs(tabCurveName) do

        id = db:getone('SELECT id FROM trends WHERE name=?', curveName)

    if id then
      filename = '/tmp/trends/t' .. id .. '.trend'
      --os.remove(file)
      wash(filename,maxVal)
    end

   
        
  end
end

[jmir]

This example will change all valid trend values by x2. Recommended way of testing is to create a new trend log with the same parameters as the source.

Code:

infile = '/tmp/trends/t1.trend'
outfile = '/tmp/trends/t7.trend'

function adjust(value)
  value = value * 2
  return value
end

ffi = require('ffi')

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

function doubletostring(val)
  local dbl = ffi.new('double[1]')
  dbl[ 0 ] = val
  return ffi.string(dbl, 8)
end

info = require('rrd').info(infile)

data = io.readfile(infile)
offset = info.header_size

buf = { data:sub(1, offset) }

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      value = adjust(value)
    end

    buf[ #buf + 1 ] = doubletostring(value)
  else
    break
  end

  offset = offset + 8
end

data = table.concat(buf)
io.writefile(outfile, data)

For counter type the editing is trickier. The stored value is not the actual counter value but a per-second change rate. This example will log the maximum of all stored values without writing to a different trend log file. You can use this value for replacement in the previous script.

Code:

infile = '/tmp/trends/t1.trend'

max = 0

function check(value)
  max = math.max(max, value)
end

ffi = require('ffi')

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

info = require('rrd').info(infile)

data = io.readfile(infile)
offset = info.header_size

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      check(value)
    end
  else
    break
  end

  offset = offset + 8
end

log(max)

Note that when replacing values you should not use direct comparison (==) for floating point values due to possible rounding errors.

Hi, 
I've used this scripts to replace erroneus values (due to power losses...) on counter type trends, but I've seen that daily grouped values are still wrong.
Are this values calculated from hourly values? Or are stored? Is there any way to update them?

[admin]

This script goes over all values including daily data.
If you only want to modify daily data then modify the script at line 20:

Code:

offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8

[jmir]

Hi,
When I use the script to find max value, it's no showing daily max value... I managed to change a wrong value at 23:00 on 30 May but daily values are still wrong...

[admin]

This will log all daily values with their respective offsets. If the whole list does not fit, you can change line 17 and add an extra offset. Just make sure that this extra offset is divisible by 8.

Code:

infile = '/tmp/trends/t1.trend'
values = {}

ffi = require('ffi')

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

info = require('rrd').info(infile)

data = io.readfile(infile)
offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8

offset = offset + 0 * 8

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      values[ #values + 1 ] = string.format('%d %.1f', offset, value)
    end
  else
    break
  end

  offset = offset + 8
end

text = table.concat(values, '\n')
log(text)

[jmir]

This will log all daily values with their respective offsets. If the whole list does not fit, you can change line 17 and add an extra offset. Just make sure that this extra offset is divisible by 8.

Code:

infile = '/tmp/trends/t1.trend'
values = {}

ffi = require('ffi')

function stringtodouble(chunk)
  local dbl = ffi.new('double[1]')
  ffi.copy(dbl, chunk, 8)
  return dbl[ 0 ]
end

info = require('rrd').info(infile)

data = io.readfile(infile)
offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8

offset = offset + 0 * 8

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      values[ #values + 1 ] = string.format('%d %.1f', offset, value)
    end
  else
    break
  end

  offset = offset + 8
end

text = table.concat(values, '\n')
log(text)

Ok, using this one I've managed to update wrong values. Thanks!

[domotiqa]

How can I display the date of the value (where I put log(date) ).
Don't find it

Code:

while true do
  chunk = data:sub(offset + 1, offset + 8)

  if #chunk == 8 then
    value = stringtodouble(chunk)

    -- skip NaN (empty/undefined value)
    if value == value then
      values[ #values + 1 ] = string.format('%d %.1f', offset, value)
      log(date)
    end
  else
    break
  end

  offset = offset + 8
end

[admin]

If you need dates then use trends.fetch function. Calculating the date using raw file access is not trivial because the data is written in a circular way.

[domotiqa]

thanks. Yes I know with fetch,
I was looking for display the date of the current record int he script above

In rrd we have it at the begining I thought:

<!-- 2024-01-17 17:07:00    / 1705507620 --> <row><v> 1.6666666667e-02 </v></row>

It would be cool in order to match the date we want to modify the value for...