Trend clean

[domotiqa]

Hi,
I hae majority of my project, where I need to setup LM before all counter are installed (modbus, powertag....).
And when electrician installe it, I have my total script for category for example that 'got a peak.
   

So I have this kind of curve. I often export in xml, then re-import in the backup with the trend ID, but it's such a pain.

I was thinking to option:
1- reset the trend file to a blank one by script
2- I would prefer to create a script that monitor all my curve and if value goes above a certain max I setup, it clear it from the rdd file.
I don't know where to start to modifiy a value, is there existing base script I could modify ?

I would definitiv prefer second option.

Regards

[Daniel]

You probably use trend type 'counter' and you have negative delta, Use 'counter with negative delta' to not have such spikes.

[domotiqa]

You probably use trend type 'counter' and you have negative delta, Use 'counter with negative delta' to not have such spikes.

I Daniel,
The problem is not that in my caze. I understand what you mean.

I have script that add for example all light counter, all hvac, hall socket.... for the building.
But sometimes, the real counter is not cable.

so it s value is 0. When the electrician fix it,maybe one day or one week later. The Total increase a lot (the counter is not any more 0 but the index value from modbus).
So this point is not an error, however I want to delete it.

So I need to export in xml, find the value in the day and month storeage area, then export in rrd, then import my backup.

I'm looking for a way by script to find value that exeed for example XXXX (setup in the script), then replace it with zero).


This day I have a SE project with 100 counter at least, and for example, the water counter install in modbus are not well connected. The electrician will come next week to fix it (there is more than 10 area in this building).  When he will do that I will have an amount collected and a peak on the curve because the index is in reality already in count.

However my HMI is already build with all srcipt to do the TOTAL by category, area, type energy, M3....
Each HMI page have trend id in it and customer already use it. So very hard to generate again everything when all counter will be well wired.

[Daniel]

I don't think that this is possible, maybe admin will have some idea.

[domotiqa]

I was thinking:
1- Easiest way is to copy to the trend file in /tmp/trends an empty template
2- the best, a script that look for value above XXX, then replace it by value YYYY (I will calculate average of last day and then add same value)

any help appreciate

[admin]

You can simply delete the trend file then it will be re-created automatically.

Code:

123

id = 99 file = '/tmp/trends/t' .. id .. '.trend' os.remove(file)

Making a script that modifies internal trend values is possible but it's a complex task.

[domotiqa]

You can simply delete the trend file then it will be re-created automatically.

Code:

123

id = 99 file = '/tmp/trends/t' .. id .. '.trend' os.remove(file)

Making a script that modifies internal trend values is possible but it's a complex task.

thks admin and daniel.
This first solution will be cool.

Can you send some key I will dig in it for the second one ?

For the first solution, this scipt with your help do the job.
I would prefer the second solution where I don't need to put the name... but only a value max

Code:

1234567891011121314151617181920

tabCurveName= { 'ECLAIRAGE-TOTAL-KWH', 'GENERAL_ELEC-TOTAL-KWH' } for _, curveName in ipairs(tabCurveName) do     id = db:getone('SELECT id FROM trends WHERE name=?', curveName)     if id then     file = '/tmp/trends/t' .. id .. '.trend'     os.remove(file)   end   end script.disable(_SCRIPTNAME)

[domotiqa]

----------------


--------------

Hi Admin, can you give me example to find trend value greater than xxx and replace it by 0 for example ?
thks

[admin]

This example will change all valid trend values by x2. Recommended way of testing is to create a new trend log with the same parameters as the source.

Code:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950

infile = '/tmp/trends/t1.trend' outfile = '/tmp/trends/t7.trend' function adjust(value)   value = value * 2   return value end ffi = require('ffi') function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end function doubletostring(val)   local dbl = ffi.new('double[1]')   dbl[ 0 ] = val   return ffi.string(dbl, 8) end info = require('rrd').info(infile) data = io.readfile(infile) offset = info.header_size buf = { data:sub(1, offset) } while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       value = adjust(value)     end     buf[ #buf + 1 ] = doubletostring(value)   else     break   end   offset = offset + 8 end data = table.concat(buf) io.writefile(outfile, data)

For counter type the editing is trickier. The stored value is not the actual counter value but a per-second change rate. This example will log the maximum of all stored values without writing to a different trend log file. You can use this value for replacement in the previous script.

Code:

123456789101112131415161718192021222324252627282930313233343536373839

infile = '/tmp/trends/t1.trend' max = 0 function check(value)   max = math.max(max, value) end ffi = require('ffi') function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end info = require('rrd').info(infile) data = io.readfile(infile) offset = info.header_size while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       check(value)     end   else     break   end   offset = offset + 8 end log(max)

Note that when replacing values you should not use direct comparison (==) for floating point values due to possible rounding errors.

[domotiqa]

great thanks for helping, I will try to test in the next day.

[domotiqa]

great this work for me !! Your top

Code:

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103

ffi = require('ffi') typName=true typCat=false typTotal=false maxVal=4/60/60 function check(value)   max = math.max(max, value)   log(value)   if value > maxVal then     value=maxVal     --log(value)   end   return value end function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end function doubletostring(val)   local dbl = ffi.new('double[1]')   dbl[ 0 ] = val   return ffi.string(dbl, 8) end function wash(filename,maxVal)   max = 0   info = require('rrd').info(filename)   data = io.readfile(filename)   offset = info.header_size   buf = { data:sub(1, offset) }   while true do     chunk = data:sub(offset + 1, offset + 8)     if #chunk == 8 then       value = stringtodouble(chunk)       -- skip NaN (empty/undefined value)       if value == value then         value=check(value)       end             buf[ #buf + 1 ] = doubletostring(value)     else       break     end     offset = offset + 8   end   log('MAX:'..max)   data = table.concat(buf)   io.writefile(filename, data)   end if typName then   tabCurveName= {   --'ADLC-TOTAL-KWH',   --'TG2-TOTAL-KWH',   --'ECLAIRAGE-TOTAL-KWH',   --'GENERAL_ELEC-TOTAL-KWH',   --'ADLC-01_ECLAIRAGE-KWH-ECL',   --'CHAUFFAGE_CLIMATISATION_VENTILATION-TOTAL-KWH',   --'RECE-01_ECLAIRAGE-KWH-ECL'   --'ADN0-TOTAL-KWH',   --'RECE-TOTAL-KWH'     --'DIVERS-TOTAL-KWH',     --'EAU_CHAUDE-TOTAL-KWH',     --'PRISES-TOTAL-KWH',     --'TG1-TOTAL-KWH',     --'RECE-01_LOCAL_FROID-M3-FLU',     'ADN2-01_ECLAIRAGE-KWH-ECL'   }   for _, curveName in ipairs(tabCurveName) do         id = db:getone('SELECT id FROM trends WHERE name=?', curveName)     if id then       filename = '/tmp/trends/t' .. id .. '.trend'       --os.remove(file)       wash(filename,maxVal)     end                end end

[jmir]

This example will change all valid trend values by x2. Recommended way of testing is to create a new trend log with the same parameters as the source.

Code:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950

infile = '/tmp/trends/t1.trend' outfile = '/tmp/trends/t7.trend' function adjust(value)   value = value * 2   return value end ffi = require('ffi') function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end function doubletostring(val)   local dbl = ffi.new('double[1]')   dbl[ 0 ] = val   return ffi.string(dbl, 8) end info = require('rrd').info(infile) data = io.readfile(infile) offset = info.header_size buf = { data:sub(1, offset) } while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       value = adjust(value)     end     buf[ #buf + 1 ] = doubletostring(value)   else     break   end   offset = offset + 8 end data = table.concat(buf) io.writefile(outfile, data)

For counter type the editing is trickier. The stored value is not the actual counter value but a per-second change rate. This example will log the maximum of all stored values without writing to a different trend log file. You can use this value for replacement in the previous script.

Code:

123456789101112131415161718192021222324252627282930313233343536373839

infile = '/tmp/trends/t1.trend' max = 0 function check(value)   max = math.max(max, value) end ffi = require('ffi') function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end info = require('rrd').info(infile) data = io.readfile(infile) offset = info.header_size while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       check(value)     end   else     break   end   offset = offset + 8 end log(max)

Note that when replacing values you should not use direct comparison (==) for floating point values due to possible rounding errors.

Hi, 
I've used this scripts to replace erroneus values (due to power losses...) on counter type trends, but I've seen that daily grouped values are still wrong.
Are this values calculated from hourly values? Or are stored? Is there any way to update them?

[admin]

This script goes over all values including daily data.
If you only want to modify daily data then modify the script at line 20:

Code:

1

offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8

[jmir]

Hi,
When I use the script to find max value, it's no showing daily max value... I managed to change a wrong value at 23:00 on 30 May but daily values are still wrong...

[admin]

This will log all daily values with their respective offsets. If the whole list does not fit, you can change line 17 and add an extra offset. Just make sure that this extra offset is divisible by 8.

Code:

12345678910111213141516171819202122232425262728293031323334353637

infile = '/tmp/trends/t1.trend' values = {} ffi = require('ffi') function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end info = require('rrd').info(infile) data = io.readfile(infile) offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8 offset = offset + 0 * 8 while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       values[ #values + 1 ] = string.format('%d %.1f', offset, value)     end   else     break   end   offset = offset + 8 end text = table.concat(values, '\n') log(text)

[jmir]

This will log all daily values with their respective offsets. If the whole list does not fit, you can change line 17 and add an extra offset. Just make sure that this extra offset is divisible by 8.

Code:

12345678910111213141516171819202122232425262728293031323334353637

infile = '/tmp/trends/t1.trend' values = {} ffi = require('ffi') function stringtodouble(chunk)   local dbl = ffi.new('double[1]')   ffi.copy(dbl, chunk, 8)   return dbl[ 0 ] end info = require('rrd').info(infile) data = io.readfile(infile) offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8 offset = offset + 0 * 8 while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       values[ #values + 1 ] = string.format('%d %.1f', offset, value)     end   else     break   end   offset = offset + 8 end text = table.concat(values, '\n') log(text)

Ok, using this one I've managed to update wrong values. Thanks!

[domotiqa]

How can I display the date of the value (where I put log(date) ).
Don't find it

Code:

1234567891011121314151617

while true do   chunk = data:sub(offset + 1, offset + 8)   if #chunk == 8 then     value = stringtodouble(chunk)     -- skip NaN (empty/undefined value)     if value == value then       values[ #values + 1 ] = string.format('%d %.1f', offset, value)       log(date)     end   else     break   end   offset = offset + 8 end

[admin]

If you need dates then use trends.fetch function. Calculating the date using raw file access is not trivial because the data is written in a circular way.

[domotiqa]

thanks. Yes I know with fetch,
I was looking for display the date of the current record int he script above

In rrd we have it at the begining I thought:

<!-- 2024-01-17 17:07:00    / 1705507620 --> <row><v> 1.6666666667e-02 </v></row>

It would be cool in order to match the date we want to modify the value for...