Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
Hi,
I hae majority of my project, where I need to setup LM before all counter are installed (modbus, powertag....).
And when electrician installe it, I have my total script for category for example that 'got a peak.
So I have this kind of curve. I often export in xml, then re-import in the backup with the trend ID, but it's such a pain.
I was thinking to option:
1- reset the trend file to a blank one by script
2- I would prefer to create a script that monitor all my curve and if value goes above a certain max I setup, it clear it from the rdd file.
I don't know where to start to modifiy a value, is there existing base script I could modify ?
I would definitiv prefer second option.
Regards
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 4479
Threads: 22
Joined: Aug 2017
Reputation:
200
You probably use trend type 'counter' and you have negative delta, Use 'counter with negative delta' to not have such spikes.
------------------------------
Ctrl+F5
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
22.11.2023, 08:51
(This post was last modified: 22.11.2023, 08:56 by domotiqa.)
(22.11.2023, 08:42)Daniel Wrote: You probably use trend type 'counter' and you have negative delta, Use 'counter with negative delta' to not have such spikes.
I Daniel,
The problem is not that in my caze. I understand what you mean.
I have script that add for example all light counter, all hvac, hall socket.... for the building.
But sometimes, the real counter is not cable.
so it s value is 0. When the electrician fix it,maybe one day or one week later. The Total increase a lot (the counter is not any more 0 but the index value from modbus).
So this point is not an error, however I want to delete it.
So I need to export in xml, find the value in the day and month storeage area, then export in rrd, then import my backup.
I'm looking for a way by script to find value that exeed for example XXXX (setup in the script), then replace it with zero).
This day I have a SE project with 100 counter at least, and for example, the water counter install in modbus are not well connected. The electrician will come next week to fix it (there is more than 10 area in this building). When he will do that I will have an amount collected and a peak on the curve because the index is in reality already in count.
However my HMI is already build with all srcipt to do the TOTAL by category, area, type energy, M3....
Each HMI page have trend id in it and customer already use it. So very hard to generate again everything when all counter will be well wired.
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 4479
Threads: 22
Joined: Aug 2017
Reputation:
200
I don't think that this is possible, maybe admin will have some idea.
------------------------------
Ctrl+F5
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
I was thinking:
1- Easiest way is to copy to the trend file in /tmp/trends an empty template
2- the best, a script that look for value above XXX, then replace it by value YYYY (I will calculate average of last day and then add same value)
any help appreciate
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 7660
Threads: 41
Joined: Jun 2015
Reputation:
441
You can simply delete the trend file then it will be re-created automatically.
Code: id = 99
file = '/tmp/trends/t' .. id .. '.trend'
os.remove(file)
Making a script that modifies internal trend values is possible but it's a complex task.
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
22.11.2023, 09:25
(This post was last modified: 22.11.2023, 09:46 by domotiqa.)
(22.11.2023, 09:01)admin Wrote: You can simply delete the trend file then it will be re-created automatically.
Code: id = 99
file = '/tmp/trends/t' .. id .. '.trend'
os.remove(file)
Making a script that modifies internal trend values is possible but it's a complex task.
thks admin and daniel.
This first solution will be cool.
Can you send some key I will dig in it for the second one ?
For the first solution, this scipt with your help do the job.
I would prefer the second solution where I don't need to put the name... but only a value max
Code: tabCurveName= {
'ECLAIRAGE-TOTAL-KWH',
'GENERAL_ELEC-TOTAL-KWH'
}
for _, curveName in ipairs(tabCurveName) do
id = db:getone('SELECT id FROM trends WHERE name=?', curveName)
if id then
file = '/tmp/trends/t' .. id .. '.trend'
os.remove(file)
end
end
script.disable(_SCRIPTNAME)
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
----------------
--------------
Hi Admin, can you give me example to find trend value greater than xxx and replace it by 0 for example ?
thks
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 7660
Threads: 41
Joined: Jun 2015
Reputation:
441
This example will change all valid trend values by x2. Recommended way of testing is to create a new trend log with the same parameters as the source.
Code: infile = '/tmp/trends/t1.trend'
outfile = '/tmp/trends/t7.trend'
function adjust(value)
value = value * 2
return value
end
ffi = require('ffi')
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
function doubletostring(val)
local dbl = ffi.new('double[1]')
dbl[ 0 ] = val
return ffi.string(dbl, 8)
end
info = require('rrd').info(infile)
data = io.readfile(infile)
offset = info.header_size
buf = { data:sub(1, offset) }
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
value = adjust(value)
end
buf[ #buf + 1 ] = doubletostring(value)
else
break
end
offset = offset + 8
end
data = table.concat(buf)
io.writefile(outfile, data)
For counter type the editing is trickier. The stored value is not the actual counter value but a per-second change rate. This example will log the maximum of all stored values without writing to a different trend log file. You can use this value for replacement in the previous script.
Code: infile = '/tmp/trends/t1.trend'
max = 0
function check(value)
max = math.max(max, value)
end
ffi = require('ffi')
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
info = require('rrd').info(infile)
data = io.readfile(infile)
offset = info.header_size
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
check(value)
end
else
break
end
offset = offset + 8
end
log(max)
Note that when replacing values you should not use direct comparison (==) for floating point values due to possible rounding errors.
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
great thanks for helping, I will try to test in the next day.
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
great this work for me !! Your top
Code: ffi = require('ffi')
typName=true
typCat=false
typTotal=false
maxVal=4/60/60
function check(value)
max = math.max(max, value)
log(value)
if value > maxVal then
value=maxVal
--log(value)
end
return value
end
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
function doubletostring(val)
local dbl = ffi.new('double[1]')
dbl[ 0 ] = val
return ffi.string(dbl, 8)
end
function wash(filename,maxVal)
max = 0
info = require('rrd').info(filename)
data = io.readfile(filename)
offset = info.header_size
buf = { data:sub(1, offset) }
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
value=check(value)
end
buf[ #buf + 1 ] = doubletostring(value)
else
break
end
offset = offset + 8
end
log('MAX:'..max)
data = table.concat(buf)
io.writefile(filename, data)
end
if typName then
tabCurveName= {
--'ADLC-TOTAL-KWH',
--'TG2-TOTAL-KWH',
--'ECLAIRAGE-TOTAL-KWH',
--'GENERAL_ELEC-TOTAL-KWH',
--'ADLC-01_ECLAIRAGE-KWH-ECL',
--'CHAUFFAGE_CLIMATISATION_VENTILATION-TOTAL-KWH',
--'RECE-01_ECLAIRAGE-KWH-ECL'
--'ADN0-TOTAL-KWH',
--'RECE-TOTAL-KWH'
--'DIVERS-TOTAL-KWH',
--'EAU_CHAUDE-TOTAL-KWH',
--'PRISES-TOTAL-KWH',
--'TG1-TOTAL-KWH',
--'RECE-01_LOCAL_FROID-M3-FLU',
'ADN2-01_ECLAIRAGE-KWH-ECL'
}
for _, curveName in ipairs(tabCurveName) do
id = db:getone('SELECT id FROM trends WHERE name=?', curveName)
if id then
filename = '/tmp/trends/t' .. id .. '.trend'
--os.remove(file)
wash(filename,maxVal)
end
end
end
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 228
Threads: 53
Joined: Sep 2015
Reputation:
0
(15.12.2023, 10:47)admin Wrote: This example will change all valid trend values by x2. Recommended way of testing is to create a new trend log with the same parameters as the source.
Code: infile = '/tmp/trends/t1.trend'
outfile = '/tmp/trends/t7.trend'
function adjust(value)
value = value * 2
return value
end
ffi = require('ffi')
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
function doubletostring(val)
local dbl = ffi.new('double[1]')
dbl[ 0 ] = val
return ffi.string(dbl, 8)
end
info = require('rrd').info(infile)
data = io.readfile(infile)
offset = info.header_size
buf = { data:sub(1, offset) }
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
value = adjust(value)
end
buf[ #buf + 1 ] = doubletostring(value)
else
break
end
offset = offset + 8
end
data = table.concat(buf)
io.writefile(outfile, data)
For counter type the editing is trickier. The stored value is not the actual counter value but a per-second change rate. This example will log the maximum of all stored values without writing to a different trend log file. You can use this value for replacement in the previous script.
Code: infile = '/tmp/trends/t1.trend'
max = 0
function check(value)
max = math.max(max, value)
end
ffi = require('ffi')
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
info = require('rrd').info(infile)
data = io.readfile(infile)
offset = info.header_size
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
check(value)
end
else
break
end
offset = offset + 8
end
log(max)
Note that when replacing values you should not use direct comparison (==) for floating point values due to possible rounding errors.
Hi,
I've used this scripts to replace erroneus values (due to power losses...) on counter type trends, but I've seen that daily grouped values are still wrong.
Are this values calculated from hourly values? Or are stored? Is there any way to update them?
Posts: 7660
Threads: 41
Joined: Jun 2015
Reputation:
441
This script goes over all values including daily data.
If you only want to modify daily data then modify the script at line 20:
Code: offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8
Posts: 228
Threads: 53
Joined: Sep 2015
Reputation:
0
Hi,
When I use the script to find max value, it's no showing daily max value... I managed to change a wrong value at 23:00 on 30 May but daily values are still wrong...
Posts: 7660
Threads: 41
Joined: Jun 2015
Reputation:
441
This will log all daily values with their respective offsets. If the whole list does not fit, you can change line 17 and add an extra offset. Just make sure that this extra offset is divisible by 8.
Code: infile = '/tmp/trends/t1.trend'
values = {}
ffi = require('ffi')
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
info = require('rrd').info(infile)
data = io.readfile(infile)
offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8
offset = offset + 0 * 8
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
values[ #values + 1 ] = string.format('%d %.1f', offset, value)
end
else
break
end
offset = offset + 8
end
text = table.concat(values, '\n')
log(text)
Posts: 228
Threads: 53
Joined: Sep 2015
Reputation:
0
(31.05.2024, 07:15)admin Wrote: This will log all daily values with their respective offsets. If the whole list does not fit, you can change line 17 and add an extra offset. Just make sure that this extra offset is divisible by 8.
Code: infile = '/tmp/trends/t1.trend'
values = {}
ffi = require('ffi')
function stringtodouble(chunk)
local dbl = ffi.new('double[1]')
ffi.copy(dbl, chunk, 8)
return dbl[ 0 ]
end
info = require('rrd').info(infile)
data = io.readfile(infile)
offset = info.header_size + (info['rra[0].rows'] + info['rra[1].rows']) * 8
offset = offset + 0 * 8
while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
values[ #values + 1 ] = string.format('%d %.1f', offset, value)
end
else
break
end
offset = offset + 8
end
text = table.concat(values, '\n')
log(text)
Ok, using this one I've managed to update wrong values. Thanks!
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
18.09.2024, 17:16
(This post was last modified: 18.09.2024, 17:40 by domotiqa.)
How can I display the date of the value (where I put log(date) ).
Don't find it
Code: while true do
chunk = data:sub(offset + 1, offset + 8)
if #chunk == 8 then
value = stringtodouble(chunk)
-- skip NaN (empty/undefined value)
if value == value then
values[ #values + 1 ] = string.format('%d %.1f', offset, value)
log(date)
end
else
break
end
offset = offset + 8
end
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
Posts: 7660
Threads: 41
Joined: Jun 2015
Reputation:
441
If you need dates then use trends.fetch function. Calculating the date using raw file access is not trivial because the data is written in a circular way.
Posts: 256
Threads: 37
Joined: Feb 2016
Reputation:
1
20.09.2024, 10:12
(This post was last modified: 20.09.2024, 10:12 by domotiqa.)
thanks. Yes I know with fetch,
I was looking for display the date of the current record int he script above
In rrd we have it at the begining I thought:
<!-- 2024-01-17 17:07:00 / 1705507620 --> <row><v> 1.6666666667e-02 </v></row>
It would be cool in order to match the date we want to modify the value for...
-----------
FRANCE SMARTHOME & SMARTBUILDING INTEGRATION
SE ECO EXPERT
|