+- Logic Machine Forum (https://forum.logicmachine.net)
+-- Forum: LogicMachine eco-system (https://forum.logicmachine.net/forumdisplay.php?fid=1)
+--- Forum: Scripting (https://forum.logicmachine.net/forumdisplay.php?fid=8)
+--- Thread: Read data from website and import them into objects. (/showthread.php?tid=4386)
Read data from website and import them into objects. - robikozo - 16.11.2022
Hi,
I'm wondering is you can take data from any website like https://www.nordpoolgroup.com/en/Market-data1/Dayahead/Area-Prices/ALL1/Hourly/?dd=LT&view=table
and export it into an virtual object.
Robert
RE: Read data from website and import them into objects. - sx3 - 17.11.2022
Check this script out that Admin helped me out with a couple of days/weeks ago. It fetches the prices from Vattenfall API, maybe it will work for you with your price area?
In my Case I live in Sweden, pricearea SN3 (SE3).
Easiest practise would be to find a API with JSON output I beleive.
Code:
require('json')
https = require 'ssl.https'
date = os.date('%Y-%m-%d')
url = 'https://www.vattenfall.se/api/price/spot/pricearea/' .. date .. '/' .. date .. '/SN3'
status = ssl.https.request(url)
data = json.decode(status)
if type(data) ~= 'table' then
return
end
ts = data[ 1 ].TimeStampDay
if type(ts) == 'string' then
ts = ts:split('-')
grp.checkupdate('5/5/28', {
year = tonumber(ts[1]),
month = tonumber(ts[2]),
day = tonumber(ts[3])
})
end
min = math.huge
max = -math.huge
cnt = 0
sum = 0
for i, item in ipairs(data) do
price = item.Value
min = math.min(min, price)
max = math.max(max, price)
sum = sum + price
cnt = cnt + 1
grp.checkupdate('5/5/' .. i, price)
end
grp.checkupdate('5/5/25', sum / cnt)
grp.checkupdate('5/5/26', max)
grp.checkupdate('5/5/27', min)RE: Read data from website and import them into objects. - robikozo - 18.11.2022
Hi,
How do you execute this script?
RE: Read data from website and import them into objects. - sx3 - 18.11.2022
Vattenfall release tomorrows prices aprox at 15.00 (Swedish time)
At midnight at 00.00 I execute a scheduled script that fetches todays prices.
And also I have a scheduled script that executes 15.00 and fetch tomorrows prices.
1-24 GA for hourly prices. TODAY
25-27 GA for min/max/median prices. TODAY
28 GA for datestamp for later verification that I have latest fetch.
29-54 GA for hourly prices. TOMORROW
55-57 GA for min/max/median prices. TOMORROW
I then send all of theese values to InfluxDB database and also 3rd party visualization server.
Attached pictures are a bit old and cointain minor bugs sending to Influx Server.. Bugs are fixed, but it gives you an idea how I represent the values.
RE: Read data from website and import them into objects. - tigi - 23.11.2022
Hi Robikozo,
Below code will fetch any data from a webpage.
All you have to do is look into the source html of the webpage and copy paste the html part you are interested in.
Take into account that hard returns have to be defined as /n
Replace the part you want to fetch with (.-)
Below example fetched the latest Endex 303 prices from the Belgian Engie website
Code:
-- Scraping a value from a webpage
-- https://www.domoticz.com/forum/viewtopic.php?t=25321
-- https://forum.logicmachine.net/showthread.php?tid=4386
https = require 'ssl.https'
url = 'https://www.engie.be/nl/energie/elektriciteit-gas/prijzen-voorwaarden/indexatieparameters/indexatieparameters-elektriciteit'
status = ssl.https.request(url)
-- find matching string and fetch the data you want with (.-)
response=status:match('<span class="table_mobile_header"><p><strong>Endex 303</strong></p>\n</span>\n<span class="table_mobile_data"><div >\n<p>(.-)</p>')
response = string.gsub(response, ',', '.') -- replace comma seperated floating number to point
response = tonumber(response) -- replace string number to number
log(response)Hope this helps!