Read data from website and import them into objects.

[robikozo]

Hi,

I'm wondering is you can take data from any website like https://www.nordpoolgroup.com/en/Market-...view=table
and export it into an virtual object.

Robert

[sx3]

Check this script out that Admin helped me out with a couple of days/weeks ago. It fetches the prices from Vattenfall API, maybe it will work for you with your price area?
In my Case I live in Sweden, pricearea SN3 (SE3).

Easiest practise would be to find a API with JSON output I beleive.

Code:

require('json')
https = require 'ssl.https'
date = os.date('%Y-%m-%d')

url = 'https://www.vattenfall.se/api/price/spot/pricearea/' .. date .. '/' .. date .. '/SN3'
status = ssl.https.request(url)

data = json.decode(status)
if type(data) ~= 'table' then
  return
end

ts = data[ 1 ].TimeStampDay
if type(ts) == 'string' then
  ts = ts:split('-')

  grp.checkupdate('5/5/28', {
    year = tonumber(ts[1]),
    month = tonumber(ts[2]),
    day = tonumber(ts[3])
  })
end

min = math.huge
max = -math.huge
cnt = 0
sum = 0

for i, item in ipairs(data) do
  price = item.Value

  min = math.min(min, price)
  max = math.max(max, price)
  sum = sum + price
  cnt = cnt + 1

  grp.checkupdate('5/5/' .. i, price)
end

grp.checkupdate('5/5/25', sum / cnt)
grp.checkupdate('5/5/26', max)
grp.checkupdate('5/5/27', min)

[robikozo]

Hi,

How do you execute this script?

[sx3]

Vattenfall release tomorrows prices aprox at 15.00 (Swedish time)
At midnight at 00.00 I execute a scheduled script that fetches todays prices.
And also I have a scheduled script that executes 15.00 and fetch tomorrows prices.

1-24 GA for hourly prices. TODAY
25-27 GA for min/max/median prices. TODAY
28 GA for datestamp for later verification that I have latest fetch.

29-54 GA for hourly prices. TOMORROW
55-57 GA for min/max/median prices. TOMORROW

I then send all of theese values to InfluxDB database and also 3rd party visualization server.

Attached pictures are a bit old and cointain minor bugs sending to Influx Server.. Bugs are fixed, but it gives you an idea how I represent the values.

[tigi]

Hi Robikozo,

Below code will fetch any data from a webpage.
All you have to do is look into the source html of the webpage and copy paste the html part you are interested in.
Take into account that hard returns have to be defined as /n
Replace the part you want to fetch with (.-)

Below example fetched the latest Endex 303 prices from the Belgian Engie website

Code:

-- Scraping a value from a webpage
-- https://www.domoticz.com/forum/viewtopic.php?t=25321
-- https://forum.logicmachine.net/showthread.php?tid=4386

https = require 'ssl.https'
url = 'https://www.engie.be/nl/energie/elektriciteit-gas/prijzen-voorwaarden/indexatieparameters/indexatieparameters-elektriciteit'
status = ssl.https.request(url)

-- find matching string and fetch the data you want with (.-)
response=status:match('<span class="table_mobile_header"><p><strong>Endex&nbsp;303</strong></p>\n</span>\n<span class="table_mobile_data"><div >\n<p>(.-)</p>')

response = string.gsub(response, ',', '.') -- replace comma seperated floating number to point
response = tonumber(response) -- replace string number to number

log(response)

Hope this helps!