Processing Australian BOM Weather XML Data

Processing Australian BOM Weather XML Data - Printable Version

+- LogicMachine Forum (https://forum.logicmachine.net)
+-- Forum: LogicMachine eco-system (https://forum.logicmachine.net/forumdisplay.php?fid=1)
+--- Forum: Scripting (https://forum.logicmachine.net/forumdisplay.php?fid=8)
+--- Thread: Processing Australian BOM Weather XML Data (/showthread.php?tid=1836)

Processing Australian BOM Weather XML Data - jamesng - 16.01.2019

Hi

I'd like to pull weather data from the Australian BOM XML data feed however need some assistance in processing their (complex) XML.

The data for the Melbourne forecast can easily be downloading via the following script and I've identified the info I need (commented out in the code below). however need some help to traverse the XML using the tools available in lua.

Any help would be greatly appreciated

Kind Regards
James

Code:
require('socket.ftp')
data = socket.ftp.get('ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml')
if data then
   require('lxp')
  
  -- retrieve data from XML node
  -- Weather_Temp_Max = tree.xpath("//forecast/area[@aac='VIC_PT042']/forecast-period[@index='1']/element[@type='air_temperature_maximum']/text()")[0]
  -- Weather_Temp_Min = tree.xpath("//forecast/area[@aac='VIC_PT042']/forecast-period[@index='1']/element[@type='air_temperature_minimum']/text()")[0]
  -- Weather_Rain = tree.xpath("//forecast/area[@aac='VIC_PT042']/forecast-period[@index='1']/text[@type='probability_of_precipitation']/text()")[0]
  
  grp.write('1/1/100',Weather_Temp_Max)
  grp.write('1/1/101',Weather_Temp_Min)
  grp.write('1/1/102',Weather_Rain)
  
else
 log('Fetch failed to retrieve weather data')
end

RE: Processing Australian BOM Weather XML Data - admin - 16.01.2019

Since Expat library is stream-oriented, xpath cannot be used. Here's an example that parses the XML and retrieves the required fields:

Code:
require('socket.ftp')
data = socket.ftp.get('ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml')

if not data then
  log('Fetch failed to retrieve weather data')
  return
end

aac = 'VIC_PT042'
forecast = {}

function starttag(p, tag, attrs)
  if tag == 'area' and attrs.aac == aac then
    in_area = true
  elseif tag == 'forecast-period' and in_area then
    index = tonumber(attrs.index)

    if index then
      forecast[ index ] = {}
    end
  elseif index then
    prop = attrs.type
  end
end

function endtag(p, tag)
  if tag == 'area' then
    in_area = false
  elseif tag == 'forecast-period' then
    index = nil
  elseif index then
    prop = nil
  end
end

function text(p, txt)
  if in_area and index and prop then
    forecast[ index ][ prop ] = txt:trim()
  end
end

require('lxp').new({
  StartElement = starttag,
  EndElement = endtag,
  CharacterData = text,
}):parse(data)

Weather_Temp_Max = forecast[ 1 ].air_temperature_maximum
Weather_Temp_Min = forecast[ 1 ].air_temperature_minimum
Weather_Rain = forecast[ 1 ].probability_of_precipitation

-- remove percent sign from string
if Weather_Rain then
  Weather_Rain = Weather_Rain:gsub('%%', '')
end

grp.write('1/1/100', Weather_Temp_Max)
grp.write('1/1/101', Weather_Temp_Min)
grp.write('1/1/102', Weather_Rain)

RE: Processing Australian BOM Weather XML Data - jamesng - 16.01.2019

Thank you so much for the solution and for the quick reply. It's working perfectly!

If you don't mind me also asking, how can I extract the start-time-local value from the same forecast-period at index 1 and convert it to a lua Date data type?

Kind Regards
James

RE: Processing Australian BOM Weather XML Data - admin - 16.01.2019

You can get start/end time as Lua table like this. Then you can either convert it to timestamp via os.time() or use it as object value directly.

Code:
function parsedatetime(str)
  local date = {}

  date.year, date.month, date.day,
  date.hour, date.min, date.sec =
    str:match('(%d+)%-(%d+)%-(%d+)T(%d+):(%d+):(%d+)')

  return date
end

function starttag(p, tag, attrs)
  if tag == 'area' and attrs.aac == aac then
    in_area = true
  elseif tag == 'forecast-period' and in_area then
    index = tonumber(attrs.index)

    if index then
      forecast[ index ] = {
        start_time = parsedatetime(attrs['start-time-local']),
        end_time = parsedatetime(attrs['end-time-local']),
      }
    end
  elseif index then
    prop = attrs.type
  end
end

-- start_time = forecast[ 1 ].start_time
-- end_time = forecast[ 1 ].end_time

RE: Processing Australian BOM Weather XML Data - jamesng - 16.01.2019

Again manu thanks for the code - I've added it to our system.

We're using this info combined with local sensors to automatically open exterior blinds to keep the property cool this summer.

Kind Regards
James

RE: Processing Australian BOM Weather XML Data - sgraystar - 12.10.2023

Fantastic. I came searching for parsing xml thinking I would have to start from scratch but the job has already been done. Thanks!!

For future users, some of the BoM files contain forecasts for multiple locations so you can generalise by changing the start of the function to search location instead of aac

Code:
-- Product ID from FTP Public Products and wanted location within the BoM file
local bomProduct = "IDV10450"
local location = "Melbourne"
function starttag(p, tag, attrs)
  if tag == 'area' and attrs.type == 'location' and attrs.description == location then

-- and later
data = socket.ftp.get("ftp://ftp.bom.gov.au/anon/gen/fwo/"..bomProduct..".xml")