Processing Australian BOM Weather XML Data

[jamesng]

Hi

I'd like to pull weather data from the Australian BOM XML data feed however need some assistance in processing their (complex) XML.

The data for the Melbourne forecast can easily be downloading via the following script and I've identified the info I need (commented out in the code below). however need some help to traverse the XML using the tools available in lua.

Any help would be greatly appreciated

Kind Regards
James

Code:

1234567891011121314151617

require('socket.ftp') data = socket.ftp.get('ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml') if data then   require('lxp')    -- retrieve data from XML node  -- Weather_Temp_Max = tree.xpath("//forecast/area[@aac='VIC_PT042']/forecast-period[@index='1']/element[@type='air_temperature_maximum']/text()")[0]  -- Weather_Temp_Min = tree.xpath("//forecast/area[@aac='VIC_PT042']/forecast-period[@index='1']/element[@type='air_temperature_minimum']/text()")[0]  -- Weather_Rain = tree.xpath("//forecast/area[@aac='VIC_PT042']/forecast-period[@index='1']/text[@type='probability_of_precipitation']/text()")[0]    grp.write('1/1/100',Weather_Temp_Max)  grp.write('1/1/101',Weather_Temp_Min)  grp.write('1/1/102',Weather_Rain)   else log('Fetch failed to retrieve weather data') end

[admin]

Since Expat library is stream-oriented, xpath cannot be used. Here's an example that parses the XML and retrieves the required fields:

Code:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859

require('socket.ftp') data = socket.ftp.get('ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml') if not data then   log('Fetch failed to retrieve weather data')   return end aac = 'VIC_PT042' forecast = {} function starttag(p, tag, attrs)   if tag == 'area' and attrs.aac == aac then     in_area = true   elseif tag == 'forecast-period' and in_area then     index = tonumber(attrs.index)     if index then       forecast[ index ] = {}     end   elseif index then     prop = attrs.type   end end function endtag(p, tag)   if tag == 'area' then     in_area = false   elseif tag == 'forecast-period' then     index = nil   elseif index then     prop = nil   end end function text(p, txt)   if in_area and index and prop then     forecast[ index ][ prop ] = txt:trim()   end end require('lxp').new({   StartElement = starttag,   EndElement = endtag,   CharacterData = text, }):parse(data) Weather_Temp_Max = forecast[ 1 ].air_temperature_maximum Weather_Temp_Min = forecast[ 1 ].air_temperature_minimum Weather_Rain = forecast[ 1 ].probability_of_precipitation -- remove percent sign from string if Weather_Rain then   Weather_Rain = Weather_Rain:gsub('%%', '') end grp.write('1/1/100', Weather_Temp_Max) grp.write('1/1/101', Weather_Temp_Min) grp.write('1/1/102', Weather_Rain)

[jamesng]

Thank you so much for the solution and for the quick reply. It's working perfectly!

If you don't mind me also asking, how can I extract the start-time-local value from the same forecast-period at index 1 and convert it to a lua Date data type?

Kind Regards
James

[admin]

You can get start/end time as Lua table like this. Then you can either convert it to timestamp via os.time() or use it as object value directly.

Code:

1234567891011121314151617181920212223242526272829

function parsedatetime(str)   local date = {}   date.year, date.month, date.day,   date.hour, date.min, date.sec =     str:match('(%d+)%-(%d+)%-(%d+)T(%d+):(%d+):(%d+)')   return date end function starttag(p, tag, attrs)   if tag == 'area' and attrs.aac == aac then     in_area = true   elseif tag == 'forecast-period' and in_area then     index = tonumber(attrs.index)     if index then       forecast[ index ] = {         start_time = parsedatetime(attrs['start-time-local']),         end_time = parsedatetime(attrs['end-time-local']),       }     end   elseif index then     prop = attrs.type   end end -- start_time = forecast[ 1 ].start_time -- end_time = forecast[ 1 ].end_time

[jamesng]

Again manu thanks for the code - I've added it to our system.

We're using this info combined with local sensors to automatically open exterior blinds to keep the property cool this summer.

Kind Regards
James

[sgraystar]

Fantastic. I came searching for parsing xml thinking I would have to start from scratch but the job has already been done. Thanks!!

For future users, some of the BoM files contain forecasts for multiple locations so you can generalise by changing the start of the function to search location instead of aac

Code:

12345678

-- Product ID from FTP Public Products and wanted location within the BoM file local bomProduct = "IDV10450" local location = "Melbourne" function starttag(p, tag, attrs)   if tag == 'area' and attrs.type == 'location' and attrs.description == location then -- and later data = socket.ftp.get("ftp://ftp.bom.gov.au/anon/gen/fwo/"..bomProduct..".xml")