Weather Data Pipeline for Precision Farming

A local farmer and winery was frustrated with having to piece together disparate sources of weather and crop information before making planting and harvest decisions. This led to analysis paralysis and delayed action, ultimately hurting their yields and business outcomes. The solution was to unify all of these data sources into a single analytics-ready view to help them make decisions quicker.

I built an end-to-end weather data pipeline to help a farming client time planting and harvest decisions around local conditions. Data ingest runs on scheduled Azure Functions (timer triggers) that execute Python scripts on a set cadence.

A daily function pulls daily observations from multiple Virginia airport stations via NOAA's NOWData/ACIS API, along with 5-day temperature and precipitation forecasts from the National Weather Service API. A weekly function pulls both Virginia state-level crop condition data from the USDA NASS Quick Stats API and Virginia drought severity data from the U.S. Drought Monitor data service.

Each feed lands in its own raw table in Snowflake, appended idempotently with a MERGE key (e.g. station, date) so any re-pulled or revised records update cleanly instead of duplicating. From there, I used dbt Cloud to transform the data inside Snowflake through a medallion architecture where bronze is for append-only raw immutable data; silver for cleaned, typed, deduplicated data; and gold for analytics-ready data marts. Power BI connects directly to the Snowflake gold layer in Import mode and refreshes the dashboards daily.

I deliberately kept the orchestration lightweight with serverless timer-triggered functions because the workload involves small and infrequent API pulls, where a full orchestration platform would add cost and operational overhead without real benefit.

The pipeline unified four public data sources - weather observations, 5-day forecasts, crop conditions, and drought severity - into a single analytics-ready source of truth that informs farming decisions to maximize yields. The medallion design kept the raw history intact and auditable while serving clean data downstream.

Instead of manually checking multiple weather sites or fragmented sources of information, the client could see trends and metrics in one place to inform planting and harvest timing and protect seasonal yield.