Python Weather Data Scraping, CSV Export, and Visualization Using Requests, BeautifulSoup, Pandas, and Matplotlib

Python Weather Data Scraping and Visualization

We regularly monitor weather forecasts to adjust clothing and travel plans. This tutorial shows how to fetch real‑time and 14‑day weather data from China Weather (http://www.weather.com.cn) with requests and BeautifulSoup , save the data as CSV files, and visualize temperature, humidity, air quality, wind direction, and future trends using matplotlib , numpy , and pandas .

1. Data Acquisition

We request the web pages for the 7‑day ( weather ) and 15‑day ( weather15d ) forecasts. The URL pattern ends with a region code (e.g., 101280701 for Zhuhai). The requests.get() method retrieves the HTML content.

def getHTMLtext(url):
    try:
        r = requests.get(url, timeout=30)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        print('成功访问')
        return r.text
    except:
        print('访问错误')
        return None

2. Extracting Useful Information

Using BeautifulSoup , we locate the div with id="7d" for the 7‑day data and the div with id="15d" for the 14‑day data. Inside these containers, dates, weather descriptions, temperatures, wind directions, wind levels, precipitation, and humidity are extracted from ul / li or script tags.

def get_content(html):
    final = []
    bs = BeautifulSoup(html, 'html.parser')
    body = bs.body
    data = body.find('div', {'id': '7d'})
    # extract hourly data ...
    return final_day, final

Special handling is added for missing high temperatures and for cleaning units (e.g., removing the "℃" symbol).

3. Saving to CSV

Collected data are written to weather1.csv (hourly) and weather14.csv (daily) using the csv module. The header rows differ for the two files.

def write_to_csv(file_name, data, day=14):
    with open(file_name, 'a', errors='ignore', newline='') as f:
        if day == 14:
            header = ['日期', '天气', '最低气温', '最高气温', '风向1', '风向2', '风级']
        else:
            header = ['小时', '温度', '风力方向', '风级', '降水量', '相对湿度', '空气质量']
        f_csv = csv.writer(f)
        f_csv.writerow(header)
        f_csv.writerows(data)

4. Visual Analysis

Four main visualizations are created:

Temperature curve for a single day (24 hours) with average, max, and min markers.

Relative humidity curve for the same period.

Air Quality Index (AQI) bar chart with color‑coded pollution levels.

Wind direction and level radar chart using polar coordinates.

For the 14‑day forecast we plot high/low temperature trends, wind radar, and a weather‑type pie chart.

Correlation analysis between temperature and humidity is performed with a scatter plot and the Pearson coefficient is displayed.

def corr_tem_hum(data):
    tem = data['温度']
    hum = data['相对湿度']
    plt.scatter(tem, hum, color='blue')
    plt.text(20, 40, '相关系数为：' + str(calc_corr(tem, hum)), fontsize=10, color='red')
    plt.show()

5. Conclusions

The analysis shows a clear negative correlation between temperature and humidity, explains diurnal temperature swings, and discusses how air quality is influenced by temperature inversions and wind patterns. Future 14‑day forecasts indicate a gradual cooling after day 8 and dominant southeast and southwest winds.

6. Code Framework

The project consists of three scripts:

weather.py – data crawling and CSV export.

data1_analysis.py – visualization for the current day.

data14_analysis.py – visualization for the 14‑day forecast.

Running weather.py followed by the analysis scripts produces all figures shown in the article.