Advanced Requestium Techniques for Python Web Automation and Scraping

Requestium is a Python library that combines Selenium (for browser automation) and Requests (for HTTP requests) to simplify working with both static and dynamically loaded web content.

Basic usage involves creating a Session instance, using s.get() for HTTP requests, and then employing Selenium methods such as find_element_by_id or execute_script to interact with dynamic elements, finally closing the driver with s.driver.quit() .

from requestium import Session
# Create a Session instance
s = Session(webdriver_path='path/to/your/webdriver')
# Use Requests style to fetch a page
s.get('http://example.com')
# Use Selenium to find an element
element = s.driver.find_element_by_id('my-element')
# Execute JavaScript
result = s.driver.execute_script("return document.title;")
# Close the WebDriver
s.driver.quit()

Before using Requestium, install the required packages:

pip install requestium
pip install selenium requests

Advanced techniques include handling dynamic loading by setting an implicit wait:

from requestium import Session
s = Session(webdriver_path='path/to/your/webdriver')
s.driver.implicitly_wait(10)  # wait up to 10 seconds
s.get('http://example.com')
element = s.driver.find_element_by_id('my-dynamic-element')

Simulating user actions such as clicks, text input, and scrolling can be done with Selenium methods:

from requestium import Session
s = Session(webdriver_path='path/to/your/webdriver')
s.get('http://example.com')
# Click an element
element = s.driver.find_element_by_id('my-button')
element.click()
# Input text
input_field = s.driver.find_element_by_name('my-input')
input_field.send_keys('Hello, World!')
# Scroll the page
s.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Cookies and headers can be managed using the Requests interface:

from requestium import Session
s = Session(webdriver_path='path/to/your/webdriver')
# Set custom headers
headers = {'User-Agent': 'My User Agent'}
s.headers.update(headers)
# Send GET request with headers
s.get('http://example.com', headers=headers)
# Save cookies
cookies = s.cookies
# Use saved cookies in subsequent request
s.get('http://another-example.com', cookies=cookies)

Although Requestium does not natively support asynchronous requests, it can be combined with asyncio for concurrent page fetching:

import asyncio
from requestium import Session
async def fetch_page(session, url):
    await session.get(url)
    return session.driver.page_source
async def main():
    s = Session(webdriver_path='path/to/your/webdriver')
    urls = ['http://example1.com', 'http://example2.com', 'http://example3.com']
    tasks = [fetch_page(s, url) for url in urls]
    results = await asyncio.gather(*tasks)
    for result in results:
        print(result)
asyncio.run(main())

To interact with elements inside an iframe, switch the driver context before performing actions and switch back afterward:

from requestium import Session
s = Session(webdriver_path='path/to/your/webdriver')
s.get('http://example.com')
iframe = s.driver.find_element_by_id('my-iframe')
s.driver.switch_to.frame(iframe)
# Operate inside iframe
element = s.driver.find_element_by_id('my-iframe-element')
element.click()
# Return to main document
s.driver.switch_to.default_content()

Handling pop‑up windows and alerts uses Selenium’s alert interface:

from requestium import Session
s = Session(webdriver_path='path/to/your/webdriver')
s.get('http://example.com')
# Trigger alert
s.driver.find_element_by_id('my-button').click()
alert = s.driver.switch_to.alert
alert.accept()  # or alert.dismiss()
# Trigger a JavaScript alert
s.driver.execute_script("window.alert('Hello, World!');")
alert = s.driver.switch_to.alert
alert.accept()

Screenshots and screen recordings can be captured with Selenium methods (recording may require external tools like ffmpeg):

from requestium import Session
import time
s = Session(webdriver_path='path/to/your/webdriver')
s.get('http://example.com')
# Screenshot
s.driver.get_screenshot_as_file('screenshot.png')
# Base64 screenshot
screenshot_base64 = s.driver.get_screenshot_as_base64()
# Record video (requires ffmpeg)
s.driver.start_recording_screen(video_name='recording.mp4')
time.sleep(5)  # perform actions
s.driver.stop_recording_screen()

These examples demonstrate how Requestium can be leveraged for sophisticated web automation and scraping tasks, provided all necessary dependencies and WebDriver configurations are correctly set up.