Context
Developed for the Metropolis of Lyon, SmartWatch is an automated data processing system that monitors and updates the opening hours of public facilities (such as town halls, swimming pools, and libraries) on the data.grandlyon.com open data portal. Previously, this verification process was performed manually by Metropolis agents who had to individually inspect and transcribe schedule details for approximately 200 distinct websites.
Challenge
Municipal websites present their opening hours in a variety of formats: different layouts, varied table structures, and natural language descriptions. Traditional rule-based data extractors (scrapers) fail as soon as the page structure changes, while manually gathering information from 200 websites is tedious and time-consuming. The main challenge is to design a robust, automated system capable of processing any web page, extracting information related to opening hours, and accurately converting these textual descriptions into a standardized format without human intervention; all while avoiding hallucinations (false or erroneous responses generated by a language model) and ensuring a structured, unambiguous output.
Approach
SmartWatch addresses this challenge by orchestrating a structured data pipeline divided into five sequential steps:
- Web scraping: the pipeline uses Playwright (a browser automation library) coupled with anti-detection headers to programmatically retrieve raw HTML from the target municipal web pages.
- Text normalization: the raw HTML is converted into clean text using the Inscriptis library, which strips out formatting tags, styles, and scripts to isolate the text content.
- Semantic filtering: to optimize processing speed and reduce the computational resources required, the normalized text is divided into segments (chunks). The system then calculates semantic lexical embeddings (vector representations of the text that capture its semantic meaning) using Jina Embeddings, and performs a similarity calculation with terms representative of opening hours to retain only the relevant segments.
- Structured information extraction: the isolated chunks are sent to a large language model (LLM) that operates independently. The request is structured according to a strict JSON schema to ensure that the model formats the schedules according to a predefined format, thereby eliminating any formatting discrepancies.
- Data standardization: the extracted JSON object is mapped to the OpenStreetMap (OSM) opening hours format using custom translation utilities, enabling direct comparison with existing portal data.
Features
- Semantic data reduction: intelligent segmentation and similarity assessment reduce the size of the text submitted to the LLM, thereby decreasing token consumption and, consequently, the GPU’s memory footprint.
- Database tracking: an SQLite database, managed via SQLAlchemy, records execution history, processing status, and historical data points for each URL.
- Automated reporting: a reporting system generates interactive HTML reports using Jinja2 templates, highlighting detected schedule differences in dedicated modal windows.
- Environmental impact measurement: incorporates the CodeCarbon library to track and record the electrical consumption and estimated CO2 emissions of the LLM inference calls.
Outcome
- 75% operational savings: reduces the time required to update the facility database from four days of manual work to just one day, including automated processing and human validation.
- Reusable architecture: the modular architecture, licensed under the MIT License and ready for use thanks to its containerization, serves as a model for other data automation projects in the metropolitan area, promoting the reuse of proven technologies.