What We Built?
Custom web crawler
Goal
To automate the process of gathering information from a large number of energy company websites across Spain. This eliminates the need for manual data collection, saving the client significant time and resources.
Challenge
Developing a strong crawler capable of efficiently navigating diverse websites from various energy companies. The challenge involved handling different website structures, content layouts, and potential security measures against web scraping.
Solution
The solution involved a custom-built web crawler utilizing technologies like Typescript, Crawlee, and Playwright.
- Crawling Logic: The crawler was programmed to navigate through the websites of various energy companies, identifying and extracting relevant information
- Data Extraction: The crawler focused on extracting specific data points as defined by the client
- Data Consolidation: Extracted data was compiled and formatted into a single, organized CSV file for easy client access and analysis
SKILLS
Typescript, Crawlee, playwright
Result
The client received a valuable dataset in a user-friendly format (CSV file). This data encompasses information from thousands of Spanish energy companies, eliminating the time and effort required for manual data collection. This allows the client to utilize the data for further analysis, market research, or other strategic purposes.
