One of our first projects – in addition to programming a WordPress theme based on a pre-approved design, optimized for search engines, a significant amount of attention and time was invested in the development of a software solution, a PHP-cURL-based crawler/data scraper. This tool saved us from manual input of approximately 12,000 parts and corresponding machines, schemas/diagrams, and their relationships.
During this process, programmed crawler tracked and found all the relationships (machine vs machine parts) as well with schemas/images and saved that all in the CSV and SQL file(ready to be imported in a wordpress database) on top the website was built.