Labour Market Intelligence Through Automated Web Scraping

Executive Summary 

The Engineering Construction Industry Training Board (ECITB) required a more dynamic and evidence-based understanding of labour market demand within the engineering construction sector. Traditional intelligence sources – surveys, employer returns, and static datasets – provided valuable insights but were often time-lagged and limited in granularity. 

ECITB sought to explore whether automated web scraping of live job vacancy data could provide a more timely, scalable, and analytically robust mechanism for understanding workforce demand, skills shortages, geographic patterns, and occupational trends. 

Axion Solutions was commissioned to design, develop, and demonstrate a proof-of-concept web scraping and analytical platform capable of extracting, structuring, validating, and analysing job vacancy data relevant to ECITB’s remit. 

Although the project remains ongoing, substantial progress has been achieved in building a functioning system, validating its outputs, and demonstrating its strategic potential to stakeholders. 

The Challenge 

ECITB’s core challenge was twofold: 

1. Timeliness – Existing labour market data often reflected historical conditions rather than current demand. 
2. Granularity – Available datasets lacked detailed insight into specific occupations, skill requirements, contract types, and regional clustering. 

Additionally, the engineering construction sector presents specific complexity: 

• Occupational titles vary significantly between employers. 
• Skills terminology is inconsistent. 
• Job boards differ in structure, formatting, and metadata quality. 
• Duplicate listings appear across platforms. 
• Not all vacancies are sector-relevant. 

The task was therefore not simply to “scrape jobs,” but to: 

• Identify relevant job boards and data sources. 
• Develop automated scraping routines. 
• Structure and normalise extracted data. 
• Filter for ECITB-relevant occupations. 
• Deduplicate and validate records. 
• Generate interpretable analytical outputs. 
• Present results in a way that informs strategic workforce planning. 

The project needed to prove that such an approach was technically viable, analytically meaningful, and strategically useful. 

Our Approach  

Phase 1: Specification and Scoping 

Working to the P2505-3 ECITB Specification, Axion began with structured scoping workshops to clarify: 

• Priority occupations 
• Geographic scope 
• Sector relevance criteria 
• Required outputs and visualisations 
• Frequency of data extraction 
• Data governance considerations 

The initial framing was deliberately focused. The aim was to prove the concept on a defined set of labour-market questions before expanding scope. 

This disciplined approach ensured: 

• Manageable technical complexity 
• Rapid iteration 
• Early demonstrable value 
• Stakeholder confidence 

Phase 2: Technical Architecture and Data Pipeline Development 

Axion designed a modular data architecture comprising: 

1. Automated Web Scraping Layer 
2. 1. Structured extraction scripts tailored to selected job boards 
   2. Scheduling and repeatable runs 
   3. Resilience to layout changes 
2. Data Cleaning and Normalisation 
3. 1. Removal of HTML artefacts 
   2. Standardisation of fields (job title, location, salary, employer) 
   3. Structured parsing of free-text descriptions 
3. Relevance Filtering 
4. 1. Occupational keyword logic 
   2. Sectoral inclusion/exclusion rules 
   3. Iterative refinement to minimise false positives 
4. Deduplication and Record Integrity 
5. 1. Matching logic to remove cross-posted duplicates 
   2. URL and content similarity checks 
   3. Record validation procedures 
5. Analytical Layer 
6. 1. Occupational clustering 
   2. Regional distribution mapping 
   3. Salary range extraction 
   4. Contract type classification 
   5. Skills frequency analysis 
6. Dashboard and Presentation Layer 
7. 1. Interactive outputs 
   2. Summary metrics 
   3. Trend indicators 
   4. Stakeholder-facing visualisations 

The system was designed for scalability, allowing additional job boards or analytical modules to be incorporated without re-engineering the entire pipeline. 

Phase 3: Iterative Testing and Validation 

A critical part of the process was validation. 

Rather than assuming scraped data was “correct,” Axion: 

• Conducted manual cross-checking of sampled vacancies. 
• Tested occupational filtering logic. 
• Assessed misclassification rates. 
• Evaluated duplicate removal accuracy. 
• Refined keyword libraries. 

This iterative cycle significantly improved precision and reliability. 

The result was not merely a data extraction tool, but a structured labour-market intelligence pipeline capable of producing consistent outputs. 

Stakeholder Engagement and Demonstration  

A key milestone was the stakeholder demonstration session. 

The presentation of live outputs marked an inflection point in the project. Once stakeholders could interact with tangible data: 

• Discussion moved beyond technical feasibility. 
• New analytical questions surfaced organically. 
• Strategic implications became clearer. 
• Confidence in the approach increased. 

Importantly, the conversation shifted from: 

“Can this work?”  

to: 

“What more could this tell us?” 

This shift indicated that the project had successfully moved beyond proof-of-concept and into potential strategic capability. 

Insights Delivered to Date 

Although the system is still evolving, the work completed so far has demonstrated the ability to: 

• Identify live demand for ECITB-relevant occupations. 
• Detect geographic clustering of engineering construction roles. 
• Analyse advertised salary bands. 
• Track frequency of specific skills and certifications. 
• Compare contract types (permanent vs temporary). 
• Identify patterns in employer demand. 

These outputs provide a more dynamic complement to traditional survey-based intelligence. 

The tool has shown particular value in: 

• Testing assumptions about shortages. 
• Identifying emerging skills requirements. 
• Understanding regional demand concentration. 
• Supporting evidence-based discussions. 

Strategic Implications 

The stakeholder session revealed two important development threads: 

1. Expansion of Job Board Coverage 

There is clear value in extending coverage to additional job boards. 
This would: 

• Deepen the evidence base. 
• Improve representativeness. 
• Reduce source bias. 
• Strengthen trend confidence. 

Such expansion aligns naturally with the optional three-month extension built into the contract and would represent a proportionate scaling of the current architecture. 

2. Broader Market Intelligence Potential 

More strategically, the engagement session revealed appetite to explore: 

• Wider labour-market modelling. 
• Scenario testing. 
• Sector demand forecasting. 
• Skills pipeline alignment. 
• Market signal monitoring. 

This suggests the platform could evolve from a vacancy scraper into a broader market intelligence capability. 

Importantly, this was not driven by scope expansion for its own sake. It emerged organically once stakeholders saw the analytical potential. 

The project may now be touching something more strategically significant than originally anticipated. 

Governance and Data Considerations  

Throughout development, Axion has: 

• Ensured ethical and compliant scraping practices. 
• Avoided storage of unnecessary personal data. 
• Focused on aggregated analysis rather than individual records. 
• Built repeatable and auditable processes. 
• Designed documentation for future handover or scaling. 

The platform has been structured to ensure transparency, maintainability, and alignment with ECITB’s data governance standards. 

Value Delivered to Date 

Even at its current stage, the project has delivered: 

• A functioning automated scraping system. 
• Cleaned and structured labour-market dataset. 
• Occupational filtering logic specific to engineering construction. 
• Demonstrated analytical outputs. 
• Stakeholder validation and engagement. 
• Clear roadmap for extension. 

The project has successfully: 

• Proved technical feasibility. 
• Demonstrated analytical credibility. 
• Generated strategic interest. 
• Built confidence among stakeholders. 

Optional Extension and Next Phase 

The optional extension provides a sensible and proportionate window to: 

• Expand job board coverage. 
• Refine occupational classification. 
• Enhance analytical dashboards. 
• Improve skills extraction logic. 
• Test broader market intelligence use cases. 

Before progressing, a structured discussion is recommended to: 

• Clarify ECITB’s appetite for expansion. 
• Define ambition level. 
• Align budget and resourcing. 
• Prioritise high-value enhancements. 

The decision should be deliberate rather than incremental. 

Conclusion 

The ECITB Web Scraper project has progressed from concept to functioning as a labour-market intelligence platform. 

What began as a tightly scoped proof-of-concept has: 

• Demonstrated viability. 
• Produced actionable insights. 
• Shifted stakeholder conversations. 
• Revealed a wider strategic opportunity. 

While not yet complete, the work to date has established a strong technical and analytical foundation. 

With careful scaling, the platform has the potential to become a core component of ECITB’s future labour market intelligence capability — moving from retrospective reporting toward real-time evidence-based workforce planning.