feat: add page-fetcher-api — centralised Playwright page fetcher

Introduces page-fetcher-api, a new internal ASP.NET Core service that
centralises all web-page fetching through a single Playwright (headless
Chromium) browser instance. All fetches are persisted to the pageFetcher
SQL schema for auditing.

New projects:
- Apis/page-fetcher-api-models: FetchPageRequest, FetchPageResponse, IPageFetcherApiClient
- Apis/page-fetcher-data: PageFetchDbContext, PageFetchEntity, InitialSchema migration (schema: pageFetcher)
- Apis/page-fetcher-api: PlaywrightBrowserService (singleton), PageFetcherService, PageController

Changes to existing services:
- cv-matcher-api: JobTextExtractor now calls IPageFetcherApiClient instead of HttpClient
- cv-search-job: HtmlJobSearcher uses IPageFetcherApiClient (removes inline Playwright);
  CvSearchJobTask fetches individual job pages and applies keyword pre-filter before
  LLM call; passes pre-fetched JobDescription to cv-matcher-api to skip re-fetch
- common: add PageFetcherApiSettings
- docker-compose.yml, build.yml: add new service + env vars for callers

Closes #43

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-08 17:43:56 +03:00
parent 1222a86eb7
commit 898dd09d50
31 changed files with 1121 additions and 110 deletions
@@ -0,0 +1,20 @@
namespace PageFetcher.Models;
/// <summary>
/// Request to fetch a web page via the page-fetcher-api.
/// </summary>
public sealed class FetchPageRequest
{
/// <summary>Absolute HTTP or HTTPS URL to fetch.</summary>
public string Url { get; set; } = string.Empty;
/// <summary>
/// Playwright wait condition. Accepted values: <c>networkidle</c> (default), <c>domcontentloaded</c>, <c>load</c>.
/// </summary>
public string WaitFor { get; set; } = "networkidle";
/// <summary>
/// Identifies the calling service for audit purposes (e.g. <c>cv-matcher-api</c>, <c>cv-search-job</c>).
/// </summary>
public string CallerService { get; set; } = string.Empty;
}
@@ -0,0 +1,25 @@
namespace PageFetcher.Models;
/// <summary>
/// Result of a page fetch operation.
/// </summary>
public sealed class FetchPageResponse
{
/// <summary>Final URL after any redirects.</summary>
public string Url { get; set; } = string.Empty;
/// <summary>HTTP status code returned by the page. <c>0</c> on network failure.</summary>
public int StatusCode { get; set; }
/// <summary>Full rendered HTML as returned by Playwright.</summary>
public string Html { get; set; } = string.Empty;
/// <summary>Plain text extracted from the HTML (script/style stripped, whitespace normalised).</summary>
public string Text { get; set; } = string.Empty;
/// <summary>Whether the fetch succeeded. <c>false</c> on timeout or network error.</summary>
public bool Success { get; set; }
/// <summary>Exception message when <see cref="Success"/> is <c>false</c>.</summary>
public string? Error { get; set; }
}
@@ -0,0 +1,16 @@
using Refit;
namespace PageFetcher.Models;
/// <summary>
/// Refit client for the internal page-fetcher-api service.
/// All calls require the <c>X-Internal-Api-Key</c> header, configured at registration time.
/// </summary>
public interface IPageFetcherApiClient
{
/// <summary>
/// Fetches a web page via headless Chromium and returns the rendered HTML and extracted plain text.
/// </summary>
[Post("/api/page/fetch")]
Task<FetchPageResponse> FetchAsync([Body] FetchPageRequest request, CancellationToken ct = default);
}
@@ -0,0 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<AssemblyName>page-fetcher-api-models</AssemblyName>
<RootNamespace>PageFetcher.Models</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Refit.HttpClientFactory" />
</ItemGroup>
</Project>