Commit graph

53 commits

Author SHA1 Message Date
changukyu
5c05382944 docs: INDEX 에 B단계 기록 2026-07-25 23:34:27 +09:00
changukyu
6be263a69d feat: LLM 을 요약기에서 증거 판독기로 (구조화 인용 + 입력 경계 + 3B)
사용자 결정: LLM 역할을 "요약"에서 "증거 판독 + 구조화 판단"으로 확대,
타깃에 GPU 있음. 최종 판정은 여전히 사람이 한다.

## 입력 경계

프롬프트가 증거 data 딕셔너리를 통째로 f-string 에 박아 제출건당
13,733~62,291 자를 만들고 있었다. 화이트리스트 9개 필드 + 상위 12건으로
제한한다. promotion_reason 을 제외한 것이 핵심 — 모델이 그대로 베껴서
"요약"이 증거 목록 복붙이 됐다. 얼굴 크롭 좌표 / privacy_note /
thumbnail_url 도 모델 입력에서 뺀다(요약 입력이 아니다).

선별은 위험을 실제로 지는 행 우선: 지문 유사도 0.9 이상 -> full/partial/page
매칭과 promoted -> 얼굴 -> 나머지.

## 구조화 인용과 검증

format=json 으로 {claims:[{text, evidence}], needs_check:[]} 를 받고, 각
claim 이 인용한 라벨이 실제로 프롬프트에 들어간 라벨 집합의 부분집합인지
검사해 아니면 버린다. 화면에 렌더되는 텍스트는 검증을 통과한 claim 으로만
조립하므로 근거 없는 문장이 운영자에게 도달할 경로가 없다.

인용을 JSON 필드로 받는 것은 의도적이다. 산문을 문장 분해해 개체(URL, 숫자)를
대조하는 방식은 'confirming the image's authenticity [E3]' 같은 근거 없는
서술어를 100% 통과시켜, 오히려 환각 문장에 "근거 확인됨" 도장을 찍는다.
집합 포함 검사는 그 실패 모드가 구조적으로 발생하지 않는다.

증거 id 대신 [E1] 라벨을 쓴다 — 24자 해시를 소형 모델에게 받아쓰게 하면
오타와 조작을 구별할 수 없다.

grounded_claims / dropped_claims 를 증거에 기록한다. 모델이 지어내기
시작하면 수치로 드러난다.

## 모델과 컨텍스트

기본 모델 qwen2.5:0.5b-instruct -> qwen2.5:3b-instruct. 0.5B 는 출력 계약을
지키지 못했다(영어로 답하고 증거 목록을 반복했다).

num_ctx 를 명시한다. Ollama 기본 컨텍스트는 수천 토큰이고 넘으면 조용히
자르는데, source_urls / source_evidence_ids 는 잘리지 않은 전체 증거에서
뽑혀 붙었다. 즉 모델이 읽은 적 없는 근거를 근거라고 표시하고 있었다.
num_predict / keep_alive / timeout(30->120s) 도 환경변수로 노출.

## 시드 잠금 해제

_ensure_llm_summary 호출 2곳을 _write_lock 밖으로 옮긴다. record_decision 이
같은 락을 쓰므로, 폴더 재적재 중에는 운영자가 보고 있지도 않은 이미지의
요약이 끝날 때까지 판정 버튼이 대기했다. 시드 1회당 20건 상한을 두되 조용히
자르지 않고 로그를 남긴다(only-if-missing 이라 다음 재적재에서 이어진다).

이 이동은 provider 커버리지 회귀를 만들었다 —
_sync_submission_provider_state 가 잠금 구간 안에서, 즉 나레이션 전에 돌아
요약이 존재하는데도 llm 을 not_run 으로 보고했다. 나레이션이 실제로
발생했을 때만 커버리지를 다시 계산한다.

pytest 515 passed.
2026-07-25 23:34:02 +09:00
changukyu
4f140efd9b feat: 실사용 투입 전 T0 스탬프 4종 + 임베딩 정책 개정
두 갈래의 작업이 한 커밋에 들어 있다. llm_assistance.py 에서 T0-4 가 앞선
작업의 프롬프트 수정을 대체했기 때문에(프롬프트 리터럴이 llm_prompts.py 로
이동), 분리하려면 실제로 존재한 적 없는 중간 상태를 만들어야 한다.

## (1) 앞서 진행 중이던 review-intelligence 작업

- decision_stats(): 위험밴드 x 실제판정 교차표 + 캘리브레이션 지표
- cleared_reference 지식엔트리 타입: 이전에 클리어된 이미지 매칭은 +80 이
  아니라 0점 긍정 신호
- LLM 요약 본문 영속화(_evidence_payload 에 summary/sourceUrls) — 그전까지
  요약이 저장되지 않아 운영자에게 한 번도 보인 적이 없었다
- 운영자 GUI 반영

## (2) 실사용 투입 전 T0 — 첫 판정 이후에는 소급 불가한 것들

- T0-1 initialRiskScore/initialRiskBand/scorerVersion 각인.
  _rescore_submission 이 12곳에서 riskScore 를 덮어쓰므로 저장된 점수는 늘
  왕복 점수였고, 그것을 재채점과 비교하는 테스트는 항상 통과하는 빈 테스트다.
  레거시 행은 scorerVersion="legacy" 로 표시만 하고 점수를 날조하지 않는다.
- T0-2 얼굴검출 Haar -> YuNet, 인물 등급(identifiable/background) 도입.
  기존의 평평한 +35 는 사람이 찍힌 모든 사진을 medium 문턱 위로 올려 밴드
  분포를 붕괴시켰다(실측 high 4 / medium 2 / low 0). 등급은 payload 라운드
  트립으로 보존한다 — 잃으면 재채점마다 조용히 +35 로 되돌아간다.
  가중치는 models/(gitignore) 에 두고 offline-install.md 에 sha256 반입 절차.
  가중치가 없으면 Haar 로 강등하고 1회만 경고한다.
- T0-3 판정 시점 피처 스냅샷. 큐가 riskScore 내림차순이고 화면이 점수를 먼저
  보여준 뒤 판정을 받으므로, 스냅샷 없이 모은 라벨은 "이 근거가 반려를
  예측한다"가 아니라 "운영자가 내가 보여준 점수에 동의했다"를 기록한다.
- T0-4 모델/프롬프트 버전 + latency_ms 스탬프. latency_ms 는 _evidence_id
  해시에서 제외한다 — 포함하면 같은 요약이 재생성될 때마다 id 가 바뀐다.

## (3) 임베딩 저장 금지 정책 개정

금지의 취지는 생체 식별이지 이미지 유사도가 아니므로 범위를 좁힌다.
장면 임베딩(전체 이미지 의미 벡터)을 이미지 유사도 판정에 한해 허용하되
얼굴 마스킹 / 인물 지배 프레임 제외 / 목적 제한 / 별도 저장 클래스 /
연쇄 삭제 5조건을 assert_scene_embedding_allowed 로 강제한다. 마스킹 증명이
없으면 거부한다 — 누락은 마스킹을 건너뛴 것과 구별할 수 없다.
얼굴 임베딩 금지는 landmarks/face_descriptor/face_encoding 추가로 강화.

pytest 502 passed (착수 전 463).
2026-07-25 23:08:19 +09:00
changukyu
8b5b3f5cf9 docs: 2026-07-24 review report, implementation checklist, spec revision
- Add the 2026-07-24 multi-agent review report (43 confirmed findings, runtime
  smoke, feature proposals with judge scores) and the implementation checklist
  tracking what was done vs. deliberately deferred (with reasons).
- Record the batch-decision decision: the v1 "no bulk approve/reject" GUI-spec
  restriction is revised, not silently overridden, and lists the guardrails that
  made lifting it acceptable.
- Fix the README `python -m rights_filter.server` invocation (needs
  PYTHONPATH=src) and start a docs/INDEX.md continuity log.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E6GD9gxUtfUiVB3JHejGpU
2026-07-24 14:18:27 +09:00
changukyu
a2d0ef7a36 feat: operator throughput features (keyboard review, batch, dupes, export, backup)
- Keyboard review mode: J/K to move, A/H to decide and jump to the next
  unreviewed case, R to focus the memo, N for next unreviewed, ? for help, with
  a live position/backlog indicator; disabled while typing.
- Batch decisions: POST /api/submissions/decision-batch applies one decision to
  many selected cases through per-case transactions/audit/watchlist, reports
  per-case success and failure, caps at 200, requires a memo for reject and
  confirms when high-risk cases are included. Lifts the v1 "no bulk
  approve/reject" spec restriction under those guardrails (spec revised, static
  layout test updated to assert the guardrails instead of the absence).
- Decision-memo presets: GET/POST /api/decision-memo-presets (upsert by label),
  one-click chips in the decision pane, "save current memo" action.
- Submission duplicate / prior-case index: seed backfills fingerprints onto the
  submission record (idempotent), exact + pHash>=0.97 clustering, queue badge
  and group select, prior-decision chips and a conflicting-decision flag.
- Decision export: GET /api/decisions/export with separate internal and
  applicant field whitelists (applicant view never carries score/evidence),
  CSV with a BOM.
- Online backup/restore: POST /api/admin/backup snapshots the WAL database via
  the sqlite backup API plus the media dirs into a timestamped zip with
  retention rotation; `--restore` is an offline CLI that keeps the pre-restore
  database. GET /api/admin/backups lists snapshots.

Also folds in the co-located HTTP hardening (non-object/non-UTF8 JSON body ->
400, non-ASCII Authorization/cookie compared as bytes -> 401, media OSError ->
404) and the line-buffered startup logging that share these files.

Verified against a live server (auth gate, seeding, duplicate index, batch
decision guardrails, presets, export, backup + restore round-trip). pytest
tests/rights_filter tests/operator_gui: 457 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E6GD9gxUtfUiVB3JHejGpU
2026-07-24 14:18:11 +09:00
changukyu
db9ee9a7ca fix: remediate 2026-07-24 multi-agent review findings
HIGH:
- face detection: guard OpenCV builds lacking CascadeClassifier and isolate
  per-submission analysis failures in BatchAnalyzer, so a broken cv2 or one bad
  image no longer takes down every seed/reload/import/upload/rerun with no HTTP
  response
- risk scoring: gate Google custom image search (match=search_image) to 0
  points so the persistence round-trip (result_url aliased to url) can no longer
  flip the same evidence to +45 and inflate bands
- serialization: exclude SEARCH_SKIPPED/ENRICHMENT_FAILURE rows from the auto
  search dedup signatures so a quota-skipped query stays retryable instead of
  being treated as already done forever
- external_policy: add a daily-window reset + lock (mirrors SearchApiPolicy) so
  Vision is not permanently disabled once the daily limit is hit
- image_store: skip a malformed manifest row (and coerce bad width/height)
  instead of letting one ValueError paralyse loading/re-analysis for the queue

MEDIUM/LOW:
- enrichment: only count a face crop as transmitted when it actually left the
  system, removing false usage debits and a false "personal image sent to
  external provider" audit record; pick lastFailure from the failing item;
  preserve the collection set when every search was quota-skipped; dedup
  candidate counts; attribute Vision failures to the google provider badge
- store: search coverage requires a real result, not a failure row
- scoring: FACE_PERSON contributes once regardless of how many rows exist
- fingerprints: refuse near-matches on degenerate (near-uniform) average
  hashes while keeping identical-hash matches
- preprocessing: apply EXIF orientation for external derivatives, face crops,
  pHash and the PIL decode fallback
- promoter: scan title/description only, never the URL path/host tokens
- integrations: map Vision responses inside the adapter try; cap HTTPError
  bodies; skip the connect probe when a proxy applies; reservation-style quota
  accounting (try_reserve/release) to make the check-then-act atomic
- governance/admin: enforce the LLM-summary rule on the real persistence path;
  reject unknown manual-entry fields; upsert the rejected_image knowledge entry
  on repeat rejections; split policy skips out of the failures group
- env_file: strip a trailing inline comment from unquoted values

Environment drift fixed out-of-tree: removed the user-site OpenCV 5.0.0 that
shadowed the pinned opencv-python-headless==4.13.0.92.

Regression tests added; pytest tests/rights_filter is green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E6GD9gxUtfUiVB3JHejGpU
2026-07-24 14:17:45 +09:00
changukyu
2c607e0df6 docs: 2026-07-11 multi-agent review report, remediation checklist, offline extras
- Add the 2026-07-11 multi-agent project-review report and the remediation
  checklist (per-item status, with deferred items and their rationale).
- offline-install: document Ollama model and Playwright Chromium transfer, a
  hash-verified lock, --only-binary, and correct the egress note (image
  comparison fetches arbitrary SSRF-guarded public URLs, not only API hosts).
2026-07-11 16:08:04 +09:00
changukyu
6ab6da3719 chore: startup path anchoring, env parsing and gitignore
- Launcher guards Python 3.13 before importing the package and chdirs to the
  repo root so cwd-relative default paths resolve consistently.
- main() logs the resolved .env path and warns on a missing static dir or a
  short auth token; env_file reads utf-8-sig so a BOM doesn't drop the first
  variable.
- .gitignore covers .claude/settings.local.json and .impeccable/.
2026-07-11 16:07:49 +09:00
changukyu
3535bfee15 fix: operator GUI robustness, persistence and dead-control fixes
- Image viewer controls (fit / 1:1 / rotate) now have handlers and reset per
  case (were dead buttons).
- 'auto-entry deactivate' is scoped to the selected case's sourceSubmissionId
  (was deactivating an arbitrary global entry).
- dismissCollectionCandidate persists via a real endpoint (was a local splice
  that resurrected on the next refresh).
- setDecision has an in-flight guard; rerunSelectedEnrichment /
  applyCoverageFilter get null guards and a best-effort refresh on partial
  failure; deactivateCorrectionEntry skips already-inactive entries, tolerates
  per-entry failures, and always refreshes (guarded).
- executeSuggestedQueries / runCandidateCollection commit only after success
  so a failed run doesn't wipe the previous view; applyBootstrap clears a stale
  error banner; renderCorrections guards derivedEntries.
- riskClass allowlists the band and case/similar/face-crop URLs route through
  safeUrl (XSS defence-in-depth).
2026-07-11 16:07:30 +09:00
changukyu
4ef12f5f95 feat: HTTP security hardening (CSP, CSRF, session tag, body caps)
- Trusted app-shell responses now carry a strict script-src CSP plus
  X-Frame-Options: DENY and nosniff (img-src stays permissive for external
  evidence thumbnails; the GUI uses no inline scripts).
- Session cookie stores an HMAC-derived tag, not the raw COPYRIGHTER_AUTH_TOKEN,
  so a leaked cookie no longer reveals the token/bearer/.env secret.
- Cookie-authenticated state-changing requests require a same-origin
  Origin/Referer (CSRF); header-authenticated requests are not forgeable.
- _body rejects negative/non-integer Content-Length (was reaching
  rfile.read(-1), an unbounded read-to-EOF) and returns a controlled 400.
- Handler timeout=30 bounds slow-client thread exhaustion; uploads use
  exclusive-create ('xb') to close a concurrent same-name overwrite TOCTOU.

Tests: app-shell security headers, negative Content-Length -> 400, cookie tag
(not raw token), cookie-authed cross-origin POST -> 403 / same-origin -> 200.
2026-07-11 16:07:13 +09:00
changukyu
1803b4e0b8 fix: harden outbound integrations (errors, quota, SSRF, size cap)
- cloud_vision: raise on a per-image error inside a 200 response so the adapter
  records a FAILURE instead of a silent "no results" that also debited quota;
  send the key via X-goog-api-key header (verified for Vision) not the URL.
- search_policy: reset the daily_limit counter on a calendar-day rollover (it
  never re-windowed, so a hit limit disabled the provider until restart) and
  guard calls_made with a lock.
- google_custom_search: map the API response inside the adapter's try so a
  malformed payload becomes ENRICHMENT_FAILURE, not an uncaught crash. (Key
  stays a query param — the header is unverified for this endpoint.)
- http_json: cap the response body (10 MiB) and convert decode/read/OS errors
  into HttpJsonError so nothing escapes the transport contract.
- store_remote_fetch: SSRF guard rejects any non-global address (`not
  is_global`), covering CGNAT 100.64/10 the flag list missed.
- env_clients: a malformed daily-limit env falls back to a conservative cap
  (not fail-open unlimited); clamp NAVER display sizes.
- preprocessing: warn when external-derivative re-encoding falls back to the
  original bytes (EXIF/GPS may not have been stripped).

Tests: Vision per-image error raises / converts to FAILURE; daily-limit
malformed falls back conservatively.
2026-07-11 16:06:55 +09:00
changukyu
6193ed242d fix: serialize store read-modify-write and harden seed/queue paths
Under ThreadingHTTPServer several store writers ran unlocked read-modify-write
cycles, so concurrent operator actions could silently lose each other's
updates or revert an in-flight change.

Concurrency (_write_lock; external I/O kept OUTSIDE the lock):
- record_decision now takes _write_lock so a decision isn't reverted by a
  concurrent rescore/sweep.
- manual_search runs the network search first, then re-reads the submission
  under the lock before the queryHistory RMW (was clobbering concurrent edits).
- _apply_provider_usage_delta merges only usage/status onto the fresh row
  instead of rewriting a stale pre-search snapshot (kept an in-flight emergency
  disable from being reverted).
- _sync_provider_payloads preserves an operator's disable via an
  operatorDisabled flag (connectivity-only disables still reconnect).
- set_provider_enabled / emergency_disable / _sync_submission_provider_state /
  collect_keyword_candidates now lock (+ single transaction where applicable).

Robustness:
- seed: a zero-byte/corrupt image now yields a 'failed' submission + audit
  instead of an IndexError that aborted the whole import.
- ensure_queue: key the UPDATE/SELECT on the existing row id on a folder_path
  match (was matching zero rows and returning None -> seed TypeError); audit
  queue activation / legacy migration.
- store_enrichment: lastFailure uses the first failed item; auto-search dedup
  reads only the submission's evidence; log a swallowed face-crop load error.
- image_store: documents the pre-existing same-stem id limitation (a prior
  sibling-dependent fix was reverted — it resurrected decided submissions).

Tests: operator-disable survives reseed, usage delta keeps an in-flight
disable, seed survives an unanalyzable image, ensure_queue mismatched-id,
dismiss persistence.
2026-07-11 16:06:28 +09:00
changukyu
ac47f0c5d0 fix: restore scoring signals lost in the evidence DB round-trip
Production scores come from _rescore_submission, which re-reads evidence via
_domain_evidence_from_ui and re-scores. Serialization dropped the promoted /
entity fields and the precise EvidenceSource, so on every rescore Naver
promotion (+40) and Google web-entity (+40) silently scored 0 and skip
evidence was mis-restored as FAILURE (+30 phantom).

- store_serialization: persist domainSource / promoted / promotionReason /
  entity / category in _evidence_payload and restore them in
  _domain_evidence_from_ui; resolve the source from the persisted enum
  (keyword heuristic kept only as a legacy fallback).
- search_result_promoter: structural no-result detection (covers image/blog/
  web variants), scan only result fields not the query, add Korean rights
  markers (excluding over-generic substrings).
- risk_scoring: dedup promoted Naver hits by query_signature; drop zero-point
  skip reasons from the operator-facing reasons list.

Tests: round-trip regression (promoted naver / web entity / two distinct
entities score > 0; skip adds no phantom points) + skip-reason exclusion.
2026-07-11 16:06:05 +09:00
changukyu
89c1fe4788 fix: Phase 1 HIGH review remediation (5 of 6)
Backend (data-integrity / audit-trail):
- mark_evidence_status: make the read-modify-write + audit atomic under
  _write_lock + _transaction (was 4 unlocked connections, last-writer-wins,
  audit could diverge from DB state).
- promote_collection_candidates: acquire _write_lock, use a deterministic
  _stable_id, wrap writes in _transaction, and guard already-promoted batches
  so a retry/double-click cannot mint duplicate knowledge entries.
- risk_scoring: bucket web-detection entities by name so two distinct entities
  in one image both count (was collapsed to one 'entity' bucket, under-scoring
  high->medium). Identical entities still dedup.

Frontend (silent loss of operator actions):
- runManualSearch: in-flight guard; rerunHistoricalQuery voids the call,
  preventing duplicate/racing search POSTs.
- deactivateCorrectionEntry: persist via the real knowledge-deactivate
  endpoint instead of a local-only mutation + fake audit event.
- provider-retry button: report honestly (no backend endpoint) instead of
  writing a fake 'Provider called' audit event.

Tests: +4 regression tests (mark_evidence_status rollback, batch-promote
idempotency, two distinct entities -> high, identical entity dedup).
Remaining Phase 1 item #6 (_auto_naver/_auto_google refactor) deferred.

Backend suite: 361 passed. app.js: node --check OK, 41 static tests pass.
2026-06-22 14:01:05 +09:00
changukyu
d008e35c2e docs: add 2026-06-22 multi-agent project review report
66 verified findings (0 CRITICAL, 6 HIGH, 43 MEDIUM, 17 LOW) with a
prioritized remediation roadmap. Generated from an adversarially-verified
multi-agent review.
2026-06-22 14:00:45 +09:00
changukyu
ed049bd158 chore: add .gitattributes to enforce LF line endings
Prevent Windows/WSL editors from rewriting source files to CRLF, which
produced large content-free line-ending diffs. Forces LF for all text
files (text=auto eol=lf) and marks fonts/images as binary.
2026-06-22 11:22:23 +09:00
changukyu
71a6da10a3 fix: separate Naver blog/web auto-search dedup; drop dead helpers
_auto_naver_search previously used a shared 'continue' so a duplicate blog
query skipped that iteration's web search entirely. Split into independent
per-signature guards so blog and web auto-searches dedup separately.

Also remove unused private helpers left by the god-file split:
_is_generic_data_image_attr, _search_result_page_image_urls, _count,
_set_active_queue (all grep-verified to have no callers).
2026-06-22 10:00:09 +09:00
changukyu
35c63ce37a docs: add project README 2026-06-21 22:35:31 +09:00
changukyu
0bfa30d7f5 test: add Playwright E2E harness for operator GUI with decision-flow spec
Introduce a reusable browser-test harness under tests/operator_gui/:
- conftest.py: shared fixtures (operator_server, browser with graceful skip,
  operator_page with screenshot-on-failure artifacts)
- pages/: OperatorWorkbench Page Object encapsulating selectors and actions
- refactor test_browser_smoke.py onto the fixtures + POM (drops duplicated
  server/browser plumbing)
- test_decision_flow.py: approve/hold/reject E2E against the real server,
  covering the reject-requires-memo guard and decision persistence

Pin playwright in requirements-dev.txt (Chromium bundled offline for the
air-gapped target) and ignore the artifacts/ screenshot dir.
2026-06-21 21:10:22 +09:00
유창욱
0f69ed06e5 refactor: drop unused imports left by the god-file split (pyflakes-clean) 2026-06-20 22:42:19 +09:00
유창욱
2eb7bd3b8b docs: mark god-file split complete in remediation plan 2026-06-20 22:36:02 +09:00
유창욱
6184d0f464 refactor: extract submission-queue management into StoreQueueMixin
Move queue id derivation, active-queue selection, legacy-submission migration,
and queue bootstrap/ensure helpers into a mixin; CopyrighterStore inherits it.
Drop now-unused hashlib import. sqlite_store.py 874 -> 724 lines (5119 -> 724,
-86%); now under the 800-line guideline.
2026-06-20 22:35:02 +09:00
유창욱
b575d2ee06 refactor: extract operator operations into StoreOperationsMixin
Move knowledge-entry lifecycle, rerun/auto/manual search drivers, LLM summary
management, manual knowledge registration, and keyword-candidate collection/
promotion into a mixin; CopyrighterStore inherits it. Drop now-unused imports;
point the rollback test at store_serialization._stable_id. sqlite_store.py 1598
-> 874 lines (5333 -> 874, -84%).
2026-06-20 22:28:30 +09:00
유창욱
3bc07d94c3 refactor: extract enrichment internals into StoreEnrichmentMixin
Move provider-state sync, local face evidence/crop sync + retention purge,
internal/Google re-analysis, face-crop web detection, and the auto Google/Naver
search drivers into a mixin; CopyrighterStore inherits it. Drop now-unused
imports. Tests now also patch HeuristicFacePersonDetector on store_enrichment
(face detection moved there). sqlite_store.py 2358 -> 1598 lines (-70% overall).
2026-06-20 22:18:38 +09:00
유창욱
8e0a8c307d refactor: extract search-result similarity and candidate storage into mixin
Move the search-result image similarity, candidate-image storage, in-memory
knowledge repository, and rescoring methods into StoreSearchCandidatesMixin;
CopyrighterStore inherits it. Drop now-unused imports. sqlite_store.py 3072 ->
2358 lines (5333 -> 2358 overall, -56%). Behavior-preserving.
2026-06-20 21:58:10 +09:00
유창욱
40501e13f1 refactor: extract SQLite persistence primitives into StorePersistenceMixin
Move connection/transaction management and the generic _put/_get/_all row access
plus evidence-read helpers into a mixin; CopyrighterStore now inherits it. Methods
rely on the host class's self.db_path / self._write_lock. Behavior-preserving.
sqlite_store.py 3368 -> 3072 lines.
2026-06-20 21:46:01 +09:00
유창욱
3be7b016ce refactor: extract store constants and remaining domain helpers
Move shared constants + _bounded_int_env into store_constants (a leaf module),
and the remaining module-level domain helpers (validation, query signatures,
search-hint evidence, watchlist selection, knowledge type/provenance) into
store_serialization. sqlite_store.py is now the CopyrighterStore class plus thin
imports: 3613 -> 3368 lines (5333 -> 3368 overall, -37%). All behavior-preserving.
2026-06-20 21:38:03 +09:00
유창욱
8e53139029 refactor: extract payload serialization helpers into store_serialization
Move submission/evidence payload builders, provider-state derivation, UI<->domain
evidence mapping, weak-label handling, and id/label/image helpers into
store_serialization (depends only on stdlib + domain + url/text helpers, no store
coupling). Behavior-preserving; imported back into sqlite_store. 3992 -> 3613 lines.
2026-06-20 21:24:58 +09:00
유창욱
e3bc99e6b9 refactor: extract text helpers and HTML/CSS image-scraping from sqlite_store
Move the pure text helpers (_text_list, _unique_texts) into store_text and the
~950-line page/CSS/JSON/srcset image-URL extraction (the _PageImageParser and
its helpers) into store_page_scrape. Both behavior-preserving; store_page_scrape
depends only on stdlib + url/text helpers + domain Evidence (no store coupling).
sqlite_store.py 4955 -> 3992 lines.
2026-06-20 21:10:22 +09:00
유창욱
bd35cf6f3f docs: record remediation implementation status in plan 2026-06-20 20:57:02 +09:00
유창욱
da917755dd refactor: extract remote-fetch and schema modules from sqlite_store
Move the pure network layer (image/page/stylesheet fetchers, SSRF guard,
redirect-validating opener) into store_remote_fetch, and the DDL/typed-column/
constraint-migration helpers into store_schema. Both are behavior-preserving
relocations imported back into sqlite_store; tests repoint their fetch
monkeypatches to store_remote_fetch. sqlite_store.py 5333 -> 4948 lines.
2026-06-20 20:50:29 +09:00
유창욱
e66f9d5001 refactor: extract URL helpers into store_url_utils
Move the SQLite store's 5 URL helpers (_decoded_nested_url, _is_http_url,
_url_path_has_image_suffix, _url_has_image_format_hint, _url_looks_like_image)
into a focused module and import them back. Pure relocation, no behavior
change. First step of splitting the 5300-line sqlite_store god file.
2026-06-20 20:35:21 +09:00
유창욱
7317bfb2b3 fix: reject protocol-relative and backslash URLs in safeUrl
Address commit security review: the same-origin branch of safeUrl accepted
//host and /\host, which browsers normalize to an external host (open
redirect). Allow only true same-origin paths.
2026-06-20 18:47:13 +09:00
유창욱
f8aa10f91b fix: frontend URL scheme allowlist, fetch ok-check, image onerror
Add safeUrl() to gate external search-result URLs into href/src (blocks
javascript:/data:), parse the response body before the ok check in apiJson
so non-JSON error bodies surface the real status, and hide broken evidence
preview images via onerror.
2026-06-20 18:44:20 +09:00
유창욱
7f5799e5e1 fix: PII retention, write-race serialization, and correctness fixes
Governance: purge biometric face crops past a retention window (env
COPYRIGHTER_FACE_CROP_RETENTION_DAYS, default 90d) with an audit trail, run
at startup and reload; audit personal-image transmission to external Vision.
Concurrency: a process write lock + atomic provider-usage delta stop lost
counter updates; candidate promotion is idempotent (deterministic id + status
guard); seeding is serialized. Correctness: skip LLM summarize when a summary
already exists; constraint migration cleans orphan temp tables on failure.
Add provider-readiness startup log. Tests pin all of the above plus risk-band
boundaries (29/30/69/70, 100 cap) and media path-traversal guards.
2026-06-20 18:44:08 +09:00
유창욱
1abb1107a2 fix: cookie-based operator auth keeps token out of URLs
Address commit security review: replace the ?token= query fallback (which
leaked the token into logs/referrers) with an HttpOnly, SameSite=Strict
session cookie minted on the first header-authenticated request, so <img>
media loads authenticate without a URL token. Use hmac.compare_digest for
constant-time comparison and add Cache-Control: no-store + Referrer-Policy:
no-referrer on untrusted biometric media. Also cover upload/import boundary
validation (400) at the HTTP layer.
2026-06-20 18:43:53 +09:00
유창욱
62e2d183f8 fix: block SSRF to internal addresses in remote fetchers
Resolve each candidate image/page/stylesheet URL and refuse loopback,
RFC1918, link-local (cloud-metadata), reserved, multicast, and unspecified
targets before fetching; re-validate on every redirect hop via a custom
opener. URLs originate from external search-result content, so this closes
the operator server fetching internal services.
2026-06-20 18:22:10 +09:00
유창욱
8958dd1b83 chore: pin runtime dependencies for offline air-gapped install
Add requirements.txt (numpy/opencv-python-headless/pillow — the only
third-party runtime imports) and requirements-dev.txt, plus an offline
install runbook. Ignore .coverage and wheelhouse/.
2026-06-20 18:19:08 +09:00
유창욱
20a6f55408 fix: SQLite concurrency safety and atomic decision writes
Enable WAL + busy_timeout in _connect (ThreadingHTTPServer concurrent
operators no longer hit 'database is locked'), add a _transaction helper and
thread an optional conn through _put/_get/add_audit_event so record_decision
commits its status change, watchlist entry, and audit event atomically.
2026-06-20 18:19:08 +09:00
유창욱
e9a15e8110 fix: harden operator HTTP server
Remove wildcard CORS (prevented cross-origin reads of biometric/case data
from localhost), add optional shared-token auth gate on data routes
(COPYRIGHTER_AUTH_TOKEN; GUI shell + /health stay open), cap request body
size (413), and map malformed JSON to 400 and SQLite lock contention to 503.
2026-06-20 18:18:54 +09:00
유창욱
62c13faafa docs: implementation plan for project-review remediation 2026-06-20 18:18:54 +09:00
유창욱
37294dc140 fix: resolve multi-agent review findings for workbench efficiency round 2026-06-12 18:44:35 +09:00
유창욱
4d98582ed3 feat: rerun enrichment evidence diff with score delta and new-evidence badges 2026-06-12 18:00:43 +09:00
유창욱
1e0f4f8690 feat: persist and display detected face crop thumbnails in workbench 2026-06-12 17:56:09 +09:00
유창욱
646b871b76 feat: knowledge base search/filter, inline edit, and server-backed lifecycle actions 2026-06-12 17:51:36 +09:00
유창욱
cd9d69dddb feat: knowledge entry update/deactivate/reactivate endpoints with audit events 2026-06-12 17:48:26 +09:00
유창욱
cf342425c5 feat: expose google_search as operator manual text-query provider 2026-06-12 17:46:45 +09:00
유창욱
4abb837aaa feat: one-click and batch execution for suggested evidence queries 2026-06-12 17:44:48 +09:00
유창욱
63bbf0d755 docs: implementation plan for operator workbench efficiency (F1-F5) 2026-06-12 17:40:34 +09:00
유창욱
b4b8f4b5d8 docs: operator workbench efficiency design (F1-F5) 2026-06-11 16:03:15 +09:00