+ =qj\^RIt^RIt^RIt^RIHtHtHtHt^RIH t H t ^RI H t H t HtHtHtHtHt^RIHt^RIHt^RIHt^RIHtHt^R IHt^R IHtH t H!t!^R I"H#t#H$t$^R I%H&t&^R I'H(t(^RI)H*t*^RI+H,t,] !RR.R7t-!RR]4t.]-P_R4]!RRR7R] !RR73RRll4t0]-PcR4RRl4t2]-PcR]3]&,R 7RCR!R"ll4t4]-P_R#]$R 7] !R43R$R%ll4t5]-PcR&4RDR'R(ll4t6R)R*lt7]-PcR+4R,R-l4t8]-PcR.4R/R0l4t9]-PcR14RER2R3ll4t:]-P_R44R5R6l4t;R7R8lt<]-PcR94RER:R;ll4t=!R<R=]4t>!R>R?]4t?]-P_R@4RERARBll4t@R#)FN)datedatetime timedeltatimezone) AnnotatedAny) APIRouterBodyDependsHeader HTTPExceptionPathstatus) BaseModel)text) AsyncSession) CurrentUser require_admin) get_session)list_scheduled_jobsschedule_backfill_for_companytrigger_manual_scrape)RaciusImportItemRaciusImportResult)ScrapingLogOut) parse_mj_html)extract_people) is_valid_nifz/adminadmin)prefixtagscHa]tRt^toRtRtRtRtRtRt Rt V3Rlt Rt Vt R#) ScrapeOptionsNc<V^8dQh/S[R,;R&S[R,;R&S[R,;R&S[S[,R,;R&S[R,;R&S[R,;R&S[R,;R&#) N date_fromdate_todaystypesmax_days max_items max_seconds)rstrlistintfloat)format __classdict__s"app/api/admin.py __annotate__ScrapeOptions.__annotate__s}d{! D[ * 9t "  Dj Tz $)__name__ __module__ __qualname____firstlineno__r&r'r(r)r*r+r,__annotate_func____static_attributes____classdictcell__r2s@r3r#r#s/!IGD"EH I $Kr6r#z/scrape/{source}.z^(distribuicao|distribuicao_backfill|cire|ptdata|dre|contracts|registry_sweep|registry_expand|registry_content|registry_nacional)$)patterndefaultc V^8dQhR\R\\\\43,R\ R,R\ \\3,/#)r%source_adminoptionsNreturn)r-rrr rr#dict)r1s"r3r4r4(sL22 2 k7=#99 :2T !2 #s(^ 2r6cx"V'dVPRR7M/p\W4GRjxL pRVRV/#L 5i)T) exclude_noneNrDtask_id) model_dumpr)rDrErFoptsrKs&&& r3 scrape_nowrN's? 5<7  4  0D)&77G fi 118s +:8 :z/scheduled-jobscV^8dQhR\\\\43,R\\ \ \3,,/#)r%rErG)rrr rr.rHr-object)r1s"r3r4r43s9!! k7=#99 :! $sF{ !r6c"\4#5i)z4List all APScheduler jobs with their next run times.)rrEs&r3scheduled_jobsrS2s   s z/scraping-logs)response_modelc V^8dQhR\\\\43,R\\\\ 43,R\ R\\,/#r%sessionrElimitrG) rrr rrrr/r.r)r1s"r3r4r4;sS |W[%99 : k7=#99 :  . r6c"VP\R4RV/4GRjxL P4pVUu.uFp\V^,V^,V^,V^,V^,V^,;'g^V^,;'g^V^,V^,V^ ,;'g^V^ ,;'g^R7 NK up#Luupi5i)a SELECT id::text, source, started_at, finished_at, status, rows_seen, rows_new, error, params, progress_current, progress_total FROM scraping_logs ORDER BY started_at DESC LIMIT :lim limN) idrD started_at finished_atr rows_seenrows_newerrorparamsprogress_currentprogress_total)executerallrrWrErXrowsrs&&& r3 scraping_logsri:soo  EN   ce &   A tAaDQqTqtAaDdiia!A$))!1Q4!qTYYQquzz     s9!CCCAC<C!C0C CCCz/racius-importcV^8dQhR\\\\43,R\\\\ 43,R\ \,R\/#)r%rWrEitemsrG) rrr rrrr.rr)r1s"r3r4r4XsT22 |W[%99 :2 k7=#99 :2  !2 2r6c <".p.p.pVEFop\VP4'g"VPRVPRR/4K@VP\ R4RVP/4GRjxL P 4pV'd"VPRVPRR/4KVP\ R4RVPR VP R VPR VP/4GRjxL P 4pVP4GRjxL VPVP4\V^,VP4VPV^,4EKr \W4\V4R 7#EL LLt5i) uXBulk-import closed/insolvent competitors from Racius. Each row lands as monitored=TRUE, active=FALSE (one-shot historical scrape only). After insert, the per-company backfill is scheduled — 180d distribuicao + all CIRE — so we capture the credor/executado footprint without enrolling the company in the daily cron (active=FALSE gates that).nifreasonzinvalid NIF checksum-SELECT id::text FROM companies WHERE nif = :nnNzalready existsa INSERT INTO companies (nif, legal_name, status, monitored, active, monitoring_type) VALUES (:nif, :legal, :status, TRUE, FALSE, :mt) RETURNING id::text legalrmt)insertedskippedtotal_scheduled_backfills) rrmappendrdrfirst legal_namermonitoring_typecommitrrlen) rWrErkrsrt backfill_idsitemexistingrows &&& r3 racius_importrWsjH$&G LDHH%% NNE488X7MN O //DEdhh  %'   NNE488X7GH I //488T__dkk$..   %'# $nn!%c!fdhh7CF#EF c,FW =  " s>A3F5F6FA/F?F%F%F&A0FFFz/false-positive-reportsc V^8dQhR\\\\43,R\\\\ 43,R\ R\\\\3,,/#rV) rrr rrrr/r.rHr-rP)r1s"r3r4r4s\ |W[%99 : k7=#99 :  $sF{  r6cV"VP\R4RV/4GRjxL P4pVUu.uF_pRV^,RV^,RV^,RV^,RV^,RV^,R V^,R V^,R V^,R V^ ,/ NKa up#L~uupi5i) a? SELECT id::text, company_id::text, company_legal_name, process_number, tribunal, source, parties_snapshot, flagged_by_email, flagged_at, reason FROM false_positive_reports ORDER BY flagged_at DESC LIMIT :lim rZNr[ company_idcompany_legal_nameprocess_numbertribunalrDparties_snapshotflagged_by_email flagged_atrn)rdrrerfs&&& r3false_positive_reportsrsoo  EN   ce * A !A$ ad,@!A$ adJ!h! !&8!A$ !A$!A$     s!!B)B"B)A%B$B)$B)c0V^8dQhR\R\/#)r% root_pathzip_namer-)r1s"r3r4r4sSCr6c >^RIp^RIp^RIHp^RIHpV!V4pVP 4'g\\PRV R24hVP4pVPVRVP4;_uu_4pVPR4FFp V P4'gKVPV \!V P#V44R7KH RRR4VP%^4V!VR R R V R 2/R 7# +'giL2;i)rN)r)StreamingResponsez/Extension source not found on server (expected z).w*)arcnamezapplication/zipzContent-Dispositionzattachment; filename="") media_typeheaders)iozipfilepathlibrfastapi.responsesrexistsr rHTTP_404_NOT_FOUNDBytesIOZipFile ZIP_DEFLATEDrglobis_filewriter- relative_toseek) rrrrrrrootbufzpaths && r3_stream_extension_ziprs 3  ?D ;;==  % %=i[ K   **,C c7#7#7 8 8AJJsOD||~~c$*:*:4*@&AB$ 9HHQK  )&*@ !(LM  9 8s(D 3/D  D z/mj-extension.zipcXV^8dQhR\\\\43,/#r%rErrr r)r1s"r3r4r4s&QQ k7=#99 :Qr6c""\RR4#5i)zzStable per-NIF capturer ZIP. Served from the host-mounted /app/extension so file edits roll out without a backend rebuild.z/app/extensionzintel-grid-mj-capturer.ziprrRs&r3mj_extension_ziprs !!13O PP z/mj-extension-bulk.zipcXV^8dQhR\\\\43,/#rr)r1s"r3r4r4s&RR k7=#99 :Rr6c""\RR4#5i)uBulk (beta) extension ZIP — iterates every monitored NIF through one captcha-backed session. Separate from the stable extension so we can experiment without affecting the per-NIF flow.z/app/extension-bulkzintel-grid-mj-bulk.ziprrRs&r3mj_extension_bulk_ziprs !!68P QQrz /mj-monitoredc V^8dQhR\\\\43,R\\R,\ RR73,R\ \\\3,,/#)r%rW x_intel_tokenN X-Intel-TokenaliasrG) rrr rr-r r.rHr)r1s"r3r4r4sU"" |W[%99 :"S4Zo)FFG" $sCx."r6c^"\W4GRjxL VP\R44GRjxL P4pVUu.uFQpRV^,RV^,RV^,RV^,RV^,'dV^,P 4MR/NKS up#LLruupi5i)zReturn every active internal+competitor NIF the Intel Grid wants captured, sorted by oldest-last-capture first so successive runs preferentially refresh the stalest entries.Na SELECT c.id::text, c.nif, c.legal_name, c.monitoring_type, MAX(d.created_at) AS last_captured_at FROM companies c LEFT JOIN dre_publications d ON d.company_id = c.id AND d.source = 'mj' WHERE c.active = true AND c.monitoring_type IN ('internal', 'competitor') GROUP BY c.id, c.nif, c.legal_name, c.monitoring_type ORDER BY MAX(d.created_at) ASC NULLS FIRST, c.nif r[rmrxrylast_captured_at)_check_mj_tokenrdrre isoformat)rWrrgrhs&& r3 mj_monitoredrs ' 111oo      ce! 2 A !A$ 1Q4 !A$ qt AaDD! 0d    %2 s7B-B$!B-B&B- 8B(B(!B-&B-(B-z /mj-tokencV^8dQhR\\\\43,R\\\\ 43,R\ \\3,/#)r%rWrrG)rrr rrrrHr-)r1s"r3r4r4sL |W[%99 : ['-"88 9 #s(^r6c v"R\P!^ 4,p\P!\P R7\ ^R7,pVP\R4RVRVPRV/4GRjxL VP4GRjxL R VR VP4/#L0L5i) uIssue a 24h token the admin embeds in their bookmarklet. Each call generates a new token and supersedes any previous one for the same user — simpler than tracking devices. Old tokens still work until they expire naturally.mj_)tz)hourszy INSERT INTO mj_import_tokens (token, user_id, expires_at) VALUES (:tok, :uid, :exp) tokuidexpNtoken expires_at) secrets token_urlsafernowrutcrrdrr[rzr)rWrrexpiress&& r3mj_generate_tokenrs G))"- -Ellhll+ib.AAG //   uehhw7  .. g'') s$BB9B5B9B7B97B9cJV^8dQhR\R\R,R\/#)r%rWrNrG)rr-)r1s"r3r4r4!s( *-*r6c"V'g\\PR4hVP\ R4RV/4GRjxL P 4pV'g\\PR4hV^,#L=5i)missing X-Intel-Tokenz SELECT user_id::text FROM mj_import_tokens WHERE token = :t AND expires_at > now() tNinvalid/expired token)r rHTTP_401_UNAUTHORIZEDrdrrw)rWrrs&& r3rr!s{ F88:QRRoo   -    eg F88:QRR q6M sABBB $Bz /mj-existingc V^8dQhR\R\\\\43,R\\R,\ RR73,R\ \\3,/#)r%rmrWrNrrrG)r-rrr rr rHr)r1s"r3r4r47sZ'' ' |W[%99 :'S4Zo)FFG' #s(^ 'r6c"\W4GRjxL VP\R4RV/4GRjxL P4pV'gRRR./#VP\R4RV^,/4GRjxL P 4pRV^,RVUu.uF2pRVP R VP R \VP4/NK4 up/#LLL`uupi5i) u'Return every MJ publication already captured for this NIF so the extension can skip re-fetching the detail of rows we already have. `has_detail` is True when raw_json.detail_body is non-empty — those rows should be skipped entirely on the next capture run to avoid hammering publicacoes.mj.pt.Nrorprrga SELECT to_char(date, 'YYYY-MM-DD') AS d, title, COALESCE(length(raw_json->>'detail_body'), 0) > 0 AS has_detail FROM dre_publications WHERE company_id = :c AND source = 'mj' crtitle has_detail) rrdrrwredrboolr)rmrWrcorgrhs&&& r3 mj_existingr6s ' 111oo @ A #J   eg  dFB//oo  "Q%L   ce  be QSS'177L$q||:L M  /2   sIC&C$C&CC&-C&>C? C&8C! C&C&C&!C&cDa]tRtRtoRtRtRtRtRtRt V3Rlt Rt Vt R#) MjImportRowiaNc<V^8dQh/S[R,;R&S[R,;R&S[R,;R&S[R,;R&S[R,;R&S[R,;R&#)r%Nrrmentityconcelhoacto detail_htmlr)r1r2s"r3r4MjImportRow.__annotate__asg * t $JDj  *  t" r6r7) r8r9r:r;rrmrrrrr<r=r>r?s@r3rras*DCFHD"K r6rc@a]tRtRtoRtRtRtRtRtV3Rlt Rt Vt R#)MjImportPayloadijNc<V^8dQh/S[R,;R&S[S[,R,;R&S[R,;R&S[R,;R&S[R,;R&#)r%Nhtmlrgrmnif_hintr)r-r.r)r1r2s"r3r4MjImportPayload.__annotate__js` * { d ")  tDjd !r6r7) r8r9r:r;rrgrmrrr<r=r>r?s@r3rrjs'D%)DCH!Jr6rz /mj-importc V^8dQhR\\\43,R\\\ \ 43,R\\ R,\RR73,R\\ \3,/#)r%payloadrWrNrrrG) rrr rr rr-r rHr)r1s"r3r4r4wslee . /e |W[%99 :eS4Zo)FFGe #s(^ er6c""V'g\\PR4hVP\ R4RV/4GRjxL P 4pV'g\\PR4hVP\ R4RV/4GRjxL VP 4GRjxL ^RIHpVP;'g VPp.pVP'EdVPEFnpVPpRp V'd0^RI H p RJF#p V PW4P4p M VP 'dV!VP 4M/p VP"VP$VP&3U u.uFq'gK V NK pp TP)R VP&;'gR R ,R R P+V4R,RV RVP&RRRRRVPRRVPRVPRVP"RVP$RVP&RV P-R4RV P-R4//4EKq M)VP.'d\1VP.VR7pV'g R^R ^R!^R"R#/#VP2pV'gTpV'g,VF%pVP-R4'gKVR,pM V'dEVP\ R$4R%V/4GRjxL P 4pV'd V^,pV'gR^R \5V4R!\5V4R"R&/#^R'IHp^p^pVEFqpVP-R4pT R(VP-R 4;'gR P;4P=4 R)T;'gR 2p\>P@!VPCR*44PE4pV!VP-R 4VP-R 44p\GVP-R4;'g/P-R44pTP\ R+4R,TRVP-R 4;'gR R-VP-R 4R.TR/VP-R4R0VP-R4R1\HPJ!VP-R4;'g/\LR27R3VR4VPNR5VPPR6V/ 4GRjxL pVP 4pRpV'd+V^,'d V^, pMV'd V^, pV'gEKVP\ R74R3V/4GRjxL P 4pV'gEK>V^,pVP-R4;'g/P-R4;'gR p\SV4Fp TP\ R84R9TR,TRV R,R:V P-R;4RV P-R?4;'gR@RAV/4GRjxL V R,PURB4'gKV P-R;4'gKVP\ RC4RV R,R:V R;,RD,/4GRjxL K EKt VP 4GRjxL RVREVR \5V4V, V, R!\5V4RFVR"RGV RHV RI2/#ELELEL \dEK-i;iuup iEL8EL;ELLLLg5i)Ka=Receive HTML scraped by the admin's browser bookmarklet, parse publication rows, insert into dre_publications with source='mj'. Auth via X-Intel-Token (short-lived, issued by /admin/mj-token). No session cookie because the POST comes from publicacoes.mj.pt origin (cross-origin, no cookies sent per our CORS config).rz~ SELECT user_id FROM mj_import_tokens WHERE token = :t AND expires_at > now() rNrz UPDATE mj_import_tokens SET last_used_at = now(), uses = uses + 1 WHERE token = :t )parse_detail_html)rr:NiNsummaryu · :NiNrtype source_urlrDmjnipcrawrmrrr detail_fieldsfields detail_bodybody) nipc_hintrsrtparsedmessagezNo publication rows found.rorpz>!#FJ ooHI*%eg   V Ay#e*hE Y[ [5HH FF6N D"&&/"7"7R!>!>!@!F!F!H I177PR)T NN3::g. / 9 9 ;rvvgy(9:266%=..B33MBC OO   "zRVVG_**RVVI&QbffVnRVVL)U[[!4!4"cBQs}}cooj #  >iik 1vvA A  :ooVW!Heg  w !vve}2277 FLL" ' 4A!//   "6!:!1U8 !%%- !%%. !%%.";";G !:x**3//AEE&MM%oo !$ #AeHdAfIdOD   ?5wN .. HH3u:(83#e*jYxjzO  a  $&! ! MJ4 Z< sA]%\5]% :]%\8]%2\;3]%!]%21]%$\>]%9]% ] ]& ]%B<]%]%#]% ]% ]%)]%> ]%]]%9]% A]%"(]% A;]%A]% A]%-1]%]]%=]% ]%]%2 ]%]]%0 ]%]%*A-]% ]%%]&]% ]%#7]%]! ]%;]#<:]%8]%;]%> ] ]% ] ]%]%]%]%!]%#]%)d)r)N)Arjsonrrrrrrtypingrrfastapir r r r r rrpydanticr sqlalchemyrsqlalchemy.ext.asyncior app.auth.depsrrapp.dbrapp.jobs.schedulerrrrapp.schemas.companyrrapp.schemas.processrrrapp.services.mj_people_parserrapp.services.nifrrouterr#postrNrrSr.rirrrrrrrrrrrr2r7r6r3rCs 88!QQQ/4 E.08) (' 3%I%  s%ij=A$($62!2 !! T.-ABC8 .@A%)I2B2j %&'L2  Q!Q $%R&RO""J[4*N''T#)# "i "\eer6