+ Hqj#^RIt^RIt^RIt^RIHtHtHtHt^RIHt^RI H t ^RI H t ^RI Ht^RIHt^RIHt^R IHt^R IHtHt^R IHtHt^R IHtHtHtHt^R I H!t!^RI"H#t#^RI$H%t%^RI&H't'H(t(H)t)H*t*^RI+H,t,^RI-H.t.H/t/^RI0H1t1H2t2H3t3^RI4H5t5^RI6H7t7^RI8H9t9^RI:H;t;^RIH?t@^RIAHBtBHCtCHDtDHEtE^RIFHGtG^RIHHItIHJtJHKtKHLtLHMtMHNtN^RIOHPtPHQtQHRtRHStS]P!]U4tV] !RR7tWR R!ltXR"^R#^R$R/R%R&lltYRlR'R(lltZRmR)R*llt[^t\RnR+R,llt]R-R.lt^RoR/R0llt_RpR1R2llt`R3R4ltaRqR5R6lltb^ZtcR7R8ltdRrR9R:llteR;tfRsR<R=lltgRtR>R?llthRuR@RAlltiRvRBRClltjRwRDRElltkRFRGltlRHRIltmRJRKltnRLRMltoRmRNROlltpRxRPRQlltqRRRSltrRTRUltsRVRWlttRXRYltuRZR[ltvR\R]ltwRoR^R_lltx0RymtyR`RaltzRoRbRcllt{RdRelt|RfRglt}RhRilt~RjRkltR#)zN)datedatetime timedeltatimezone)Any)AsyncIOScheduler)text) AsyncSessionAsyncSessionLocal) random_delay)scrape_cire_for_nif)PAGE_CAPCireAnnouncementSweeper)IrnVersionChanged MjIrnClient)discover_tribunal_codesscrape_distribuicao_range"scrape_distribuicao_tribunal_sweepshort_search_term) lookup_nif)sync_impic_contracts)sync_alvara_type)fetch_and_archive_parte_efetch_dre_for_companiesretroactively_match_companysave_new_publications)upsert_announcement)parse_announcement_text pdf_to_text)carregar_distintivos match_filings upsert_filing) mark_checked)upsert_process)refresh_all_risk)dispatch_pending)match_announcement) sync_company)apply_identity_merges parser_healthrefresh_degreessuggest_identity_links)refresh_incident_summary) enqueue_daysprocess_contentrefresh_acts_countrefresh_priorities sweep_dayssweep_progress)enqueue_corporate_neighbours process_queueseed_all_companiesseed_monitoredz Europe/Lisbon)rcjV^8dQhR\R\R\\\3,R\/#)sessionsourceparamsreturn)r strdictr)formats"app/jobs/scheduler.py __annotate__rCHs1lCc3hTWc "VP\R4RVR\P!V\R7/4GRjxL P 4pVP 4GRjxL V^,#L3L5i)z INSERT INTO scraping_logs (source, started_at, status, params) VALUES (:s, now(), 'running', CAST(:p AS JSONB)) RETURNING id::text sp)defaultN)executer jsondumpsr?firstcommit)r;r<r=rows&&& rB _log_startrOHsooo  &#tzz&#> ?   eg ..  q6M s"=A7A3%A7%A5&A75A7 rows_seenrows_newerrorcrV^8dQhR\R\R\R\R\R\R,RR/#) r:r;log_idstatusrPrQrRNr>)r r?int)rAs"rBrCrCYsR         : rDc"TP\R4RTRTRTRTRT;'gRR,;'gR/4GRjxL VP4GRjxL R#LL5i) z UPDATE scraping_logs SET finished_at = now(), status = :st, rows_seen = :rs, rows_new = :rn, error = :err WHERE id = :id idstrsrnerr:NiNN)rIr rM)r;rTrUrPrQrRs&&$$$$rB _log_finishr^Ysy //    & & ) ( EKKR'//4   " .. #"s(8A) A)A%A)A'A)'A)c V^8dQhR\R\\R3,R\R,R\R,R\\ \\ 3,,/#)r:r;types. company_idNisolatedr>)r tupler?boollistr@r)rAs"rBrCrCvsY<< < c?<d <Tk< $sCx.<rDc "Ve5VP\R4RV/4GRjxL P4pMURpR\V4/pVRJdRpMVRJdR pVP\R V R 24V4GRjxL P4pVUu.uF)pR V^,R V^,RV^,RV^,/NK+ up#LLJuupi5i)aCompanies eligible for scheduled scraping. Analysis-type companies are one-off and excluded by default. Internal first so partial failures still cover priority work. `company_id`: when set, return just that single company (used for on-create backfill). Still respects `monitored = TRUE` but ignores `isolated`. `isolated`: False excludes companies flagged `scrape_isolated` (the default for the main batch), True returns ONLY isolated, None means no filter.Nz SELECT id::text, nif, legal_name, monitoring_type FROM companies WHERE id = :cid AND monitored = TRUE cidr]r`TzAND scrape_isolated = TRUEFzAND scrape_isolated = FALSEa  SELECT id::text, nif, legal_name, monitoring_type FROM companies WHERE monitored = TRUE AND active = TRUE AND monitoring_type = ANY(:types) zb ORDER BY (monitoring_type = 'internal') DESC, legal_name rXnif legal_namemonitoring_type)rIr allre)r;r`rarbrows iso_clause params_sqlrs&&&& rB_monitored_companiesrpvs &// #   #%  &-tE{%; t 5J  6J// "l#     #% "Y] X\STqtUAaD,!6G1NX\ G (  s4%CCAC=C>C/C CC CcJV^8dQhR\R\R\R,/#)r:r;rbr>N)r rdr)rAs"rBrCrCs(## #%)# D[#rDc"TP\R4RV'dRMR/4GRjxL P4pV'd V^,#R#L%5i)zLatest date_to covered by a successful (or partial) distribuicao run. Isolated and non-isolated runs have separate catch-up histories so they don't clobber each other's progress.a SELECT (params->>'date_to')::date AS d FROM scraping_logs WHERE source = 'distribuicao' AND status IN ('ok','partial') AND params->>'date_to' IS NOT NULL AND COALESCE(params->>'isolated','false') = :iso ORDER BY (params->>'date_to')::date DESC LIMIT 1 isotruefalseNrIr rL)r;rbrNs&& rB"_last_successful_distribuicao_daterwsYoo   hFG 4   eg!"3q6"d"! s+AAA Ac V^8dQhR\R,R\R,R\\R3,R\R,R\R,R\/#) r: date_fromNdate_tor`.rarbr>)rrcr?rd)rAs"rBrCrCsd[[d{[ D[[ c?[d [ Tk [  [rDc"\P!4pV\^R7, pVRJ;'dVRJpV'd#V'dV\\R7, pTpMV'd\ 4;_uu_4GRjxL p\ VRR7GRjxL p RRR4GRjxL X 'd8W8d2\ V \^R7,V\^R7, 4pTpMT;rM VfTpMVfTpVeVfQhW8dYrV\\R7, p W 8d\PRW 4T p\ 4;_uu_4GRjxL pRVP4RVP4R\V4R TR RR V'dR MR /p V'd W;R&RV R&\VRV 4GRjxL p \WVRR7GRjxL p RRR4GRjxL V'dcX 'gF\ 4;_uu_4GRjxL p\VX R^^R7GRjxL RRR4GRjxL V #\X WV4GRjxL #\X W4GRjxL #ELELEL +GRjxL 'giEL;iEL3LLL +GRjxL 'giL;iLLL~ +GRjxL 'giX #;iLLt \ dp\P#RT4\ 4;_uu_4GRjxL p\TX R\%T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i)uYScrape Distribuição for a date range. Two modes: * **Sweep** (default) — one HTTP pass per tribunal without a party filter; each row matched locally against every monitored company's distintivo. ~50× fewer requests than the old per-company mode. Progress is tribunal-scoped. * **Per-company** (when `company_id` is passed) — legacy path, used for on-create backfill of a single new company (faster than warming up a full sweep just for one NIF). When both `date_from` and `date_to` are None (scheduled run), looks up the last successful scrape's `date_to` and scrapes from (last + 1) through yesterday. Capped at 30 days back. If no history: yesterday only. Manual windows hard-capped at 180 days (CITIUS serves only ~6 months). Commits per-tribunal (sweep) or per-company (legacy) — survives mid-run crashes either way. daysNF)rbz %s (CITIUS 180d limit)ryrzr`autorbmode per_companysweepraT backfill_new distribuicaor`rarbokrUrPrQzrun_distribuicao failed: %rrRrUrR)rtodayrDISTRIBUICAO_MAX_LOOKBACK_DAYSr rwmaxloggerinfo isoformatrerOrpr^_run_distribuicao_per_company_run_distribuicao_sweep Exception exceptionrepr)ryrzr`rarbr yesterday auto_moder;last capped_from params_logrT companieses&&&&& rBrun_distribuicaors4 JJLE q))IT!55goIZ /M NN  $&&&';GeTTD'& D$D9!#44i)QSBT6TUIG"+ +I     W%88 8$7I+IJJK J    """g ,,. w((* T%[ I  ZMW &  '1| $)-J~ &!'>:FF. Z$  #"" ,...'%gvdaZ[\\\/. 6 g  -VYHHHi'T'&&&.#G #"""(/\/...  I 6:$&&&'gvgT!WM M M'&& '&& s AM-M- !M--M-I'M-I0I*I0 M-(I-) M-5B%M-J M-AJ$J=J>JJJ M-#J$M-)K1K9KJ2KJ8'J4(J8, K7J68K=M->KKKM-K"K#K&M-*I0-M-0J 6I97 J J  M-JJM-J/ J J/ (J/ *M-2K4J86K8K >K? K K KM-KK M*$,M%L M%M 1L4 2M 7 M%MM% M" M M" M" M%%M**M-cHV^8dQhR\R\R\R\/#)r:rTryrzr>)r?r)rAs"rBrCrC0s6@@ @@@ @rDc aaaa"\4GRjxL p\W!, P^,4Uu.uFqA\VR7,NK pp\ 4;_uu_4GRjxL pVP \ R4R\V4\V4,RS/4GRjxL VP4GRjxL RRR4GRjxL ^;o;op^oRVVVV3RllpVFGp \W9WR7GRjxL wrp SV , oSV , oW|, pS\V4, oKI V'dR MR p V'dV R 2MRp\ 4;_uu_4GRjxL p\VSV SSVR 7GRjxL RRR4GRjxL S#ELuupiEL7LLL +GRjxL 'giL;iLL^LGL9 +GRjxL 'giS#;i5i) uVarrimento: uma passagem por tribunal, tudo guardado, cruzado a seguir. O `companies` já não é preciso aqui — o cruzamento deixou de viver dentro do scraper e passou para o `services/court_filings.py`, para poder voltar a correr sobre o arquivo. Uma empresa sem distintivo utilizável já não impede a recolha: antes, se nenhuma desse um distintivo seguro, a corrida terminava sem ir buscar nada, e a informação do dia perdia-se. **Um dia de cada vez, mesmo quando a janela é maior.** O apanhar-atrasos pode pedir até 30 dias de uma vez; numa consulta só, o Porto sozinho passava dos mil processos e aproximava-se do tecto de páginas — que trunca e regista `ok` à mesma. É a armadilha que o varrimento dos anúncios já pagou uma vez, e não se paga duas. Nr|;UPDATE scraping_logs SET progress_total = :t WHERE id = :idtrXc@V^8dQhR\R\R\RR/#)r:feitosgar>N)rV)rAs"rBrC-_run_distribuicao_sweep..__annotate__Rs(######rDc `<"\4;_uu_4GRjxL pVP\R4RSV,RSV,RSV,RS/4GRjxL VP4GRjxL RRR4GRjxL R#LnL0LL +GRjxL 'giR#;i5i)N_UPDATE scraping_logs SET rows_seen = :rs, rows_new = :rn, progress_current = :pc WHERE id = :idrZr[pcrXr rIr rM)rrrr;acertos feitos_antes guardadasrTs&&& rB progresso*_run_distribuicao_sweep..progressoRs$&&&'//= )a-'A+,/&    .." " "'&&  #'&&&snB.B B.G1 G1G1/G0G13GG G G1GG1"G1*F0,F0.G10G 6F97 G G  G1G1GG1G. G G. &G. ( G1c V^8dQhR\\\\3,,R\R\R\R\ \ \ \ 3,/#)r:rryrzrr>)rer@r?rrrcrV)rAs"rBrCrCssR9&9&DcN#9&9&9& 9&  3S= 9&rDc `aaaa"^RIHp\P!VP4o\ 4;_uu_4GRjxL p\ V4GRjxL pRRR4GRjxL ^p^p^o^p RVVVV3Rllp VU u.uFp \P!V !V 44NK! p p \P!V 4EF p V GRjxL pV ^, p V'd\ 4;_uu_4GRjxL p.pVF+p\VV4GRjxL wppVPV4K- VP4GRjxL \W_X4GRjxL pVP4GRjxL RRR4GRjxL V\V4, pVXR,VR,,, pV'gKV!WV4GRjxL EK WxS3#ELELELu +GRjxL 'giEL;iuup iEL7ELLLLLL +GRjxL 'giL;iLd5i)uVarre os tribunais dados, guarda tudo e cruza. (guardadas, acertos, falhas). Partilhado entre a corrida diária e o backfill, para não haver duas versões da mesma sequência — guardar primeiro, cruzar depois — a divergir em silêncio. settingsNcV^8dQhR\\\3,R\\\\3,,/#)r:rr>)r@r?rer)rAs"rBrC'_varrer_e_guardar..__annotate__s/  S#X 4S#X+? rDc<"S;_uu_4GRjxL \VR,VR,SS4GRjxL uuRRR4GRjxL #L;LL \dGpS^, o\PRTR,T4.uRp?uuRRR4GRjxL #Rp?ii;i +GRjxL 'giR#;i5i)Nvaluelabelz)distribuicao sweep tribunal=%s failed: %r)rrrwarning)rrryrzrsems& rBworker!_varrer_e_guardar..workers33 ?gJ' Iw33  ! JAgJXYZ 33   333sCACB(AAA CA CAC B%(B B%B( CBC B%%B((C .B1/ C :C < Chits_nif hits_nome) app.configrasyncio SemaphoreSCRAPE_CONCURRENCYr r! create_task as_completedr#appendrMr"r)rryrzr _settingsr; distintivosrrrrrtaskscorolinhasidsrN filing_id_novo resultadorrs&ff& @@rBrrss1   I88 9C!"""g099 #"IG F F  6? ?YW  +YE ?$$U+!  (***g!C-:7C-H'H$IuJJy)"nn&&&#0k"JJ nn&&&+* V $I y,y/EE EG ;f9 9 9#,$ v %%S#9#""", @+(I&K&+*** :s%=H.GH.GGG H.&G'H.%G;(!H. H H.H.5H 6H.9HH /HH HH H,H -H1 H.<H=9H.; H.H,H.GH.G8 $G'% G8 0G8 2H.H.HH H HH.H) H H) "H) $ H.cVV^8dQhR\R\R\R,R\/#)r:max_daysdias_vazios_para_parar comecar_emNr>)rVrr?)rAs"rBrCrCs:SSSSt S SrDc "\4;_uu_4GRjxL p\VRRV/4GRjxL pVP\R4RVRV/4GRjxL VP 4GRjxL RRR4GRjxL \ 4GRjxL pT;'g'\ P!4\^R7, p^p^p^p ^p W8EdWq8EdVP4^8dV\^R7,pK8\WVV4GRjxL wrp W, pW, p V ^, p V ^8Xd V^,M^p\PRYkY'dR V R 2MR 4\4;_uu_4GRjxL pVP\R 4R VRV RV RX/4GRjxL VP 4GRjxL RRR4GRjxL V\^R7,pEK#RV\^R7, 2p\4;_uu_4GRjxL p\VXRWV RV 2R7GRjxL RRR4GRjxL \PRWV4X#ELPEL<ELELEL +GRjxL 'giEL;iELELELLLL +GRjxL 'giL;iLLLx +GRjxL 'giL;i \dp\PRT4\4;_uu_4GRjxL p\TXR\!T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i)uEnche o arquivo para trás, um dia de cada vez, até a fonte secar. **Um dia de cada vez, nunca uma janela larga.** O paginador do CITIUS pára aos 100 ecrãs e a consulta devolve `ok` na mesma; foi assim que o varrimento dos anúncios deu janelas por cobertas sem o estarem. Um dia inteiro do país são ~1.000 processos, longe de qualquer tecto — e o scraper compara agora o que guardou com o "N Processos encontrados" da fonte, portanto uma truncagem passa a gritar em vez de passar despercebida. **Pára quando a fonte deixar de servir**, não numa contagem de dias codificada: são três dias úteis seguidos a zero. Medido a 03/08/2026, o limite caía a ~04/02/2026, mas isso desliza todos os dias e não é nosso. Fins-de-semana nem se consultam — não há distribuição e só gastavam pedidos. O `comecar_em` serve para retomar: um deploy mata a corrida e sem isto repetiam-se horas de pedidos já feitos, só para chegar ao mesmo sítio. A gravação é idempotente de qualquer forma, portanto repetir não estraga — apenas custa. Ndistribuicao_backfillrrrrXr|z1backfill distribuicao %s: %d linhas, %d acertos%sz, z tribunais falharamr]rrZr[ruaté rz dias, rz7backfill distribuicao terminado: %d dias, %d linhas, %sz$run_distribuicao_backfill failed: %rrRr)r rOrIr rMrrrrweekdayrrrr^rrr)rrrr;rTrrvaziostotal_guardadas total_acertos dias_feitosrrralcancers&&& rBrun_distribuicao_backfillrs0!"""g! ,z8.D  oo N O (D& )   nn#"1133 >>TZZ\I1,==  $)H{{}!ya((/@QT/U)U &I  (O  $M 1 K#,>VaZqF KKCV2fX5H)IY[ )***gooA?D-;f6 nn&&&+* 9!$ $C# q 1123$&&&')$ WWI6  '&  E '  m#   #"""4*V+'+***' '&&& ?C$&&&'gvgT!WM M M'&& '&& sM6I)M6I8I,&I8I/I8/I20I84 M6?I5M6 K"J K" A6K"JA-K"JK"'J$/J0J$J J$ K"J"AK"KK""K;K<K K" K K"(M6,I8/I82I85M68J >J? J J M6K"K"K"J$ J$"K"$J= *J-+ J= 6J= 8 K"KK"K K K K K"" M3-,M.L M.M:L= ;M M. M M.M+ M M+ $M+ &M..M33M6c V^8dQhR\R\\\\3,,R\R\R\/#)r:rTrryrzr>)r?rer@rr)rAs"rBrCrCsNSS SDcN#SS S  SrDcaaaaaaa"\4GRjxL p^p^p^o\4;_uu_4GRjxL pVP\R4R\ V4RV/4GRjxL VP 4GRjxL RRR4GRjxL ^RIHp\P!VP4oRVVVV3Rllo\V^4EFwp o\SR,4oS'g2\PR V \ V4SR,R ,4KR\P!VVV3R lV4R R /GRjxL p V U U u.uF qFqNK K p p p V 'd^p\4;_uu_4GRjxL pV F,p \!W|4GRjxL wppV'gK#V^, pK. VP 4GRjxL RRR4GRjxL V\ V 4, pWn, p\4;_uu_4GRjxL pVP\R4RVRVRV RV/4GRjxL VP 4GRjxL RRR4GRjxL EK S'dRMRpS'dS R2MRp\4;_uu_4GRjxL p\#WpVVVVR7GRjxL RRR4GRjxL V#ELELELsEL^ELQ +GRjxL 'giELi;iELuup p iELELlEL=EL0 +GRjxL 'giELH;iELLLL +GRjxL 'giEK;iLLL +GRjxL 'giT#;i5i)u Legacy per-company mode, retained for on-create backfill (single empresa → no point in warming up a full tribunal sweep). Kept aligned with the archived v1 code under scrapers/_archive/ — so a future revive of the old mode for bulk runs is a one-commit restore.NrrrXrc V^8dQhR\\\3,R\\\3,R\R\\\\3,,/#)r:companyrsearchr>)r@r?rre)rAs"rBrC3_run_distribuicao_per_company..__annotate__sIc3h$(cN.workers33 6gJ' IwIF33  q DL)1W:q 33   333sCACB4#AAA C A CAC B1#0B,B1B4 C%B(&C,B11B44C :B=; C C  Criz7distribuicao [%d/%d] %s: SKIPPED (ambiguous/short name):N(Nc38<"TFpS!SVS4xK R#5iN).0rrrrs& rB 0_run_distribuicao_per_company..:s <)QfWa(()sreturn_exceptionsFrrZr[rrrz" (company,tribunal) pair(s) failedr)rr rIr rrMrrrrr enumeraterrrgatherr%r^)rTrryrzr total_seen total_newr;ridxresultsrorNmatches company_new_is_newrUrrrrrrs&&ff @@@@@rBrrs.//IJIK """goo N O #i.$ /   nn #"1   I88 9C$")Q/ W"7<#89 KKIS^W\%:3%?   <) < #  7>(Kg#g(K K(***g"C&4W&B BIAvv#q( #nn&&& +* #g, &J  $I$&&&'//uz4D#tVL    .." " "'&&/0@&Y4FDO ?@UYG """g Fjg   #" MQ0 #   #"""H )L+ B' +***'  #'&&&# #""" MsM=K M=KM=,K$(K)K$KK$ M=K!B9M= L M=L&#M= L M= L!L " L3 LL L M=#L$4M=L3 M='L<L6 L<L8 L< M=+L:,M== M= M=&M'M=*M >M?M  M=MM=M=K$K$!M=$K> *K-+ K> 6K> 8 M=M= LLM=L0 L L0 (L0 * M=6L<8L<:M=<M M M M  M=M M= M: &M)' M: 2M: 4 M=c V^8dQhR\R\\R3,R\R,R\R,R\/#)r:r}r`.raNrbr>)r?rcrd)rAs"rBrCrC[sQEE E c?Ed ETk E  ErDc8 "\4;_uu_4GRjxL pRTR\V4RVe \V4MR/pV'd W%R&RVR&\VR V4GRjxL p\ WAW#R 7GRjxL pRRR4GRjxL \4;_uu_4GRjxL pVP \ R 4R \X4R X/4GRjxL VP4GRjxL RRR4GRjxL ^p^p Rp \X^4EF*wr\V R,V R ,VR7GRjxL p V\V 4, pV 'dw\4;_uu_4GRjxL pV F,p\WN4GRjxL wppV'gK#V ^, p K. VP4GRjxL RRR4GRjxL M\4;_uu_4GRjxL pVP \ R4RVRV RV R X/4GRjxL VP4GRjxL RRR4GRjxL EK- \4;_uu_4GRjxL p\YGU u.uF qR ,NK up 4GRjxL \!YGU u.uF qR ,NK up 4GRjxL \#VXWV R7GRjxL VP4GRjxL RRR4GRjxL V#ELELEL~ELq +GRjxL 'giEL;iELuELHEL3EL& +GRjxL 'giEL>;iEL ELELELEL +GRjxL 'giEL;i \d,p\PRT R,T4Rp Rp?ELRp?ii;iELELELELx +GRjxL 'giEK;iELyuup iELXuup iEL7EL#ELEL +GRjxL 'giX#;i \dp\PRT4\4;_uu_4GRjxL p\#TXR\%T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i)zScrape CIRE for every monitored company. `days` is '15', '30', or 'todos'. `types` filters which monitoring_type categories to include. `company_id`: restrict to a single company (used for on-create backfill). `isolated`: see `run_distribuicao`.Nr}r`rbFraTrcirerrrrXrrhr|zcire nif=%s failed: %srrrZr[rrzrun_cire failed: %srRr)r rerdrOrprIr rrMrrr%rrrr$r&r^r?)r}r`rarbr;rrTrrrrUrcrrNrrrs&&&& rBrun_cirer[s!"""g D T%[ (*>XE& '1| $)-J~ &!'6:>>. Z  #".$&&&'//RSc)ndF3  .." " " '&    1-FC # 3AeHagD QQc'l* 0222g#*C.64M**M +M*.M M  M  M 4M 5M 9 M*M M* P!N# "P%'N/ N& N/$N) %N/) P4N,5PO P O)"O3 O)=O> O) O O)&O'O)=O >O)O#O) P%O&&P+R/K82K85R8L >L? L L RPL!L!P!L; 'L*( L; 3L; 5 P>M*M*M M  M* M' M M' M' !M*%P'M** N 5 NPN  P&N/)N/,P/O 5N86 O O  PO)O) O)#O)&P)P /O20 P ;P =PRP R,R=Q >RQ6Q! Q6$ R/Q20R6R <Q?= R R RRRc>V^8dQhR\R\R,/#)r:r;r>N)r r)rAs"rBrCrCs##l#td{#rDc"VP\R44GRjxL P4pV'd V^,#R#L%5i)aV SELECT (params->>'date_to')::date AS d FROM scraping_logs WHERE source = 'cire_announcements' AND status IN ('ok','partial') AND params->>'date_to' IS NOT NULL ORDER BY (params->>'date_to')::date DESC LIMIT 1 Nrv)r;rNs& rB"_last_successful_announcement_dater sIoo     eg3q6"d" sAAA AcXV^8dQhR\R,R\R,R\/#)r:ryNrzr>)rr?)rAs"rBrCrCs3AAd{A D[A ArDc "\P!4pVfTVfP\\4GRjxL pV\ ^R7, pV'dV\ ^R7,MTpW8dTpT;'gV\ ^R7, pT;'gV\ ^R7, pW, P \ 8dV\ \ R7, p\W, P ^,4Uu.uFq@\ VR7,NK pp\4;_uu_4GRjxL p\VRR\V4R\V4R\V4/4GRjxL pVP\R4R\V4R V/4GRjxL VP4GRjxL RRR4GRjxL ^p^p ^p R p \4;_uu_4GRjxL p \!V^4EF?wr^pV P#W4RjxL pV^, pVR ,FcpVP%R 4'gKV P'VP%R 44GRjxL p\)V4pVVR&\+V4VR&Ke \4;_uu_4GRjxL pVR ,FRp\3VV4GRjxL wppT^, pT'gK.T ^, p T \7TT4GRjxL , p KT VP4GRjxL RRR4GRjxL EK( RRR4GRjxL \.P=RW\V4WW4V 'd@\4;_uu_4GRjxL p\?V4GRjxL RRR4GRjxL \4;_uu_4GRjxL p\AVXWV R7GRjxL RRR4GRjxL V#ELuupiELELELlELWELJ +GRjxL 'giELb;iELFELEL \,d3p\.P1RTP%R4T4Rp?EK@Rp?ii;iELEL \,d%p\.P5RT4Rp Rp?EKRp?ii;iELELEL +GRjxL 'giEK;iEDM1 \,d$p\.P5RTT4Rp Rp?MRp?ii;iT\88d\.P;RT4Rp \4;_uu_4GRjxL pTP\R4RTRT RT R X/4GRjxL TP4GRjxL RRR4GRjxL EK +GRjxL 'giEK;iELx +GRjxL 'giEL;iELRELBEL5 +GRjxL 'giELM;i \,d"p\.P5RT4Rp?ELwRp?ii;iELiELUELH +GRjxL 'giX#;i \,dp\.P5RT4\4;_uu_4GRjxL p\ATXR\T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i) ulSweep nacional dos anúncios CIRE do CITIUS. Sem filtro de entidade: varre tudo o que foi publicado no intervalo e guarda, dê ou não match hoje. É o que permite detectar a insolvência de um devedor que ninguém tinha adicionado, e o que dá histórico ao cruzamento retroactivo quando entram clientes novos na watchlist. Sem argumentos (corrida agendada) apanha desde o dia a seguir ao último `date_to` bem-sucedido até ontem; sem histórico, só ontem. Janelas manuais são limitadas a 90 dias por corrida. **Varre dia a dia**, não a janela toda de uma vez. Uma janela de 90 dias tem ~18 mil anúncios e batia no tecto de páginas do sweeper, que truncava em silêncio e mesmo assim registava 'ok' — pior do que falhar, porque o catch-up dava a janela por coberta e os dias em falta nunca mais eram revisitados. Um dia são ~11 páginas, longe de qualquer tecto. Nr|cire_announcementsryrzrrrrXrrl wants_pdf pdf_tokenpdf_textdetailuanúncio %s: PDF falhou: %s referenciauupsert de anúncio falhou: %srz cire_announcements %s falhou: %su1cire_announcements %s: tecto de páginas atingidorrZr[rzFcire_announcements %s..%s (%d dias) -> vistos=%d novos=%d hits=%d [%s]u*envio de avisos de insolvência falhou: %srz!run_cire_announcements failed: %srRr)!rr_run_in_sessionr rr}CIRE_ANN_MAX_WINDOW_DAYSrr rOr?rrIr rMrrpagesgetfetch_pdf_bytesr rrrrrrr(rrRrr'r^)ryrzrrrrr;rTrr total_hitsrUsweeperdia_idxrpaginaspagerNpdftext_pdfrann_idrs&& rBrun_cire_announcementsrs* JJLEW_$%GHH)++26TI1--G  I88eiQ&77I44%)"33G!!$<<i-EFF 38':M9S9SVW9W3X Y3Xa q) ) )3XD Y """g!  #i.)S\63t9 U   oo N O #d)T6 *   nn#"JIJ FN*,,, )$ 2 "'&-mmC&=33d1 #'<<!'$$%GaP&F'h&LL!TVYZ&F,...'!//Ez4D'4QWX "..***/.....[-,,,~/3/... R  !MqQQ R' '&&& 3[+62P (P)/P PP 0P1P 5 [+P  [+Y$P*%Y(V?>SSP- S0S#P3 +P0,"P3 S(Q3)S, R4 :Q9 Q6 Q9R4 $R4 =R+>!R4 R. R4 $ S/R1 0S7V?9 YV<.Y4X W XW%WW%" X-W".X2YX1 Y X:X4 X:$ Y/X70Y5[+9[+P P P  [+ P' P P' P' ! [+*Y-S0P3 3 Q0 >&Q+ $S+Q0 0S6Q99 R(R#R4 #R((R4 .R4 1S4S:R=; SS SV? T S> 9V?>T )r?rcrV)rAs"rBrCrCBsFaad a c?a4Za arDc "\4;_uu_4GRjxL p\VRR\V4RVRV/4GRjxL pV'dRMRpVeV'gR\V4 R 2MRpTP \ R V R V R 24R\V4/V'dR V/M/C4GRjxL P 4pRRR4GRjxL ^;p;rRp \4;_uu_4GRjxL pVP \ R4R\X4RX/4GRjxL VP4GRjxL RRR4GRjxL \4;_uu_4GRjxL p \X^4EFwp wrp\4;_uu_4GRjxL p\W z))z SELECT c.id::text, c.nif, c.legal_name FROM companies c WHERE c.monitored = TRUE AND c.monitoring_type = ANY(:types)  zb ORDER BY c.registry_fetched_at NULLS FIRST, c.legal_name rgrrrrXseennewpeoplez mj_irn: %srRrzmj_irn nif=%s (%s) falhou: %srrrZr[ruBmj_irn: %d empresas, %d publicações vistas, %d novas, %d pessoasrzrun_mj_irn failed: %s)r rOrerVrIr rkrrMrrsync_mj_companyrrrRr^r?rrr r)rar`r#r;rTclausestalerrr total_peoplerUclientrrgrhnamerFrs&&& rB run_mj_irnr1BsM!"""g! X d5k<\? [  '1"b*: %%(%9$:" > //haw' $u+U5**=QSU   #% #"6-.-J- F6$&&&'//RSc)ndF3  .." " " '&===F)29a)@%%c'0222g"1'3"LL%nn... 32!F)+J5)I AhK/L-...'!//Ez4D#tVT "..***/.#n$$9*A!=>  P  NJ< %&&&'y  '& Y#  #"""<'  # '&&&! 3L. 3222 )LLq10222g)'6'QTUVQWXXX 322 322 '$$%Dc4QRS&F'/+/...%;!===F' '&&&  0!4$&&&'gvgSVL L L'&& '&& sU K:U  LK= L LAL)L*L< U L U R,*L#+R,.,L/L&L/2L)3L/7 R,L,R,M R,!Q)9M8MM8M #M$M ;M<M M8 M 4M8Q)P:Q)'Q P=Q QQ  Q))Q *Q);Q# < Q) R,Q&;R, R R,R#R $R( R,3R 4R,9U =LLU L L L L  U #R,&L/)L/,R,/M 5L86 M M  R,M8M M M8M5!M$" M5-M5/ M88 P7 ,P /N20P 4O( OO(  P !O$ "P (P.O1/ P:P<P  P7 P7 P2 ,Q)2P7 7Q)=Q Q Q)Q  Q Q Q  Q)&R,)R /Q20 R ;R = R, R R,R) R R) !R) #R,(U )R,,UU,U.S1 /U3T'T T' U T#!U'U -T0. U 9U ;UUU cHV^8dQhR\R\R\R\/#r: max_items max_secondsseedr>rVfloatrdr?)rAs"rBrCrCs2FFF*/FAEFFrDc "\4;_uu_4GRjxL p\VRRVRVRV/4GRjxL pV'dB\V4GRjxL pVP4GRjxL \P RV4VP \R44GRjxL P4;'g^pVP \R4R VR V/4GRjxL VP4GRjxL RRR4GRjxL \WR R 7GRjxL p\4;_uu_4GRjxL pVP \R44GRjxL P4;'g^p\TXV'dR MRVR,VR,V'dV R2MRR7GRjxL RRR4GRjxL \P RVX4X#ELELELELELJELLL +GRjxL 'giEL;iLLLLfLX +GRjxL 'giLo;i \dqp \4;_uu_4GRjxL p\TXR\T 4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp ? i\dp \PRT 4\4;_uu_4GRjxL p\TXR\T 4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp ? ii;i5i)uFase A: listar o histórico por NIPC de todas as entidades colectivas. **Só lista.** Uma chamada por empresa, sem abrir corpos — medido, 102 empresas/min, portanto as ~507 mil que não têm histórico próprio são cerca de 83 horas. Os actos que aparecerem ficam na fila de conteúdo e são abertos depois, pela escada de prioridades que já põe a cap table mais recente de cada entidade à frente do resto do histórico. Porquê: a listagem diária do IRN **não é um índice completo**. Numa amostra de 30 empresas, 28 tinham actos que nunca tínhamos visto, e só temos ~1.200 prestações de contas por ano num país onde centenas de milhares as entregam. A fila vive na base de dados: isto pára ao fim do orçamento e retoma na noite seguinte onde ficou, sem repetir nada. Nregistry_nacionalr4r5r6z(registry_nacional: %d entidades semeadas_SELECT count(*) FROM registry_entity_fetch WHERE status IN ('pending','error') AND attempts < 3rrrXF)r4r5 fetch_contentrr entidadesnovosz entidades por listarrz%registry_nacional: %s | %d por listarrRrz run_registry_nacional failed: %s)r rOr7rMrrrIr scalarr6r^rr?rr) r4r5r6r;rTpostos pendentestotalsrestamrs &&& rBrun_registry_nacionalrDs$!"""g! ( )]K N   -g66F.." " " KKBF K//L  &( oo N O )T6 *   nn+#".$  %&&&'ooP fh $*y -w:@ 56d   '&  ;VVL W# 7 "   +#"""0 ' '&&&" $&&&'gvgSVL L L'&& '&& ;Q?$&&&'gvgSVL L L'&& '&& s"MGMG3G  G3G3G#G3'G&(7G3G) G39%G3G,G36G/7G3; MG1M H6HH67H8H6;HHH3,H H/H0H4 H6?HH6M G3#G3&G3)G3,G3/G31M3H 9G<: H H  MH6H6HHH6H3 H#! H3 ,H3 .H66 MJ,I J,J8I; 9J> J, J  J,J) J J) "J) $J,, M9M:,L;&K) 'L;+LL L L;LL;L8 %L(& L8 1L8 3L;;MMcHV^8dQhR\R\R\R\/#r3r7)rAs"rBrCrCs2CCC&+C<@CCrDc "\4;_uu_4GRjxL p\VRRVRV/4GRjxL pV'dU\V4GRjxL p\V4GRjxL pVP 4GRjxL \ P RWV4RRR4GRjxL \WR7GRjxL p\4;_uu_4GRjxL p\V4GRjxL \V4GRjxL \V4GRjxL VP\R44GRjxL P4;'g^pVP 4GRjxL RRR4GRjxL X'dRMR p Rp \4;_uu_4GRjxL p\V4GRjxL p \TXT VR ,VR ,V'dV R 2MRR 7GRjxL RRR4GRjxL V 'd.V PR4'd\ P!RV 4\ P RWx4X#EL"EL ELELELEL +GRjxL 'giEL;iELELELELELoELPEL#EL +GRjxL 'giEL.;iEL LLL +GRjxL 'giL;i \"dp \ P!RT 4\4;_uu_4GRjxL p\TXR\%T 4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp ? i\&dp \ P)RT 4\4;_uu_4GRjxL p\TXR\%T 4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp ? ii;i5i)uRegisto comercial por NIPC: as monitorizadas e, em BFS, os sócios-empresa. É a via que dá **histórico completo** de uma entidade, porque a pesquisa por NIPC do IRN não tem limite de datas. Complementa o varrimento nacional, que cobre em largura mas só a janela recolhida. Cobre os cinco tipos de monitorização, ao contrário do `run_mj_irn`: as 7 de análise estavam sem qualquer informação de registo e as 17 `related` nunca eram sincronizadas, o que travava o grafo no primeiro hop. Nregistry_expandr4r5z6registry_expand: %d monitorizadas, %d vizinhas na filar4r5r;rrlistadosr>z entidades ainda na filaralarme$registo: extracção suspeita — %szregistry_expand: %s (fila: %d)zregistry_expand: %srRrzrun_registry_expand failed: %s)r rOr8r5rMrrr6r,r-rIr r?r+r^rrRrr?rr) r4r5r6r;rTseededgrownrBpendingrUhealthrs &&& rBrun_registry_expandrPs!"""g! &iP[(\   )'22F6w??E.." " " KKPRX `#",$yRR$&&&'/w7 7 7!'* * *(1 1 1ooP fh .." " "'&$&4$&&&'(11F ,vg I I  II  I!I 8I9I  O'I O'$K3I*4KI-KJ!I0"J4I35JI6!J)I9*JJI<J K)I?* K6KJKJ(%J"&3J(J$J( K)J&* K6K -K:O'>I I I I  O' I' I I' I' ! O'*K-K0J3J6J9J<J?KJ J  J J  K"J($J(&K(K .J1/ K :K <K O$,M;K> <ML4L L4" M-L0.M4M :L=; M M M O$O$,O N OO+N. ,O1 O<N?=OO O  O O OO$$O'c pV^8dQhR\R,R\R,R\R\R\/#)r:ryNrzrr5r>)rrVr8r?)rAs"rBrCrC5sB<<d{< D[<< <  <rDc"\4;_uu_4GRjxL pV'dV'd\W@V4GRjxL pMK\P!4\ ^R7, p\WF\ ^R7, V4GRjxL pVP 4GRjxL \ TRRTRTRV'd \V4MRRV'd \V4MR/4GRjxL pRRR4GRjxL \W#R7GRjxL p\4;_uu_4GRjxL p\V4GRjxL \V4GRjxL \V4GRjxL p VP 4GRjxL RRR4GRjxL X PR 4'dR MR p \4;_uu_4GRjxL p\TXT VR ,VR ,V PR 4'd V R , R2MRR7GRjxL RRR4GRjxL \PRW4X#ELELELELELZELM +GRjxL 'giELe;iELXEL@EL0EL ELLL +GRjxL 'giEL;iLLL +GRjxL 'giL;i \ dqp \4;_uu_4GRjxL p\TXR\T 4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp ? i\"dp \P%RT 4\4;_uu_4GRjxL p\TXR\T 4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp ? ii;i5i)uVarrimento nacional dos actos de registo, dia a dia. É a via que responde a "em que outras empresas é que esta pessoa é sócia": o IRN só pesquisa por NIPC de entidade, e com o NIF de um indivíduo devolve zero. Sem índice nacional, essa pergunta não tem resposta. Um dia de cada vez porque é o único filtro fiável — intervalos explícitos devolvem contagens absurdas e sem filtro devolve zero. Nr|registry_sweeprdias_novos_na_filaryrz)rr5dias_por_fazerrrrIr>z dias por varrerrzregistry_sweep: %s | %srRrzrun_registry_sweep failed: %s)r r/rrrrMrOr?r3r2r1r4rr^rrrrr) ryrzrr5r;queuedaterTrBprogressrUrs &&&& rBrun_registry_sweeprY5s!"""g 'GDDF**,!22C'ya7H1H#NNFnn! % #7 I#i.4 G T ;  #"!8MM$&&&'$W- - -%W- - -+G44H.." " " '&'ll+;<<$$&&&' ,vg ||$455 0122BC;?   '&  -v@ I#DO #"""N& - .4 " '&&&' '&&& $&&&'gvgSVL L L'&& '&& 8!<$&&&'gvgSVL L L'&& '&& swOH2OIIH5A I H8 I!H;",II$H>%I) O4I5O:J3 I! J3$I$%J3(I47I'8I4 I* I4I-I45I06I4: J3I2J3!J3;J<J3?AJJJ J3JJ31O5I8I;I>IOI I  I I  O!J3$J3'I4*I4-I40I42J34J :I=; J J  J3JJ3J0 J  J0 )J0 +J33 N=>L)K L)L 5K8 6L ; L)L L) L& L L& L& !L)) N=6N=7,N8#M& $N8(NN N N8NN8N5 "N%# N5 .N5 0N88N==Oc<V^8dQhR\R\R\/#)r:r4r5r>)rVr8r?)rAs"rBrCrCts"..#.5.VY.rDc "\4;_uu_4GRjxL p\VRRVRV/4GRjxL pRRR4GRjxL \WR7GRjxL p\4;_uu_4GRjxL p\V4GRjxL \ V4GRjxL \ V4GRjxL \ V4GRjxL p\V4GRjxL pVP4GRjxL RRR4GRjxL XPR4'dRMRp\4;_uu_4GRjxL p\TXTVR ,VR ,VR ,,VPR4'd VR, R 2MRR 7GRjxL RRR4GRjxL XPR4'd\PRV4\PRWE4X#ELELEL +GRjxL 'giEL;iELELELELxELhELXELHEL3EL& +GRjxL 'giEL>;iELLL +GRjxL 'giL;i \dqp\4;_uu_4GRjxL p\TXR\T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?i\ dp\P#RT4\4;_uu_4GRjxL p\TXR\T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i)uwAbre e parseia o conteúdo dos actos na fila, por prioridade. A listagem é barata (uma chamada por página) e o conteúdo são três chamadas por acto, portanto só se abre o que traz pessoas ou quotas. A escada de prioridades faz o útil aterrar primeiro: o nosso universo, a vizinhança do grafo, e a cap table mais recente de cada entidade antes do resto do histórico. Nregistry_contentr4r5rHconteudo_por_abrirrractosquotascargosz actos por abrirrrJrKzregistry_content: %s | %srRrzrun_registry_content failed: %s)r rOr0r,r-r*r4r+rMrr^rrRrrr?rr) r4r5r;rTrBrXrOrUrs && rBrun_registry_contentrats!"""g! '+y-Q\)]  #"!&TT$&&&'!'* * *(1 1 1(0 0 0+G44H(11F.." " "'&'ll+?@@d$&&&' /F84DvhGW4W ||$899 4566FG?C   '& ::h   LL? H /B 9# #""" U& * 1 141 "'&&&' '&&& $&&&'gvgSVL L L'&& '&& :A>$&&&'gvgSVL L L'&& '&& s^M/G!M/G*G$G* M/G'M/I"HI"2H 3I"6H"H H"HH"+H,H">H?H"HH")H*H". I"9H:I"I"/H?0I"3AIII I"II"3-I" M/$G*'M/*H 0G31 H <H > M/I" I" H"H"H"H"H"H"I""H< (H+) H< 4H< 6 I"II"I I I I I"" M,-KJ KJ<$J' %J<* K5J86K<K K K K K M,%M,&,M'L M'M 3L6 4M 9 M'MM' M$ M M$ M$ M''M,,M/c$V^8dQhR\/#r:r>r?)rAs"rBrCrCscrDc "\4;_uu_4GRjxL p\VR/4GRjxL pRRR4GRjxL \4;_uu_4GRjxL pVP\P !4\ ^R7, ^R7GRjxL pRRR4GRjxL \4;_uu_4GRjxL p\VXR\X4R7GRjxL RRR4GRjxL V#LLL +GRjxL 'giL;iLLLw +GRjxL 'giL;iLzL\LN +GRjxL 'giX#;i \dp\PRT4\4;_uu_4GRjxL p\TXR\T4R 7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i) uUm handshake e uma listagem, de hora a hora. Quando o IRN faz deploy, os `apiVersion` codificados no scraper ficam obsoletos e tudo passa a devolver 403. Sem isto sabia-se na manhã seguinte, com três jobs falhados; assim sabe-se dentro de uma hora. N irn_canaryr|) max_pagesr)rUrPzirn_canary: %srRr) r rOrpublications_for_dayrrrr^rrrrRr?)r;rTr/rlrs rBrun_irn_canaryrisF!"""g!'<<<#" ===F44TZZ\ISTDU5Uab4ccD!=$&&&'gvdc$iP P P'& #<#"""!c!===& P'&&&  %q)$&&&'gvgSVL L L'&& '&& sG6C;G6DC=D G6C?G6E"DE";D#DD# E")D!*E"D?E"E$E%E) E"4E5E":G6=D?G6D D  D D G6E"D#!E"#D< )D,* D< 5D< 7 E"EE"E E E E E"G6E"" G3-,G.F G.G:F= ;G G. G G.G+ G G+ $G+ &G..G33G6c$V^8dQhR\/#rcrd)rAs"rBrCrCsCrDc "\4;_uu_4GRjxL p\VR/4GRjxL pRRR4GRjxL \4;_uu_4GRjxL p\V4GRjxL pVP4GRjxL RRR4GRjxL \4;_uu_4GRjxL p\ VXRXVR7GRjxL RRR4GRjxL \ P RX4X#LLL +GRjxL 'giL;iLLLL +GRjxL 'giL;iLLtLf +GRjxL 'giL};i \dp\ PRT4\4;_uu_4GRjxL p\ TXR\T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i) u^Incidentes por NIF, para cada nó do grafo poder ser pintado. O cruzamento é com o `cire_announcement_parties`, que é **nacional** — 240 296 partes e 55 321 NIFs distintos — portanto cobre pessoas e entidades que não monitorizamos. É isso que permite ver, a dois hops de distância, que um sócio de um sócio tem uma insolvência a correr. Nentity_incidentsrrzentity_incidents: %d NIFszrun_entity_incidents failed: %srRr) r rOr.rMr^rrrrr?)r;rTrlrs rBrun_entity_incidentsrmsU!"""g!'+=rBB#" $&&&'1'::D.." " "'&%&&&'gvddUYZ Z Z'& /6 #B#"""': "'&&&' Z'&&& :A>$&&&'gvgSVL L L'&& '&& sG1C5G1C;C7C; G1C9G1EDED-D.DDD EDE0D;1E4ED= E ED?E4G17C;9G1;D D D D G1EDDED8 %D(& D8 1D8 3 E=E?EE E  E E E G.(,G)F G)G 5F8 6G ; G)G G) G& G G& G& !G))G..G1c$V^8dQhR\/#rcrd)rAs"rBrCrCsSrDc "\4;_uu_4GRjxL p\VR/4GRjxL pRRR4GRjxL \4;_uu_4GRjxL p\V^R7GRjxL pRRR4GRjxL \4;_uu_4GRjxL p\VXRXVR7GRjxL RRR4GRjxL V#LLL +GRjxL 'giL;iLL}Lo +GRjxL 'giL;iLrL\LN +GRjxL 'giX#;i \dp\ P RT4\4;_uu_4GRjxL p\TXR\T4R7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i) uRede de segurança diária do envio de avisos. O sweep já dispara os avisos dos hits que cria. Isto apanha o que tenha ficado pendente por o SMTP estar em baixo, ou hits criados fora do sweep (por exemplo, quando um NIF novo entra na watchlist e cruza com o arquivo). Ninsolvency_digest)limitrrz run_insolvency_digest failed: %srRr)r rOr'r^rrrr?)r;rTsentrs rBrun_insolvency_digestrss4!"""g!'+>CC#" $&&&')'==D'&$&&&'gvddUYZ Z Z'& #C#"""'='&&&& Z'&&& ;Q?$&&&'gvgSVL L L'&& '&& sGC GCC C GC GD0C+D0C1/C-0C14 D0?C/D0D D0D2D3D7 D0DD0G C GC( C C( !C( #G+D0-C1/D01D 7C:8 D D  D0DD0D- D D- %D- 'D0,G-D00 G;,F<'E* (F<,F F F  F<FF< F9 &F)' F9 2F9 4F<<GGc$V^8dQhR\/#rcrd)rAs"rBrCrCsDDsDrDc"\4;_uu_4GRjxL p\VR/4GRjxL pVP\R44GRjxL P 4pRRR4GRjxL ^pXEF3wrE\ V4GRjxL pVP R4R8XdK2\4;_uu_4GRjxL pTP\R4R\P!VP R44R VP R 4;'gR R VP R 4R VP R4;'gR RV/4GRjxL pVP4GRjxL VP'd V^, pRRR4GRjxL \4GRjxL EK6 \4;_uu_4GRjxL p\VXR\!V4VR7GRjxL RRR4GRjxL \P#R\!V4V4X#ELELELEL +GRjxL 'giEL;iELELLLL +GRjxL 'giL;i \d!p\PRYX4Rp?LRp?ii;iLLLL +GRjxL 'giL;i5i)uEnriquece empresas sem CAE / nome / morada, via VIES + SICAE. Substitui o `run_ptdata_refresh`, que estava a ser um no-op silencioso: só gravava quando `data["source"] == "ptdata"`, e como o VIES responde primeiro a condição nunca era verdadeira. As últimas corridas liam 95 empresas e actualizavam 0 — mas ficavam registadas como 'ok'. Passa a aceitar qualquer fonte e a gravar só o que veio preenchido. N nif_refresha] SELECT id::text, nif FROM companies WHERE cae IS NULL OR ptdata_fetched_at IS NULL OR ptdata_fetched_at < now() - interval '30 days' ORDER BY (cae IS NULL) DESC, ptdata_fetched_at NULLS FIRST LIMIT 400 r<locala UPDATE companies SET ptdata_payload = CAST(:pl AS JSONB), ptdata_fetched_at = now(), legal_name = COALESCE(NULLIF(:ln, ''), legal_name), cae = COALESCE(:cae, cae), address = COALESCE(NULLIF(:addr, ''), address), updated_at = now() WHERE id = :id AND (cae IS DISTINCT FROM COALESCE(:cae, cae) OR ptdata_fetched_at IS NULL OR address IS NULL) plrawlnrir]caeaddraddressrXznif_refresh nif=%s falhou: %srrz&nif_refresh: %d lidas, %d actualizadas)r rOrIr rkrrrJrKrMrowcountrrrr r^rr) r;rTr- refreshedrgrhdataresrs rBrun_nif_refreshrs !"""g!'="==//    #% #""I! D#C(Dxx!W,(***g#OO  djj%9dhh|4::txx 3 9 9rc !0nn&&&<<<NI7+*<nGJ!"""g'6$#e*W`aaa#" KK8#e*iP Mu#= #"""()+0'3+***8 D NN:C C C D"a#"""sKHKH)H !H)H#H)# K.H&/K>I. I I.'K)I.?I I.A?I II I(I )I? I I.II. K%J&KJK J$&J 'J$+ K6J"7'K H)#H)&K)I /H20 I ;I = KI. I. III.I+ I I+ $I+ &I.*K+I.. J9JKJKK J$"K$J= *J-+ J= 6J= 8Kc0V^8dQhR\R\/#)r:forcer>)rdr?)rAs"rBrCrC1s$3rDc "\4;_uu_4GRjxL p\VRRRRV/4GRjxL pRRR4GRjxL \4;_uu_4GRjxL p\WR7GRjxL pRRR4GRjxL \4;_uu_4GRjxL p\TXR\ XP R4;'g/P 44VP R ^4R 7GRjxL RRR4GRjxL V#LLL +GRjxL 'giL;iLLL +GRjxL 'giL;iLL\LN +GRjxL 'giX#;i \dp\PR T4\4;_uu_4GRjxL p\TXR \T4R 7GRjxL RRR4GRjxL h +GRjxL 'gih;iRp?ii;i5i)uBulk sync public contracts straight from IMPIC dumps on dados.gov.pt. The replacement for the old ptdata-based `run_contracts_sync`. Runs weekly (IMPIC refreshes the dumps biweekly). Downloads the per-year ZIPs whose `last_modified` moved forward, stream-parses the JSON, keeps only rows where a monitored NIF appears as adjudicatário or adjudicante, upserts `public_contracts`, and refreshes the summary cache on `companies`. `force=True` re-downloads every year regardless of sync state — used for the initial full-history backfill.N contractsr<zimpic_dados.gov.ptr)rrrows_kept_by_year rows_insertedrz#run_contracts_impic_sync failed: %rrRr) r rOrr^sumrvaluesrrrr)rr;rTstatsrs& rBrun_contracts_impic_syncr1s{!"""g! [8-A7E"R  #"$&&&'.wDDE'&$&&&'uyy)<=CCKKMN?A6  '&  # #""" 'D'&&&& '&&&  >B$&&&'gvgT!WM M M'&& '&& sHD HDDD HDHE3D.E3"D42D03D47 E3D2E3EE3!AE5E6E: E3EE3 HDHD+ D D+ $D+ &H.E30D42E34E :D=; E E  E3EE3E0 E E0 (E0 *E3/H0E33 H>,G?*F- +G?/G# G G# G?GG?#G< )G,* G< 5G< 7G??HHc0V^8dQhR\R\/#)r: alvara_typer>rd)rAs"rBrCrCRscCrDc b"\4;_uu_4GRjxL p\VRVP4 2/4GRjxL p\W4GRjxL p\ YRVP R4;'g^\ VP R4;'g.4R7GRjxL TuuRRR4GRjxL #LLLnLL \d=p\PRT4\ YR\T4R 7GRjxL hRp?ii;i +GRjxL 'giR#;i5i) uDaily scrape of the PSP sigesponline roster for the given alvará type. Upserts alvara_companies and auto-registers newly seen NIPCs as monitoring_type='related'.N psp_alvara_rcountr)rzrun_psp_alvara failed: %rrRr) r rOlowerrr^rrrrrr)rr;rTresultrs& rBrun_psp_alvararRs!"""g!'[9J9J9L8M+NPRSS +GAAF **W-22VZZ.44"5   #""SA  #    8! <ggT!WM M M  #"""sD/B>D/"DCDCC"C7CC%C&C+ D/8C9D/DCCD/ D0D D D  DDD, D D, $D, & D/c$V^8dQhR\/#rcrd)rAs"rBrCrCfs!!S!rDc :"\4;_uu_4GRjxL p\VR/4GRjxL p\V4GRjxL pRRR4GRjxL ^p^p^pRp\4GRjxL p\X4GRjxL pVP4F]wrV\V 4, p\4;_uu_4GRjxL pT\W V 4GRjxL , pRRR4GRjxL K_ \4;_uu_4GRjxL p\VXVW5,WE,R7GRjxL RRR4GRjxL V#EL.ELEL L +GRjxL 'giEL;iEL \d"p\ P RT4Rp?EL'Rp?ii;iEL!LLL +GRjxL 'giEK0;i \d#p\ PRT4RpRp?LRp?ii;iLLL +GRjxL 'giX#;i5i)uFetch official DRE RSS feeds (Série I + II) once for per-company matching, AND archive Série II Parte E into `dre_raw` for retroactive match when a new company is added later.Ndrerzdre raw archive failed: %szdre run failed: %srRr) r rOrprrrrritemsrrrr^) r;rTrrr archived_newrUr by_companyrgrs rB run_dre_fetchrfs!"""g!'5"55.w77 #"JIL F +M+*** -q1# #""" MsHEHEE EE E HE H E2-E/.E22GF!AGF$ GF*F&  F*" G-F(.G6H G8 H G>0G:1G>5 HG<H E EHE, E E, $E, & H/E22 F=FGFG$G&F*(G*G 0F31 G <G > G G5G0+H0G55H:G><H>H H H H  HcV^8dQhRR/#r:r>Nr)rAs"rBrCrCs4rDc"\4;_uu_4GRjxL pVP\R44GRjxL VP\R44GRjxL VP\R44GRjxL VP4GRjxL RRR4GRjxL R#LLtLTL4LL +GRjxL 'giR#;i5i)zhMaintenance: drop old logs, expire sessions, recover stale 'running' jobs (killed by container restart).NzHDELETE FROM scraping_logs WHERE started_at < now() - interval '180 days'z2DELETE FROM user_sessions WHERE expires_at < now()a8 UPDATE scraping_logs SET status = 'error', finished_at = now(), error = COALESCE(error, 'killed by restart or timeout') WHERE status = 'running' AND started_at < now() - interval '2 hours' rr;s rB vacuum_logsrs!"""goo [ \   ood#WXYYYoo    nn)#""  Z  )#"""sCB.CB:B0!B:B2!B:?B4B:B6B: C'B8(C0B:2B:4B:6B:8C:C C C C  CcV^8dQhRR/#rr)rAs"rBrCrCsTrDc""\4;_uu_4GRjxL pVP\R44GRjxL VP4GRjxL RRR4GRjxL R#LPL0LL +GRjxL 'giR#;i5i)uCalled once on backend lifespan startup. Any 'running' row from a previous process is by definition orphaned — mark them failed immediately.Nz UPDATE scraping_logs SET status = 'error', finished_at = now(), error = COALESCE(error, 'killed by backend restart') WHERE status = 'running' rrs rBreap_stale_running_on_startuprsm!"""goo    nn#""  #"""slBA*BA2A,A2A.A2 B#A0$B,A2.A20B2B 8A;9 B B  Bc@V^8dQhR\R\R\RR/#)r:r<argskwargsr>N)r?r)rAs"rBrCrCs.GGcGcGSGTGrDc"V!V/VBGRjxL R#L \d"p\PRY4Rp?R#Rp?ii;i5i)Nzbackground scrape %s failed: %s)rrr)r<r!rrrs&&*, rB _runner_forrsCG$!&!!! G:FFFGs3A A A?AAAc\V^8dQhR\R\R\R\P/#)r:r<rrr>)r?rrTask)rAs"rBrCrCs2FFFFsFw||FrDch\P!4pVP\W.VO5/VB4#r)rget_running_looprr)r<r!rrloops&&*, rBschedule_backgroundrs0  # # %D   KDTDVD EErDc(V^8dQhR\RR/#)r:rar>Nrd)rAs"rBrCrCs G G3 G4 GrDc"\4;_uu_4GRjxL pVP\R4RV/4GRjxL P4pV'gRRR4GRjxL R#\ WV^,4GRjxL RRR4GRjxL R#LL\L7LL +GRjxL 'giR#;i5i)u~Scheduler wrapper around `retroactively_match_company` — resolves the company's legal_name from DB and runs the retro match.Nz/SELECT legal_name FROM companies WHERE id = :idrX)r rIr rLr)rar;rNs& rB_run_dre_retro_matchrs!"""g//FGz"  %'  #""*'s1vFFF#"" # G#"""sCBC!B#BB#B# C%B&C,B#BB# CB!CB#CB#!C#B= )B,* B= 5B= 7 CcBV^8dQhR\R\R,RR/#)r:rarhNr>rd)rAs"rBrCrCs%( ( c( d ( d( rDc\R\RRRV4\R\RRV4\R\R4\R\V4TpR#) u-Fire distribuicao + cire + contracts history for a newly-created company, off the request thread. All three paths use the same canonical sources as the daily/weekly crons (no third-party aggregators): * distribuicao — CITIUS per-company sweep, 180-day window * cire — CITIUS per-NIF, days=todos * contracts — IMPIC dumps on dados.gov.pt, force=True so old ZIPs are re-processed with the new NIF included in the monitored set The contracts path re-downloads ~500 MB of ZIPs (~2 min); accepted as a one-off on-create cost in exchange for using the same authoritative pipeline as the weekly cron (no ptdata 504 mystery, no stale cache). Also fires the publicacoes.mj.pt historical scrape (Playwright + Whisper audio bypass of reCAPTCHA v2) — that's the full atos societários archive.rNrtodosrT dre_retro)internal competitoranalysis)rrrrr)rarhrs&& rBschedule_backfill_for_companyrsa" d:J 7    ArDcJV^8dQhR\R\\R3,/#)r:ryr>.)rrcr?)rAs"rBrCrCs"--s-uS#X-rDcV'gR#\V\4'dV.p\;QJd.RV4FNK 5M !RV44pT;'gR#)z=Default to (internal, competitor) when caller omits a filter.c3>"TFq\9gKVxK R#5ir) _VALID_TYPES)rrs& rBr _coerce_types..s43a|"33s  )rr) isinstancer?rc)ryouts& rB _coerce_typesrsL ))#se %434%%434 4C  , ,,,rDclV^8dQhR\R\\\3,R,R\/#)r:r<optionsNr>)r?r@r)rAs"rBrCrC s6FF FsCx.4/FFrDc "T;'g/p\VPR44pVR8XdVPR4pVPR4p\V\4'd&\P !VR4P 4p\V\4'd&\P !VR4P 4p\R\W4V4pEMVR8Xdr\R\\VPR4;'gR4\VPR 4;'gR 4\VPR 444pEM+VR 8Xd6\R \\VPR 4;'gR44pEMVR8Xd&VPRR4p\R\Wb4pEMVR9d\R\4pEMVR8Xd\R\ 4pEMVR9EdIVPR4pVPR4p\V\4'd&\P !VR4P 4p\V\4'd&\P !VR4P 4p\VPR 4;'gR4p\VPR4;'gR4pVR8Xd7\R\"Y4\VPR 4;'gR4V4pMlVR8Xd\R\$W4pMS\R\&W4pMAVR8Xd-\VPR44p \R\(V 4pM\+RV 24hR\-V4 2#5i)zcEnqueue a manual scrape. Returns a log_id placeholder (the real log is inserted by the job itself).r`rryrzz%Y-%m-%dr:r4@ r5 @r6rrrr}rrvrrSrGr\ @irrzunknown source: ztask-)ptdatarh)rSrGr\)rrrr?rstrptimerrrrDrVr8rdrrrrrYrPrar ValueErrorrX) r<rr`ryrztaskr}budgetrrs && rBtrigger_manual_scraper s mmG '++g. /E KK , ++i( i % % )))Z@EEGI gs # #''<AACG" ,i%  & &" !6  K(22F 3 '++m,77 8 V$ %   * *# # %  J'..3 /  6 {{67+"68TA $ $"=/B 5"5-8 L L KK , ++i( i % % )))Z@EEGI gs # #''<AACGw{{=1;;V<GKK ,445 % %& "4iGKK +22s3VD( (&'8:Mu]D&'9;OQV_D ; W[[)*";0H%P+F8455 2d8* s'DN  "N .AN D'N +#N B>N cV^8dQhRR/#rr)rAs"rBrCrCRsPPtPrDc Z\P\R^^RRR7\P\R^^RRRR/R7\P\R^^RRR7\P\ R^^R RR7\P\ R^^-R RR7\P\R^^R ^xR R /RRR7\P\R^^R RR R/RRR7\P\R^^ RRR7\P\R^^R^ R R/RRR7\P\RR^(R RR R/RRR7\P\R^RRR7\P\R^^RRR7\P\R^^RRRR/R7\P\R^^R RR7\P\R^^R!RR7\P\ RR"^^R#RR$7\P\"RR%^^R&RR$7R'#)(crondistribuicao_dailyT)hourminuterXreplace_existing cire_dailyr}30)rrrXrrcire_announcements_dailyinsolvency_digest_daily mj_irn_dailyr4r5g@registry_expand_daily)rrrrXrrrregistry_nacional_nightlyentity_incidents_dailyrg@registry_sweep_dailyz 20-23,0-7g@registry_content_loopirn_canary_hourly)rrXr refresh_nifpsp_alvara_dailyrAfetch_dre_morningfetch_dre_nightmoncontracts_weekly) day_of_weekrrrXrsunrN) scheduleradd_jobrrrrsr1rPrDrmrYrarirrrrrrrDrB register_jobsrRs&q $&q $~Qr %vAa $tF2 DV!BS-8 "TvBrV]G< &f1R #dF2B v6 !D0f;rV]F; "Tq  a 4Qr s#vAa vBr t &" VQq 4rDc\V^8dQhR\\\\3,,/#rc)rer@r?r)rAs"rBrCrCs  T$sCx.1 rDc(.p\P4FpTPRVPRVP'dVPP M VPR\ VP4RVP'dVPP4MRRVP;'g/P4UUu/uF)wr#T\V\4'd \V4MTbK+ upp/4K V#uuppi)zCSnapshot of all scheduled jobs and their next run times for the UI.rXr0trigger next_run_timeNr)rget_jobsrrXfunc__name__r?rrrrrrrcre)rjkvs rBlist_scheduled_jobsrs "C    ! !$$ qvvvAFFOO144 s199~ AOOOQ__668QU ahhnnZ\McMcMefMeTQqZ5%9%9$q'q@Mef   " Jgs/DcV^8dQhRR/#rr)rAs"rBrCrCstrDcB\4\P4R#r)rrstartrrDrBrrsO OOrDcV^8dQhRR/#rr)rAs"rBrCrCs##$#rDc2\PRR7R#)F)waitN)rshutdownrrDrBrrs E"rD)rrr/NF)F)NNrNFr)rN)rrNF)NN)Nr)rg@F)< @T)NNr)ir)r>rrr)rrJloggingrrrrtypingrapscheduler.schedulers.asyncior sqlalchemyr sqlalchemy.ext.asyncior app.dbr app.scrapers.baser app.scrapers.cirerapp.scrapers.cire_announcementsrrapp.scrapers.mj_irnrrapp.scrapers.distribuicaorrrrapp.services.nif_lookuprapp.services.contracts_impicrapp.services.alvara_syncrapp.services.drerrrr%app.services.cire_announcement_ingestr%app.services.cire_announcement_parserrr app.services.court_filingsr!r"r#app.services.coverager$app.services.ingestr%app.services.riskr&app.services.insolvency_alertsr' app.services.insolvency_matchingr(app.services.mj_irn_syncr)r+app.services.registry_graphr*r+r,r-app.services.registry_queryr.app.services.registry_sweepr/r0r1r2r3r4app.services.registry_syncr5r6r7r8 getLoggerrrrrOr^rprwrrrrrrrrr rrr1rDrPrYrarirmrsrrrrrrrrrrrrrrrrrrrDrBr"s 88;/$*1M> /=5 FVYY...;?D A   8 $ o 6 "    :<~#4"%[|@F9&xSlSlEP#&AH! aHFRCL<~.b,2,DNB(!H4$GF G( V6 -FRPf  #rD