python - Hvordan har du delte logfiler under Windows?

Indlæg af Hanne Mølgaard Plasc

Problem



Jeg har flere forskellige processer, og jeg vil gerne have, at de alle logger på den samme fil. Disse processer kører på et Windows 7-system. Nogle er python scripts, og andre er cmd batch-filer.


Under Unix skal du bare åbne alle filer i tilføjelsesmodus og skrive væk. Så længe hver proces skrev mindre end PIPE\_BUF bytes i en enkelt besked, ville hver write opkald ikke kunne interleave med nogen anden.


Er der en måde at gøre dette til under Windows? Den naive Unix-lignende tilgang mislykkes, fordi Windows ikke kan lide mere end én proces, der har en fil åben til skrivning ad gangen som standard.

Bedste reference


Det er muligt at skrive flere batchprocesser sikkert til en enkelt logfil. Jeg ved intet om Python, men jeg forestiller mig, at begreberne i dette svar kunne integreres med Python.


Windows giver højst en proces til at have en bestemt fil åben for skriveadgang på ethvert tidspunkt. Dette kan bruges til at implementere en filbaseret låsemekanisme, der garanterer hændelser er serialiseret på tværs af flere processer. Se https://stackoverflow.com/a/9048097/1012053 og http://www.dostips.com/forum/viewtopic.php?p=12454 for nogle eksempler. [11]


Da alt du forsøger at gøre er at skrive til en logfil, kan du selv bruge logfilen som lås. Log-operationen er indkapslet i en subrutine, der forsøger at åbne logfilen i append-tilstand. Hvis åbningen fejler, løber rutinen tilbage og prøver igen. Når den åbne er vellykket, loges log og derefter lukkes, og rutinen vender tilbage til den, der ringer op. Rutinen udfører, hvad kommandoen er sendt til den, og alt skrevet til stdout i rutinen omdirigeres til loggen.


Her er et test batch script, der skaber 5 barns processer, som hver skriver til logfilen 20 gange. Skrifterne er sikkert sammenflettet.


@echo off
setlocal
if "\%~1" neq "" goto :test

:: Initialize
set log="myLog.log"
2>nul del \%log\%
2>nul del "test*.marker"
set procCount=5
set testCount=10

:: Launch \%procCount\% processes that write to the same log
for /l \%\%n in (1 1 \%procCount\%) do start "" /b "\%~f0" \%\%n

:wait for child processes to finish
2>nul dir /b "test*.marker" | find /c "test" | >nul findstr /x "\%procCount\%" || goto :wait

:: Verify log results
for /l \%\%n in (1 1 \%procCount\%) do (
  <nul set /p "=Proc \%\%n log count = "
  find /c "Proc \%\%n: " <\%log\%
)

:: Cleanup
del "test*.marker"
exit /b

==============================================================================
:: code below is the process that writes to the log file

:test
set instance=\%1
for /l \%\%n in (1 1 \%testCount\%) do (
  call :log echo Proc \%instance\% says hello!
  call :log dir "\%~f0"
)
echo done >"test\%1.marker"
exit

:log command args...
2>nul (
  >>\%log\% (
    echo ***********************************************************
    echo Proc \%instance\%: \%date\% \%time\%
    \%*
    (call ) \%= This odd syntax guarantees the inner block ends with success  =\%
            \%= We only want to loop back and try again if redirection failed =\%
  )
) || goto :log
exit /b


Her er output, der viser, at alle 20 skriver var vellykkede for hver proces


Proc 1 log count = 20
Proc 2 log count = 20
Proc 3 log count = 20
Proc 4 log count = 20
Proc 5 log count = 20


Du kan åbne den resulterende 'myLog.log' -fil for at se, hvordan skrivningerne er sikkert sammenflettet. Men output er for stort til at poste her.


Det er let at demonstrere, at samtidige skrivninger fra flere processer kan mislykkes ved at ændre: logrutinen, så den ikke forsøger igen ved fejl.


:log command args...
>>\%log\% (
  echo ***********************************************************
  echo Proc \%instance\%: \%date\% \%time\%
  \%*
)
exit /b


Her er nogle prøveresultater efter 'breaking': log routinen


The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Proc 1 log count = 12
Proc 2 log count = 16
Proc 3 log count = 13
Proc 4 log count = 18
Proc 5 log count = 14

Andre referencer 1


Du kan give dette Python-modul et forsøg:
http://pypi.python.org/pypi/ConcurrentLogHandler[12]


Det giver en indskiftning af RotatingFileHandler, som gør det muligt for flere processer at logge på en enkelt fil samtidigt uden at tabe eller afbryde loghændelser.


Jeg har ikke brugt det, men jeg fandt ud af det under læsning på en relateret fejl (Issue 4749) i Python. [13]


Hvis du implementerer din egen kode for at gøre det i stedet for at bruge det modul, skal du sørge for at læse fejlen!


Du kan bruge output omdirigering på Windows som du gør i Bash. Rør output fra batch-filer til et Python-script, der logger igennem ConcurrentLogHandler. [14]