Watching something be written to a file live with tail The 2019 Stack Overflow Developer Survey Results Are InRedirect output of remote python app started through ssh to fileUbuntu RFID Screensaver lock-unlockHow can I offer an XP-compatible migration to Linux Mint?Is there a command in Linux which waits till it will be terminated?Ctrl+c in a sub process is killing a nohup'ed process earlier in the scripthow to get cronjob running every minuteCtrl-C'd an in-place recursive gzip - is this likely to have broken anything?Getting output of another script while preserving line-breakssocat and rich terminal again“Ctrl + c” combination works different on different SSH clients

Why is Grand Jury testimony secret?

What is the motivation for a law requiring 2 parties to consent for recording a conversation

Why can Shazam do this?

Why isn't airport relocation done gradually?

How do you say "canon" as in "official for a story universe"?

Inline version of a function returns different value than non-inline version

aging parents with no investments

Inversion Puzzle

What is this 4-propeller plane?

Is bread bad for ducks?

It's possible to achieve negative score?

Which Sci-Fi work first showed weapon of galactic-scale mass destruction?

Where does the "burst of radiance" from Holy Weapon originate?

Inflated grade on resume at previous job, might former employer tell new employer?

Why could you hear an Amstrad CPC working?

Pristine Bit Checking

Is this food a bread or a loaf?

What tool would a Roman-age civilization have to grind silver and other metals into dust?

Limit the amount of RAM Mathematica may access?

How to deal with fear of taking dependencies

Is three citations per paragraph excessive for undergraduate research paper?

Geography at the pixel level

How to make payment on the internet without leaving a money trail?

The difference between dialogue marks



Watching something be written to a file live with tail



The 2019 Stack Overflow Developer Survey Results Are InRedirect output of remote python app started through ssh to fileUbuntu RFID Screensaver lock-unlockHow can I offer an XP-compatible migration to Linux Mint?Is there a command in Linux which waits till it will be terminated?Ctrl+c in a sub process is killing a nohup'ed process earlier in the scripthow to get cronjob running every minuteCtrl-C'd an in-place recursive gzip - is this likely to have broken anything?Getting output of another script while preserving line-breakssocat and rich terminal again“Ctrl + c” combination works different on different SSH clients



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








25















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question



















  • 10





    How about python myprog.py | tee output.txt instead?

    – n8te
    Apr 3 at 23:35






  • 8





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    Apr 4 at 2:42






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    Apr 4 at 18:41






  • 1





    Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    Apr 4 at 19:02







  • 2





    Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    Apr 4 at 19:04

















25















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question



















  • 10





    How about python myprog.py | tee output.txt instead?

    – n8te
    Apr 3 at 23:35






  • 8





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    Apr 4 at 2:42






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    Apr 4 at 18:41






  • 1





    Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    Apr 4 at 19:02







  • 2





    Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    Apr 4 at 19:04













25












25








25


4






I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question
















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?







linux command-line redirection stdout






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Apr 4 at 19:01









Peter Cordes

2,5451622




2,5451622










asked Apr 3 at 23:31









interstarinterstar

438613




438613







  • 10





    How about python myprog.py | tee output.txt instead?

    – n8te
    Apr 3 at 23:35






  • 8





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    Apr 4 at 2:42






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    Apr 4 at 18:41






  • 1





    Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    Apr 4 at 19:02







  • 2





    Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    Apr 4 at 19:04












  • 10





    How about python myprog.py | tee output.txt instead?

    – n8te
    Apr 3 at 23:35






  • 8





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    Apr 4 at 2:42






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    Apr 4 at 18:41






  • 1





    Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    Apr 4 at 19:02







  • 2





    Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    Apr 4 at 19:04







10




10





How about python myprog.py | tee output.txt instead?

– n8te
Apr 3 at 23:35





How about python myprog.py | tee output.txt instead?

– n8te
Apr 3 at 23:35




8




8





@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

– JPhi1618
Apr 4 at 2:42





@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

– JPhi1618
Apr 4 at 2:42




1




1





stdbuf can be used to alter the buffering status of file descriptors.

– studog
Apr 4 at 18:41





stdbuf can be used to alter the buffering status of file descriptors.

– studog
Apr 4 at 18:41




1




1





Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

– Peter Cordes
Apr 4 at 19:02






Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

– Peter Cordes
Apr 4 at 19:02





2




2





Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

– Mark Wagner
Apr 4 at 19:04





Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

– Mark Wagner
Apr 4 at 19:04










5 Answers
5






active

oldest

votes


















42














You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



import sys
...
print('Some message')
sys.stdout.flush()





share|improve this answer










New contributor




Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.















  • 7





    If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

    – mckenzm
    Apr 4 at 3:46






  • 11





    You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

    – Dan
    Apr 4 at 11:48







  • 12





    It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

    – glglgl
    Apr 4 at 13:38






  • 1





    Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

    – Roel Schroeven
    Apr 5 at 9:30






  • 2





    @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

    – Charles Duffy
    Apr 6 at 1:00


















33














Run python with the unbuffered flag:



python -u myprog.py > output.txt


Output will then print in real time.






share|improve this answer








New contributor




BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.















  • 4





    This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

    – Roel Schroeven
    Apr 5 at 9:29











  • Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

    – ivanivan
    Apr 5 at 22:23


















19














Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



From man tee:




tee(1) - Linux man page



Name tee - read from standard input and write to standard output and files



Synopsis



tee [OPTION]... [FILE]...


Description



Copy standard input to each FILE, and also to standard output.



-a, --append 
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit


If a FILE is -, copy again to standard output.




So in your case you'd run:



python myprog.py | tee output.txt


EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






share|improve this answer




















  • 1





    tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

    – Baldrickk
    Apr 4 at 7:45






  • 11





    That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

    – eckes
    Apr 4 at 9:22






  • 8





    This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

    – Barmar
    Apr 4 at 17:46


















9














Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



This is a redirect to a regular file.



C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






share|improve this answer






























    0














    When I use tail, it is almost-always tracking a log file, eg (email) messages.



    This might be slightly off-the-wall, but instead of using print/print()/write() in your Python code, why not use the logging module? (from the PSL) NB a logging formatter can be configured NOT to output all those time and ID-codes associated with a traditional log.



    The output can be configured to go to a (data) file, and because there is no buffering delay or redirection, tail works happily and with immediacy.



    Regards






    share|improve this answer










    New contributor




    d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.




















      Your Answer








      StackExchange.ready(function()
      var channelOptions =
      tags: "".split(" "),
      id: "3"
      ;
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function()
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled)
      StackExchange.using("snippets", function()
      createEditor();
      );

      else
      createEditor();

      );

      function createEditor()
      StackExchange.prepareEditor(
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader:
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      ,
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      );



      );













      draft saved

      draft discarded


















      StackExchange.ready(
      function ()
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');

      );

      Post as a guest















      Required, but never shown

























      5 Answers
      5






      active

      oldest

      votes








      5 Answers
      5






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      42














      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 7





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        Apr 4 at 3:46






      • 11





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        Apr 4 at 11:48







      • 12





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        Apr 4 at 13:38






      • 1





        Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

        – Roel Schroeven
        Apr 5 at 9:30






      • 2





        @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

        – Charles Duffy
        Apr 6 at 1:00















      42














      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 7





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        Apr 4 at 3:46






      • 11





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        Apr 4 at 11:48







      • 12





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        Apr 4 at 13:38






      • 1





        Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

        – Roel Schroeven
        Apr 5 at 9:30






      • 2





        @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

        – Charles Duffy
        Apr 6 at 1:00













      42












      42








      42







      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.










      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()






      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this answer



      share|improve this answer








      edited Apr 4 at 4:35









      user2313067

      2,1001911




      2,1001911






      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      answered Apr 4 at 0:05









      DaveyDavey

      49235




      49235




      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.







      • 7





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        Apr 4 at 3:46






      • 11





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        Apr 4 at 11:48







      • 12





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        Apr 4 at 13:38






      • 1





        Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

        – Roel Schroeven
        Apr 5 at 9:30






      • 2





        @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

        – Charles Duffy
        Apr 6 at 1:00












      • 7





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        Apr 4 at 3:46






      • 11





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        Apr 4 at 11:48







      • 12





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        Apr 4 at 13:38






      • 1





        Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

        – Roel Schroeven
        Apr 5 at 9:30






      • 2





        @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

        – Charles Duffy
        Apr 6 at 1:00







      7




      7





      If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

      – mckenzm
      Apr 4 at 3:46





      If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

      – mckenzm
      Apr 4 at 3:46




      11




      11





      You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

      – Dan
      Apr 4 at 11:48






      You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

      – Dan
      Apr 4 at 11:48





      12




      12





      It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

      – glglgl
      Apr 4 at 13:38





      It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

      – glglgl
      Apr 4 at 13:38




      1




      1





      Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

      – Roel Schroeven
      Apr 5 at 9:30





      Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.

      – Roel Schroeven
      Apr 5 at 9:30




      2




      2





      @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

      – Charles Duffy
      Apr 6 at 1:00





      @wizzwizz4, it's only different in that Python follows the standard C library's convention of, by default, configuring stderr to be unbuffered and stdout to be line-buffered when they point to a TTY, but stdout to be unbuffered if it's opened to a non-TTY device.

      – Charles Duffy
      Apr 6 at 1:00













      33














      Run python with the unbuffered flag:



      python -u myprog.py > output.txt


      Output will then print in real time.






      share|improve this answer








      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 4





        This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

        – Roel Schroeven
        Apr 5 at 9:29











      • Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

        – ivanivan
        Apr 5 at 22:23















      33














      Run python with the unbuffered flag:



      python -u myprog.py > output.txt


      Output will then print in real time.






      share|improve this answer








      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 4





        This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

        – Roel Schroeven
        Apr 5 at 9:29











      • Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

        – ivanivan
        Apr 5 at 22:23













      33












      33








      33







      Run python with the unbuffered flag:



      python -u myprog.py > output.txt


      Output will then print in real time.






      share|improve this answer








      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.










      Run python with the unbuffered flag:



      python -u myprog.py > output.txt


      Output will then print in real time.







      share|improve this answer








      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this answer



      share|improve this answer






      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      answered Apr 4 at 16:14









      BHCBHC

      43913




      43913




      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.







      • 4





        This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

        – Roel Schroeven
        Apr 5 at 9:29











      • Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

        – ivanivan
        Apr 5 at 22:23












      • 4





        This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

        – Roel Schroeven
        Apr 5 at 9:29











      • Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

        – ivanivan
        Apr 5 at 22:23







      4




      4





      This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

      – Roel Schroeven
      Apr 5 at 9:29





      This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.

      – Roel Schroeven
      Apr 5 at 9:29













      Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

      – ivanivan
      Apr 5 at 22:23





      Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."

      – ivanivan
      Apr 5 at 22:23











      19














      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer




















      • 1





        tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        Apr 4 at 7:45






      • 11





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        Apr 4 at 9:22






      • 8





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        Apr 4 at 17:46















      19














      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer




















      • 1





        tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        Apr 4 at 7:45






      • 11





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        Apr 4 at 9:22






      • 8





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        Apr 4 at 17:46













      19












      19








      19







      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer















      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.







      share|improve this answer














      share|improve this answer



      share|improve this answer








      edited Apr 4 at 21:29

























      answered Apr 4 at 0:11









      n8ten8te

      5,39172235




      5,39172235







      • 1





        tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        Apr 4 at 7:45






      • 11





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        Apr 4 at 9:22






      • 8





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        Apr 4 at 17:46












      • 1





        tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        Apr 4 at 7:45






      • 11





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        Apr 4 at 9:22






      • 8





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        Apr 4 at 17:46







      1




      1





      tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

      – Baldrickk
      Apr 4 at 7:45





      tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

      – Baldrickk
      Apr 4 at 7:45




      11




      11





      That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

      – eckes
      Apr 4 at 9:22





      That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

      – eckes
      Apr 4 at 9:22




      8




      8





      This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

      – Barmar
      Apr 4 at 17:46





      This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

      – Barmar
      Apr 4 at 17:46











      9














      Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



      This is a redirect to a regular file.



      C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



      You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



      On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



      Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




      Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






      share|improve this answer



























        9














        Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



        This is a redirect to a regular file.



        C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



        You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



        On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



        Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




        Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






        share|improve this answer

























          9












          9








          9







          Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



          This is a redirect to a regular file.



          C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



          You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



          On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



          Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




          Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






          share|improve this answer













          Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



          This is a redirect to a regular file.



          C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



          You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



          On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



          Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




          Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Apr 4 at 19:18









          Peter CordesPeter Cordes

          2,5451622




          2,5451622





















              0














              When I use tail, it is almost-always tracking a log file, eg (email) messages.



              This might be slightly off-the-wall, but instead of using print/print()/write() in your Python code, why not use the logging module? (from the PSL) NB a logging formatter can be configured NOT to output all those time and ID-codes associated with a traditional log.



              The output can be configured to go to a (data) file, and because there is no buffering delay or redirection, tail works happily and with immediacy.



              Regards






              share|improve this answer










              New contributor




              d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
              Check out our Code of Conduct.
























                0














                When I use tail, it is almost-always tracking a log file, eg (email) messages.



                This might be slightly off-the-wall, but instead of using print/print()/write() in your Python code, why not use the logging module? (from the PSL) NB a logging formatter can be configured NOT to output all those time and ID-codes associated with a traditional log.



                The output can be configured to go to a (data) file, and because there is no buffering delay or redirection, tail works happily and with immediacy.



                Regards






                share|improve this answer










                New contributor




                d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.






















                  0












                  0








                  0







                  When I use tail, it is almost-always tracking a log file, eg (email) messages.



                  This might be slightly off-the-wall, but instead of using print/print()/write() in your Python code, why not use the logging module? (from the PSL) NB a logging formatter can be configured NOT to output all those time and ID-codes associated with a traditional log.



                  The output can be configured to go to a (data) file, and because there is no buffering delay or redirection, tail works happily and with immediacy.



                  Regards






                  share|improve this answer










                  New contributor




                  d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.










                  When I use tail, it is almost-always tracking a log file, eg (email) messages.



                  This might be slightly off-the-wall, but instead of using print/print()/write() in your Python code, why not use the logging module? (from the PSL) NB a logging formatter can be configured NOT to output all those time and ID-codes associated with a traditional log.



                  The output can be configured to go to a (data) file, and because there is no buffering delay or redirection, tail works happily and with immediacy.



                  Regards







                  share|improve this answer










                  New contributor




                  d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.









                  share|improve this answer



                  share|improve this answer








                  edited yesterday









                  Alfabravo

                  528413




                  528413






                  New contributor




                  d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.









                  answered yesterday









                  d-nd-n

                  11




                  11




                  New contributor




                  d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.





                  New contributor





                  d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.






                  d-n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.



























                      draft saved

                      draft discarded
















































                      Thanks for contributing an answer to Super User!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid


                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.

                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function ()
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');

                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      Popular posts from this blog

                      រឿង រ៉ូមេអូ និង ហ្ស៊ុយលីយេ សង្ខេបរឿង តួអង្គ បញ្ជីណែនាំ

                      Crop image to path created in TikZ? Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Crop an inserted image?TikZ pictures does not appear in posterImage behind and beyond crop marks?Tikz picture as large as possible on A4 PageTransparency vs image compression dilemmaHow to crop background from image automatically?Image does not cropTikzexternal capturing crop marks when externalizing pgfplots?How to include image path that contains a dollar signCrop image with left size given

                      Romeo and Juliet ContentsCharactersSynopsisSourcesDate and textThemes and motifsCriticism and interpretationLegacyScene by sceneSee alsoNotes and referencesSourcesExternal linksNavigation menu"Consumer Price Index (estimate) 1800–"10.2307/28710160037-3222287101610.1093/res/II.5.31910.2307/45967845967810.2307/2869925286992510.1525/jams.1982.35.3.03a00050"Dada Masilo: South African dancer who breaks the rules"10.1093/res/os-XV.57.1610.2307/28680942868094"Sweet Sorrow: Mann-Korman's Romeo and Juliet Closes Sept. 5 at MN's Ordway"the original10.2307/45957745957710.1017/CCOL0521570476.009"Ram Leela box office collections hit massive Rs 100 crore, pulverises prediction"Archived"Broadway Revival of Romeo and Juliet, Starring Orlando Bloom and Condola Rashad, Will Close Dec. 8"Archived10.1075/jhp.7.1.04hon"Wherefore art thou, Romeo? To make us laugh at Navy Pier"the original10.1093/gmo/9781561592630.article.O006772"Ram-leela Review Roundup: Critics Hail Film as Best Adaptation of Romeo and Juliet"Archived10.2307/31946310047-77293194631"Romeo and Juliet get Twitter treatment""Juliet's Nurse by Lois Leveen""Romeo and Juliet: Orlando Bloom's Broadway Debut Released in Theaters for Valentine's Day"Archived"Romeo and Juliet Has No Balcony"10.1093/gmo/9781561592630.article.O00778110.2307/2867423286742310.1076/enst.82.2.115.959510.1080/00138380601042675"A plague o' both your houses: error in GCSE exam paper forces apology""Juliet of the Five O'Clock Shadow, and Other Wonders"10.2307/33912430027-4321339124310.2307/28487440038-7134284874410.2307/29123140149-661129123144728341M"Weekender Guide: Shakespeare on The Drive""balcony"UK public library membership"romeo"UK public library membership10.1017/CCOL9780521844291"Post-Zionist Critique on Israel and the Palestinians Part III: Popular Culture"10.2307/25379071533-86140377-919X2537907"Capulets and Montagues: UK exam board admit mixing names up in Romeo and Juliet paper"Istoria Novellamente Ritrovata di Due Nobili Amanti2027/mdp.390150822329610820-750X"GCSE exam error: Board accidentally rewrites Shakespeare"10.2307/29176390149-66112917639"Exam board apologises after error in English GCSE paper which confused characters in Shakespeare's Romeo and Juliet""From Mariotto and Ganozza to Romeo and Guilietta: Metamorphoses of a Renaissance Tale"10.2307/37323537323510.2307/2867455286745510.2307/28678912867891"10 Questions for Taylor Swift"10.2307/28680922868092"Haymarket Theatre""The Zeffirelli Way: Revealing Talk by Florentine Director""Michael Smuin: 1938-2007 / Prolific dance director had showy career"The Life and Art of Edwin BoothRomeo and JulietRomeo and JulietRomeo and JulietRomeo and JulietEasy Read Romeo and JulietRomeo and Julieteeecb12003684p(data)4099369-3n8211610759dbe00d-a9e2-41a3-b2c1-977dd692899302814385X313670221313670221