Sponge – A Linux tool which makes you feel you missed it for years

How often do you come across a situation where you need to parse a file, change some value and write the content back to the file? This mostly occurs when you are dealing with configuration files, which requires some script automated editing. More often than not you will involve a temperory file to do the intermediate changes and then later overwrite the original file. The reason you use temporary  file is because if you use  pipes and redirections then the output content stream starts flowing  before the input stream gets completed. To understand lets look at this example

Here is the content of the file pattern.txt


Now lets try to replace the string “test” with “tested” and remove the last two lines.

$sed "s/test/tested/" pattern.txt|head -n -2

Now lets write the resulting output back to the same file i.e pattern.txt and see what

 $sed "s/test/tested/" pattern.txt|head -n -2 >pattern.txt 
$cat pattern.txt 


You see the file is empty thats why a temperory file is used.

$sed "s/test/tested/" pattern.txt|head -n -2>tmp.txt
$cp tmp.txt pattern.txt

But the above solution is a two step process and it a work around as compared to a one step process. Now lets try to solve this problem using the command sponge. sponge reads standard input and writes it out to the specified file.
Unlike a shell redirect,sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.

sudo apt-get install moreutils
$sed "s/test/tested/" pattern.txt|head -n -2|sponge pattern.txt
$cat pattern.txt

Now the stream is piggybacked by sponge smoothly such that we neither need a temporary file
nor need two steps of execution.

While I was testing the examples for the blog I overwrote my pattern file many times.
But I had a backup of that file, So my piece of advice is always to have a backup of
what you are working whether you are working with pattern.txt or Pixar.

Playing with your original file is like “Cutting a branch on which you are sitting”

2 thoughts on “Sponge – A Linux tool which makes you feel you missed it for years

  • Pingback: Links 19/3/2017: Linux Sightings, What’s Wrong With Microsoft, and Death of Docker | Techrights

  • June 19, 2019 at 7:38 am

    Sponge is awesome and I use it all the time, but your example is not great because in that scenario using sed’s built in in-place editing would be much better:

    $ sed -i -e “s/test/tested/” pattern.txt

    That’s it. No redirects, no temp files, no extra utilities, just in-place editing. Note the argument order matters because some versions of sed (the BSD one on OSX) expect `-i` to have an argument so you can’t put it before the `pattern.txt`, putting it before `-e` makes it clear your intent is to not supply a backup pattern value.


Leave a Reply

Your email address will not be published. Required fields are marked *