SED (Linux) – Joining All Lines Using Sed command line tool

PRE-SCENARIO

You have to join all lines in a txt file to a single line through a script. So you probably want to use sed to make multiple modification to a source file...

To join all lines to a single line you could use this command:

Unix/Linux:  sed -e :a -e ‘$!N;s/\n/ /;ta’

Windows:  sed -e :a -e "$!N;s/\n/ /;ta"

SCENARIO

Recently I was trying to install Python 2.5 on a CentOS 5 OpenVZ VE. I found a blog post over on the blog.bashton.com site that provided some RPMS but they were for thex86_64 architecture and I needed them for i386. Not to worry, they also provided a SRPM so I could just hit it with rpmbuild.

Now I’m a keyboard and shell kind of guy but I’m lazy. So anytime I can take the output from one command and use it as arguments into another command I’m all for it. Here is a cool trick where I can take the output from the rpmbuild of the python 2.5 SRPM and create a list of all the packages that I need to yum install.

[tab:The Victim]

The Victim

Doing the rpmbuild of a SRPM returns this:

% rpmbuild --rebuild python25-2.5.1-bashton1.src.rpm 2>&1 | grep "is needed" | sed 's/^[ \t]\+//g'
readline-devel is needed by python25-2.5.1-bashton1.i386
openssl-devel is needed by python25-2.5.1-bashton1.i386
gmp-devel is needed by python25-2.5.1-bashton1.i386
ncurses-devel is needed by python25-2.5.1-bashton1.i386
gdbm-devel is needed by python25-2.5.1-bashton1.i386
zlib-devel is needed by python25-2.5.1-bashton1.i386
expat-devel is needed by python25-2.5.1-bashton1.i386
libGL-devel is needed by python25-2.5.1-bashton1.i386
tk is needed by python25-2.5.1-bashton1.i386
tix is needed by python25-2.5.1-bashton1.i386
gcc-c++ is needed by python25-2.5.1-bashton1.i386
libX11-devel is needed by python25-2.5.1-bashton1.i386
glibc-devel is needed by python25-2.5.1-bashton1.i386
pkgconfig is needed by python25-2.5.1-bashton1.i386
tcl-devel is needed by python25-2.5.1-bashton1.i386
tk-devel is needed by python25-2.5.1-bashton1.i386
tix-devel is needed by python25-2.5.1-bashton1.i386
bzip2-devel is needed by python25-2.5.1-bashton1.i386
sqlite-devel is needed by python25-2.5.1-bashton1.i386
autoconf is needed by python25-2.5.1-bashton1.i386
db4-devel >= 4.3 is needed by python25-2.5.1-bashton1.i386
tcl is needed by python25-2.5.1-bashton1.i386

Here we’re picking out all the lines that contain “is needed” with this: grep “is needed”.

Next we’re cutting out any spaces or tabs that prefix each line with this command:sed ‘s/^[ \t]\+//g’.

[tab:Pre-Magic]

Pre Magic

Next we can use a quick awk “{ print $1 }” to truncate everything after the package names to get to this:

% rpmbuild --rebuild python25-2.5.1-bashton1.src.rpm 2>&1 | grep "is needed" | sed 's/^[ \t]\+//g' | awk '{print $1}'
readline-devel
openssl-devel
gmp-devel
ncurses-devel
gdbm-devel
zlib-devel
expat-devel
libGL-devel
tk
tix
gcc-c++
libX11-devel
glibc-devel
pkgconfig
tcl-devel
tk-devel
tix-devel
bzip2-devel
sqlite-devel
autoconf
db4-devel
tcl

[tab:Solution]
The Magic (Solution)

The real magic and the reason I’m taking the time to write this post up is this little bit of sed magic here:

% rpmbuild --rebuild python25-2.5.1-bashton1.src.rpm 2>&1 | grep "is needed" | sed 's/^[ \t]\+//g'| awk '{print $1}' | sed -e :a -e '$!N;s/\n/ /;ta'
readline-devel openssl-devel gmp-devel ncurses-devel gdbm-devel zlib-devel expat-devel libGL-devel tk tix gcc-c++ libX11-devel glibc-devel pkgconfig tcl-devel tk-devel tix-devel bzip2-devel sqlite-devel autoconf db4-devel tcl

This little gem:

sed -e :a -e ‘$!N;s/\n/ /;ta’

takes all the lines coming in and joins them into a single line.

[tab:Final Product]

The Final Product

Now you can wrap this command with a yum install `…` like this, and let the power of the shell do all the dirty work for you.

yum -y install `rpmbuild --rebuild python25-2.5.1-bashton1.src.rpm 2>&1 | grep "is needed" | sed 's/^[ \t]\+//g' | awk '{print $1}' | sed -e :a -e '$!N;s/\n/ /;ta'`

[tab:END]