How to separate command output to individual lines

Question

list=`ls -a R*`
echo $list

Inside a shell script, this echo command will list all the files from current directory starting with R, but in one line. How can I print each item on one line?

I need a generic command for all the scenarios happening with ls, du, find -type -d, etc.

Answer 1

If the output of the command contains multiple lines, then quote your variables to preserve those newlines when echoing:

echo "$list"

Otherwise, the shell will expand and split the variable contents on whitespace (spaces, newlines, tabs) and those will be lost.

Answer 2

Instead of ill-advisedly putting ls output in a variable and then echoing it, which removes all the colors, use

ls -a1

From man ls

       -1     list one file per line.  Avoid '\n' with -q or -b

I don't advise you do anything with the output of ls, except display it :)

Use, for example, shell globs and a for loop to do something with files...

shopt -s dotglob                  # to include hidden files*

for i in R/*; do echo "$i"; done

* this won't include the current directory . or its parent .. though

Answer 3

While putting it in quotes as @muru suggested will indeed do what you asked for, you might also want to consider using an array for this. For example:

IFS=$'\n' dirs=( $(find . -type d) )

The IFS=$'\n' tells bash to only split the output on newline characcters o get each element of the array. Without it, it will split on spaces, so a file name with spaces.txt would be 5 separate elements instead of one. This approach will break if your file/directory names can contain newlines (\n) though. It will save each line of the command's output as an element of the array.

_{Note that I also changed the old-style `command` to $(command) which is the preferred syntax.}

You now have an array called $dirs, each bof whose elements is a line of the output of the previous command. For example:

$ find . -type d
.
./olad
./ho
./ha
./ads
./bar
./ga
./da
$ IFS=$'\n' dirs=( $(find . -type d) )
$ for d in "${dirs[@]}"; do
    echo "DIR: $d"
  done
DIR: .
DIR: ./olad
DIR: ./ho
DIR: ./ha
DIR: ./ads
DIR: ./bar
DIR: ./ga
DIR: ./da

Now, because of some bash strangeness (see here), you will need to reset IFS back to the original value after doing this. So, either save it and reasign:

oldIFS="$IFS"
IFS=$'\n' dirs=( $(find . -type d) ) 
IFS="$oldIFS"

Or do it manually:

IFS=" "$'\t\n '

Alternatively, just close the current terminal. Your new one will have the original IFS set again.

Answer 4

If you must store the output and you want an array, mapfile makes it easy.

First, consider if you need to store your command's output at all. If you don't, just run the command.

If you decide you want to read the output of a command as an array of lines, it's true that one way to do it is to disable globbing, set IFS to split on lines, use command substitution inside the ( ) array creation syntax, and reset IFS afterwards, which is more complex than it seems. terdon's answer covers some of that approach. But I suggest you use mapfile, the Bash shell's built-in command for reading text as an array of lines, instead. Here's a simple case where you're reading from a file:

mapfile < filename

This reads lines into an array called MAPFILE. Without the input redirection < filename, you would be reading from the shell's standard input (typically your terminal) instead of the file named by filename. To read into an array other than the default MAPFILE, pass its name. For example, this reads into the array lines:

mapfile lines < filename

Another default behavior you might choose to change is that the newline characters at the ends of lines are left in place; they appear as the last character in each array element (unless the input ended without a newline character, in which case the last element doesn't have one). To chomp these newlines so they don't appear in the array, pass the -t option to mapfile. For example, this reads to the array records and does not write trailing newline characters to its array elements:

mapfile -t records < filename

You can also use -t without passing an array name; that is, it works with an implicit array name of MAPFILE, too.

The mapfile shell builtin supports other options, and it can alternatively be invoked as readarray. Run help mapfile (and help readarray) for details.

But you don't want to read from a file, you want to read from the output of a command. To achieve that, use process substitution. This command reads lines from the command some-command with arguments... as its command-line arguments and places in them in mapfile's default array MAPFILE, with their terminating newline characters removed:

mapfile -t < <(some-command arguments...)

Process substitution replaces <(some-command arguments...) with an actual filename from which the output of running some-command arguments... can be read. The file is a named pipe rather than a regular file and, on Ubuntu, it will be named like /dev/fd/63 (sometimes with some other number than 63), but you don't need to concern yourself with the details, because the shell takes care of it all behind the scenes.

^{You might think you could forgo process substitution by using some-command arguments... | mapfile -t instead, but that won't work because, when you have a pipeline of multiple commands separated by |, Bash runs runs all commands in subshells. Thus both some-command arguments... and mapfile -t run in their own environments, initialized from but separate from the environment of the shell in which you run the pipeline. In the subshell where mapfile -t runs, the MAPFILE array does get populated, but then that array is discarded when the command ends. MAPFILE is never created or modified for the caller.}

Here's what the example in terdon's answer looks like, in entirety, if you use mapfile:

mapfile -t dirs < <(find . -type d)

for d in "${dirs[@]}"; do
    echo "DIR: $d"
done

That's it. You don't need to check if IFS was set, keep track of whether not it was set and with what value, set it to a newline, then reset or re-unset it later. You don't have to disable globbing (for example, with set -f)--which is really needed if you are to use that method seriously, since filenames may contain *, ?, and [--then re-enable it (set +f) afterwards.

You can also replace that particular loop entirely with a single printf command--though this is not actually a benefit of mapfile, as you can do that whether you use mapfile or another method to populate the array. Here's a shorter version:

mapfile -t < <(find . -type d)
printf 'DIR: %s\n' "${MAPFILE[@]}"

It's important to keep in mind that, as terdon mentions, operating line-by-line is not always appropriate, and will not work correctly with filenames that contain newlines. I recommend against naming files that way, but it can happen, including by accident.

There isn't really a one-size-fits-all solution.

You asked for "a generic command for all the scenarios," and the approach of using mapfile somewhat approaches that goal, but I urge you to reconsider your requirements.

The task shown above is better achieved with just a single find command:

find . -type d -printf 'DIR: %p\n'

You could also use an external command like sed to add DIR: to the beginning of each line. This is arguably somewhat ugly, and unlike that find command it will add extra "prefixes" within filenames that contain newlines, but it does work regardless of its input so it sort of meets your requirements and it is still preferable to reading the output into a variable or array:

find . -type d | sed 's/^/DIR: /'

If you need to list and also perform an action on each directory found, such as running some-command and passing the directory's path as an argument, find lets you do that too:

find . -type d -print -exec some-command {} \;

As another example, let's return to the general task of adding a prefix to each line. Suppose I want to see the output of help mapfile but number the lines. I would not actually use mapfile for this, nor any other method that reads it into a shell variable or shell array. Supposing help mapfile | cat -n doesn't give the formatting I want, I can use awk:

help mapfile | awk '{ printf "%3d: %s\n", NR, $0 }'

Reading all the output of a command in to a single variable or array is sometimes useful and appropriate, but it has major disadvantages. Not only do you have to deal with the additional complexity of using your shell when an existing command or combination of existing commands may already do what you need as well or better, but the entire output of the command has to be stored in memory. Sometimes you may know that isn't a problem, but sometimes you may be processing a large file.

An alternative that is commonly attempted--and sometimes done right--is to read input line-by-line with read -r in a loop. If you don't need to store previous lines while operating on later lines, and you have to use long input, then it may be better than mapfile. But it, too, should be avoided in cases when you can just pipe it to a command that can do the work, which is most cases.